2024-03-28T13:53:55.049 INFO:root:teuthology version: 0.0.1.dev262+ge691533 2024-03-28T13:53:55.049 DEBUG:teuthology.run:Teuthology command: teuthology --name pdonnell-2024-03-28_07:14:07-fs-wip-batrick-testing-20240327.230800-distro-default-smithi --description fs/shell/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1-mds-1-client-coloc conf/{client mds mgr mon osd} distro/ubuntu_latest mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile pg_health} tasks/cephfs-shell} --owner scheduled_pdonnell@teuthology --verbose --archive /home/teuthworker/archive/pdonnell-2024-03-28_07:14:07-fs-wip-batrick-testing-20240327.230800-distro-default-smithi/7627919 -- /home/teuthworker/archive/pdonnell-2024-03-28_07:14:07-fs-wip-batrick-testing-20240327.230800-distro-default-smithi/7627919/orig.config.yaml 2024-03-28T13:53:55.097 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-28T13:53:55.181 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/pdonnell-2024-03-28_07:14:07-fs-wip-batrick-testing-20240327.230800-distro-default-smithi/7627919 branch: wip-batrick-testing-20240327.230800 description: fs/shell/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1-mds-1-client-coloc conf/{client mds mgr mon osd} distro/ubuntu_latest mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile pg_health} tasks/cephfs-shell} email: pdonnell@redhat.com first_in_suite: false job_id: '7627919' kernel: kdb: true sha1: distro ktype: distro last_in_suite: false log-rotate: ceph-mds: 10G ceph-osd: 10G machine_type: smithi mgrmodules: sequential: - print: Enabling mgr modules name: pdonnell-2024-03-28_07:14:07-fs-wip-batrick-testing-20240327.230800-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 20 - machine: disk: 200 os_type: ubuntu os_version: '22.04' overrides: admin_socket: branch: wip-batrick-testing-20240327.230800 ceph: conf: client: client mount timeout: 600 debug client: 20 debug ms: 1 pid file: '' rados mon op timeout: 900 rados osd op timeout: 900 mds: debug mds: 20 debug mds balancer: 20 debug ms: 1 mds debug frag: true mds debug scatterstat: true mds op complaint time: 180 mds verify scatter: true osd op complaint time: 180 rados mon op timeout: 900 rados osd op timeout: 900 mgr: debug client: 20 debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon down mkfs grace: 300 mon op complaint time: 120 osd: bdev async discard: true bdev enable discard: true bluestore allocator: bitmap bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug ms: 1 debug osd: 20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore osd op complaint time: 180 flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - FS_DEGRADED - FS_INLINE_DATA_DEPRECATED - FS_WITH_FAILED_MDS - MDS_ALL_DOWN - MDS_DAMAGE - MDS_DEGRADED - MDS_FAILED - MDS_INSUFFICIENT_STANDBY - MDS_UP_LESS_THAN_MAX - filesystem is online with fewer MDS than max_mds - POOL_APP_NOT_ENABLED - overall HEALTH_ - Replacing daemon - deprecated feature inline_data - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding - PG_AVAILABILITY - PG_DEGRADED - Reduced data availability sha1: e0e48ba6ee8c8b4c8f38d3b879d6eafbbb1efc15 ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} osd: bdev async discard: true bdev enable discard: true bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs install: ceph: flavor: default sha1: e0e48ba6ee8c8b4c8f38d3b879d6eafbbb1efc15 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-batrick-testing-20240327.230800 sha1: e0e48ba6ee8c8b4c8f38d3b879d6eafbbb1efc15 owner: scheduled_pdonnell@teuthology priority: 75 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mgr.y - mds.a - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - - mon.b - mon.c - mgr.x - mds.b - osd.4 - osd.5 - osd.6 - osd.7 seed: 7752 sha1: e0e48ba6ee8c8b4c8f38d3b879d6eafbbb1efc15 sleep_before_teardown: 0 subset: 1/512 suite: fs suite_branch: wip-batrick-testing-20240327.230800 suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_e0e48ba6ee8c8b4c8f38d3b879d6eafbbb1efc15/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph-ci.git suite_sha1: e0e48ba6ee8c8b4c8f38d3b879d6eafbbb1efc15 syslog: ignorelist: - WARNING*.*check_session_state - WARNING*.*__ceph_remove_cap targets: smithi104.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBGXlb930Fp5uCSc+HMfKppybY3GP8wFgyECDsGraqfNuCbbML8e2DjaXRdJia3zcv5XKdUADEYo0s8OGWouN7lM= smithi171.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBBOgEufC093sZkDP4SZ4QwXTRVg3TWB3hAP05pEnLzCzkgNvfFQrEf22R9X6bBjKRSoeKd/zi5NHter+nJ2py2c= tasks: - install: extra_packages: deb: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror rpm: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror extra_system_packages: deb: - bison - flex - libelf-dev - libssl-dev - network-manager - iproute2 - util-linux - dump - indent - libaio-dev - libtool-bin - uuid-dev - xfslibs-dev - postgresql - postgresql-client - postgresql-common - postgresql-contrib rpm: - bison - flex - elfutils-libelf-devel - openssl-devel - NetworkManager - iproute - util-linux - libacl-devel - libaio-devel - libattr-devel - libtool - libuuid-devel - xfsdump - xfsprogs - xfsprogs-devel - libaio-devel - libtool - libuuid-devel - xfsprogs-devel - postgresql - postgresql-server - postgresql-contrib - ceph: create_rbd_pool: false - sequential: - mgrmodules - ceph-fuse: null - cephfs_test_runner: modules: - tasks.cephfs.test_cephfs_shell teuthology: fragments_dropped: [] meta: {} postmerge: - local function is_kupstream() return false end - local function is_kdistro() return false end - local function is_fuse() return true end - local function syntax_version() return '' end teuthology_branch: main teuthology_sha1: e691533f9cbb33d85b2187bba20d7102f098636d timestamp: 2024-03-28_07:14:07 tube: smithi user: pdonnell verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-03-28T13:53:55.181 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_e0e48ba6ee8c8b4c8f38d3b879d6eafbbb1efc15/qa; will attempt to use it 2024-03-28T13:53:55.182 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_e0e48ba6ee8c8b4c8f38d3b879d6eafbbb1efc15/qa/tasks 2024-03-28T13:53:55.182 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-03-28T13:53:55.184 INFO:teuthology.task.internal:Checking packages... 2024-03-28T13:53:55.207 INFO:teuthology.task.internal:Checking packages for os_type 'ubuntu', flavor 'default' and ceph hash 'e0e48ba6ee8c8b4c8f38d3b879d6eafbbb1efc15' 2024-03-28T13:53:55.208 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-03-28T13:53:55.208 INFO:teuthology.packaging:ref: None 2024-03-28T13:53:55.208 INFO:teuthology.packaging:tag: None 2024-03-28T13:53:55.208 INFO:teuthology.packaging:branch: wip-batrick-testing-20240327.230800 2024-03-28T13:53:55.208 INFO:teuthology.packaging:sha1: e0e48ba6ee8c8b4c8f38d3b879d6eafbbb1efc15 2024-03-28T13:53:55.208 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&ref=wip-batrick-testing-20240327.230800 2024-03-28T13:53:55.351 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-2499-ge0e48ba6-1jammy 2024-03-28T13:53:55.353 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-03-28T13:53:55.361 INFO:teuthology.task.internal:no buildpackages task found 2024-03-28T13:53:55.362 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-03-28T13:53:55.375 INFO:teuthology.task.internal:Saving configuration 2024-03-28T13:53:55.392 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-03-28T13:53:55.400 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-28T13:53:55.425 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi104.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/pdonnell-2024-03-28_07:14:07-fs-wip-batrick-testing-20240327.230800-distro-default-smithi/7627919', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '22.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-28 13:48:17.374019', 'locked_by': 'scheduled_pdonnell@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIP9F/HkNpF69yDhqYqtDCKQOGJJ4n4puozT3RXJXEwGn'} 2024-03-28T13:53:55.451 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi171.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/pdonnell-2024-03-28_07:14:07-fs-wip-batrick-testing-20240327.230800-distro-default-smithi/7627919', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '22.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-28 13:48:17.376125', 'locked_by': 'scheduled_pdonnell@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIBGvEbT4UIlVHsMi8oczMkqOit2LCOfOs+NyVbFSRFtJ'} 2024-03-28T13:53:55.451 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-03-28T13:53:55.459 INFO:teuthology.task.internal:roles: ubuntu@smithi104.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'mds.a', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0'] 2024-03-28T13:53:55.460 INFO:teuthology.task.internal:roles: ubuntu@smithi171.front.sepia.ceph.com - ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'osd.4', 'osd.5', 'osd.6', 'osd.7'] 2024-03-28T13:53:55.460 INFO:teuthology.run_tasks:Running task console_log... 2024-03-28T13:53:55.535 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f586df003a0>, signals=[15]) 2024-03-28T13:53:55.536 INFO:teuthology.run_tasks:Running task internal.connect... 2024-03-28T13:53:55.547 INFO:teuthology.task.internal:Opening connections... 2024-03-28T13:53:55.547 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi104.front.sepia.ceph.com 2024-03-28T13:53:55.549 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi104.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-28T13:53:55.629 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi171.front.sepia.ceph.com 2024-03-28T13:53:55.630 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi171.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-28T13:53:55.706 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-03-28T13:53:55.718 DEBUG:teuthology.orchestra.run.smithi104:> uname -m 2024-03-28T13:53:55.723 INFO:teuthology.orchestra.run.smithi104.stdout:x86_64 2024-03-28T13:53:55.724 DEBUG:teuthology.orchestra.run.smithi104:> cat /etc/os-release 2024-03-28T13:53:55.770 INFO:teuthology.orchestra.run.smithi104.stdout:PRETTY_NAME="Ubuntu 22.04.1 LTS" 2024-03-28T13:53:55.770 INFO:teuthology.orchestra.run.smithi104.stdout:NAME="Ubuntu" 2024-03-28T13:53:55.770 INFO:teuthology.orchestra.run.smithi104.stdout:VERSION_ID="22.04" 2024-03-28T13:53:55.770 INFO:teuthology.orchestra.run.smithi104.stdout:VERSION="22.04.1 LTS (Jammy Jellyfish)" 2024-03-28T13:53:55.770 INFO:teuthology.orchestra.run.smithi104.stdout:VERSION_CODENAME=jammy 2024-03-28T13:53:55.770 INFO:teuthology.orchestra.run.smithi104.stdout:ID=ubuntu 2024-03-28T13:53:55.770 INFO:teuthology.orchestra.run.smithi104.stdout:ID_LIKE=debian 2024-03-28T13:53:55.770 INFO:teuthology.orchestra.run.smithi104.stdout:HOME_URL="https://www.ubuntu.com/" 2024-03-28T13:53:55.770 INFO:teuthology.orchestra.run.smithi104.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2024-03-28T13:53:55.771 INFO:teuthology.orchestra.run.smithi104.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2024-03-28T13:53:55.771 INFO:teuthology.orchestra.run.smithi104.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2024-03-28T13:53:55.771 INFO:teuthology.orchestra.run.smithi104.stdout:UBUNTU_CODENAME=jammy 2024-03-28T13:53:55.771 INFO:teuthology.lock.ops:Updating smithi104.front.sepia.ceph.com on lock server 2024-03-28T13:53:55.813 DEBUG:teuthology.orchestra.run.smithi171:> uname -m 2024-03-28T13:53:55.818 INFO:teuthology.orchestra.run.smithi171.stdout:x86_64 2024-03-28T13:53:55.818 DEBUG:teuthology.orchestra.run.smithi171:> cat /etc/os-release 2024-03-28T13:53:55.863 INFO:teuthology.orchestra.run.smithi171.stdout:PRETTY_NAME="Ubuntu 22.04.1 LTS" 2024-03-28T13:53:55.863 INFO:teuthology.orchestra.run.smithi171.stdout:NAME="Ubuntu" 2024-03-28T13:53:55.863 INFO:teuthology.orchestra.run.smithi171.stdout:VERSION_ID="22.04" 2024-03-28T13:53:55.863 INFO:teuthology.orchestra.run.smithi171.stdout:VERSION="22.04.1 LTS (Jammy Jellyfish)" 2024-03-28T13:53:55.863 INFO:teuthology.orchestra.run.smithi171.stdout:VERSION_CODENAME=jammy 2024-03-28T13:53:55.863 INFO:teuthology.orchestra.run.smithi171.stdout:ID=ubuntu 2024-03-28T13:53:55.863 INFO:teuthology.orchestra.run.smithi171.stdout:ID_LIKE=debian 2024-03-28T13:53:55.863 INFO:teuthology.orchestra.run.smithi171.stdout:HOME_URL="https://www.ubuntu.com/" 2024-03-28T13:53:55.863 INFO:teuthology.orchestra.run.smithi171.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2024-03-28T13:53:55.863 INFO:teuthology.orchestra.run.smithi171.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2024-03-28T13:53:55.863 INFO:teuthology.orchestra.run.smithi171.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2024-03-28T13:53:55.863 INFO:teuthology.orchestra.run.smithi171.stdout:UBUNTU_CODENAME=jammy 2024-03-28T13:53:55.864 INFO:teuthology.lock.ops:Updating smithi171.front.sepia.ceph.com on lock server 2024-03-28T13:53:55.887 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-03-28T13:53:55.901 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-03-28T13:53:55.912 INFO:teuthology.task.internal:Checking for old test directory... 2024-03-28T13:53:55.912 DEBUG:teuthology.orchestra.run.smithi104:> test '!' -e /home/ubuntu/cephtest 2024-03-28T13:53:55.914 DEBUG:teuthology.orchestra.run.smithi171:> test '!' -e /home/ubuntu/cephtest 2024-03-28T13:53:55.919 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-03-28T13:53:55.929 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-03-28T13:53:55.929 DEBUG:teuthology.orchestra.run.smithi104:> test -z $(ls -A /var/lib/ceph) 2024-03-28T13:53:55.963 DEBUG:teuthology.orchestra.run.smithi171:> test -z $(ls -A /var/lib/ceph) 2024-03-28T13:53:55.973 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-03-28T13:53:56.036 INFO:teuthology.run_tasks:Running task kernel... 2024-03-28T13:53:56.055 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2024-03-28T13:53:56.055 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}, 'mon.b': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2024-03-28T13:53:56.055 DEBUG:teuthology.orchestra.run.smithi104:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-28T13:53:56.056 DEBUG:teuthology.orchestra.run.smithi171:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-28T13:53:56.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-28T13:53:56.060 DEBUG:teuthology.orchestra.run.smithi104:> uname -r 2024-03-28T13:53:56.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-28T13:53:56.061 DEBUG:teuthology.orchestra.run.smithi171:> uname -r 2024-03-28T13:53:56.106 INFO:teuthology.orchestra.run.smithi104.stdout:5.15.0-56-generic 2024-03-28T13:53:56.106 INFO:teuthology.task.kernel:Running kernel on smithi104: 5.15.0-56-generic 2024-03-28T13:53:56.106 DEBUG:teuthology.orchestra.run.smithi104:> sudo apt-get clean 2024-03-28T13:53:56.107 INFO:teuthology.orchestra.run.smithi171.stdout:5.15.0-56-generic 2024-03-28T13:53:56.108 INFO:teuthology.task.kernel:Running kernel on smithi171: 5.15.0-56-generic 2024-03-28T13:53:56.108 DEBUG:teuthology.orchestra.run.smithi171:> sudo apt-get clean 2024-03-28T13:53:56.231 DEBUG:teuthology.orchestra.run.smithi104:> sudo apt-get update 2024-03-28T13:53:56.258 DEBUG:teuthology.orchestra.run.smithi171:> sudo apt-get update 2024-03-28T13:53:56.381 INFO:teuthology.orchestra.run.smithi104.stdout:Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-03-28T13:53:56.403 INFO:teuthology.orchestra.run.smithi171.stdout:Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-03-28T13:53:56.406 INFO:teuthology.orchestra.run.smithi171.stdout:Get:2 http://security.ubuntu.com/ubuntu jammy-security InRelease [110 kB] 2024-03-28T13:53:56.408 INFO:teuthology.orchestra.run.smithi104.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy-updates InRelease [119 kB] 2024-03-28T13:53:56.410 INFO:teuthology.orchestra.run.smithi171.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease [119 kB] 2024-03-28T13:53:56.490 INFO:teuthology.orchestra.run.smithi104.stdout:Get:3 http://security.ubuntu.com/ubuntu jammy-security InRelease [110 kB] 2024-03-28T13:53:56.494 INFO:teuthology.orchestra.run.smithi104.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-backports InRelease [109 kB] 2024-03-28T13:53:56.511 INFO:teuthology.orchestra.run.smithi171.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-backports InRelease [109 kB] 2024-03-28T13:53:56.603 INFO:teuthology.orchestra.run.smithi104.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 Packages [1,518 kB] 2024-03-28T13:53:56.644 INFO:teuthology.orchestra.run.smithi171.stdout:Get:5 http://security.ubuntu.com/ubuntu jammy-security/main i386 Packages [435 kB] 2024-03-28T13:53:56.714 INFO:teuthology.orchestra.run.smithi104.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/main i386 Packages [602 kB] 2024-03-28T13:53:56.721 INFO:teuthology.orchestra.run.smithi104.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/main Translation-en [293 kB] 2024-03-28T13:53:56.722 INFO:teuthology.orchestra.run.smithi171.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 Packages [1,518 kB] 2024-03-28T13:53:56.724 INFO:teuthology.orchestra.run.smithi104.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 c-n-f Metadata [16.1 kB] 2024-03-28T13:53:56.725 INFO:teuthology.orchestra.run.smithi104.stdout:Get:9 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 Packages [1,644 kB] 2024-03-28T13:53:56.730 INFO:teuthology.orchestra.run.smithi171.stdout:Get:7 http://security.ubuntu.com/ubuntu jammy-security/main amd64 Packages [1,298 kB] 2024-03-28T13:53:56.759 INFO:teuthology.orchestra.run.smithi104.stdout:Get:10 http://archive.ubuntu.com/ubuntu jammy-updates/restricted i386 Packages [35.3 kB] 2024-03-28T13:53:56.759 INFO:teuthology.orchestra.run.smithi104.stdout:Get:11 http://archive.ubuntu.com/ubuntu jammy-updates/restricted Translation-en [274 kB] 2024-03-28T13:53:56.763 INFO:teuthology.orchestra.run.smithi104.stdout:Get:12 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 c-n-f Metadata [520 B] 2024-03-28T13:53:56.764 INFO:teuthology.orchestra.run.smithi104.stdout:Get:13 http://archive.ubuntu.com/ubuntu jammy-updates/universe i386 Packages [698 kB] 2024-03-28T13:53:56.773 INFO:teuthology.orchestra.run.smithi104.stdout:Get:14 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 Packages [1,060 kB] 2024-03-28T13:53:56.807 INFO:teuthology.orchestra.run.smithi104.stdout:Get:15 http://archive.ubuntu.com/ubuntu jammy-updates/universe Translation-en [241 kB] 2024-03-28T13:53:56.809 INFO:teuthology.orchestra.run.smithi104.stdout:Get:16 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 c-n-f Metadata [22.1 kB] 2024-03-28T13:53:56.809 INFO:teuthology.orchestra.run.smithi104.stdout:Get:17 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 Packages [49.6 kB] 2024-03-28T13:53:56.810 INFO:teuthology.orchestra.run.smithi104.stdout:Get:18 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse i386 Packages [6,028 B] 2024-03-28T13:53:56.810 INFO:teuthology.orchestra.run.smithi104.stdout:Get:19 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse Translation-en [12.0 kB] 2024-03-28T13:53:56.810 INFO:teuthology.orchestra.run.smithi104.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 c-n-f Metadata [472 B] 2024-03-28T13:53:56.812 INFO:teuthology.orchestra.run.smithi171.stdout:Get:8 http://security.ubuntu.com/ubuntu jammy-security/main Translation-en [232 kB] 2024-03-28T13:53:56.820 INFO:teuthology.orchestra.run.smithi171.stdout:Get:9 http://security.ubuntu.com/ubuntu jammy-security/main amd64 c-n-f Metadata [11.4 kB] 2024-03-28T13:53:56.820 INFO:teuthology.orchestra.run.smithi171.stdout:Get:10 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 Packages [1,605 kB] 2024-03-28T13:53:56.838 INFO:teuthology.orchestra.run.smithi104.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy-backports/main i386 Packages [59.2 kB] 2024-03-28T13:53:56.839 INFO:teuthology.orchestra.run.smithi104.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 Packages [67.1 kB] 2024-03-28T13:53:56.839 INFO:teuthology.orchestra.run.smithi104.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy-backports/main Translation-en [11.0 kB] 2024-03-28T13:53:56.865 INFO:teuthology.orchestra.run.smithi171.stdout:Get:11 http://security.ubuntu.com/ubuntu jammy-security/restricted i386 Packages [33.7 kB] 2024-03-28T13:53:56.866 INFO:teuthology.orchestra.run.smithi171.stdout:Get:12 http://security.ubuntu.com/ubuntu jammy-security/restricted Translation-en [270 kB] 2024-03-28T13:53:56.870 INFO:teuthology.orchestra.run.smithi171.stdout:Get:13 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 c-n-f Metadata [520 B] 2024-03-28T13:53:56.871 INFO:teuthology.orchestra.run.smithi171.stdout:Get:14 http://security.ubuntu.com/ubuntu jammy-security/universe i386 Packages [599 kB] 2024-03-28T13:53:56.875 INFO:teuthology.orchestra.run.smithi104.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 c-n-f Metadata [388 B] 2024-03-28T13:53:56.875 INFO:teuthology.orchestra.run.smithi104.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy-backports/universe i386 Packages [17.2 kB] 2024-03-28T13:53:56.876 INFO:teuthology.orchestra.run.smithi104.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 Packages [28.4 kB] 2024-03-28T13:53:56.876 INFO:teuthology.orchestra.run.smithi104.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy-backports/universe Translation-en [16.2 kB] 2024-03-28T13:53:56.876 INFO:teuthology.orchestra.run.smithi104.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 c-n-f Metadata [644 B] 2024-03-28T13:53:56.885 INFO:teuthology.orchestra.run.smithi171.stdout:Get:15 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 Packages [852 kB] 2024-03-28T13:53:56.897 INFO:teuthology.orchestra.run.smithi171.stdout:Get:16 http://security.ubuntu.com/ubuntu jammy-security/universe Translation-en [163 kB] 2024-03-28T13:53:56.900 INFO:teuthology.orchestra.run.smithi171.stdout:Get:17 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 c-n-f Metadata [16.8 kB] 2024-03-28T13:53:56.900 INFO:teuthology.orchestra.run.smithi171.stdout:Get:18 http://security.ubuntu.com/ubuntu jammy-security/multiverse i386 Packages [1,356 B] 2024-03-28T13:53:56.901 INFO:teuthology.orchestra.run.smithi171.stdout:Get:19 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 Packages [37.1 kB] 2024-03-28T13:53:56.914 INFO:teuthology.orchestra.run.smithi104.stdout:Get:29 http://security.ubuntu.com/ubuntu jammy-security/main amd64 Packages [1,298 kB] 2024-03-28T13:53:56.929 INFO:teuthology.orchestra.run.smithi171.stdout:Get:20 http://security.ubuntu.com/ubuntu jammy-security/multiverse Translation-en [7,476 B] 2024-03-28T13:53:56.930 INFO:teuthology.orchestra.run.smithi171.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy-updates/main i386 Packages [602 kB] 2024-03-28T13:53:56.930 INFO:teuthology.orchestra.run.smithi171.stdout:Get:22 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 c-n-f Metadata [260 B] 2024-03-28T13:53:56.940 INFO:teuthology.orchestra.run.smithi171.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy-updates/main Translation-en [293 kB] 2024-03-28T13:53:56.945 INFO:teuthology.orchestra.run.smithi171.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 c-n-f Metadata [16.1 kB] 2024-03-28T13:53:56.946 INFO:teuthology.orchestra.run.smithi171.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy-updates/restricted i386 Packages [35.3 kB] 2024-03-28T13:53:56.947 INFO:teuthology.orchestra.run.smithi171.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 Packages [1,644 kB] 2024-03-28T13:53:56.982 INFO:teuthology.orchestra.run.smithi171.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy-updates/restricted Translation-en [274 kB] 2024-03-28T13:53:56.987 INFO:teuthology.orchestra.run.smithi171.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 c-n-f Metadata [520 B] 2024-03-28T13:53:56.987 INFO:teuthology.orchestra.run.smithi171.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 Packages [1,060 kB] 2024-03-28T13:53:57.006 INFO:teuthology.orchestra.run.smithi171.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy-updates/universe i386 Packages [698 kB] 2024-03-28T13:53:57.017 INFO:teuthology.orchestra.run.smithi171.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy-updates/universe Translation-en [241 kB] 2024-03-28T13:53:57.020 INFO:teuthology.orchestra.run.smithi171.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 c-n-f Metadata [22.1 kB] 2024-03-28T13:53:57.026 INFO:teuthology.orchestra.run.smithi171.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 Packages [49.6 kB] 2024-03-28T13:53:57.027 INFO:teuthology.orchestra.run.smithi171.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse i386 Packages [6,028 B] 2024-03-28T13:53:57.027 INFO:teuthology.orchestra.run.smithi171.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse Translation-en [12.0 kB] 2024-03-28T13:53:57.058 INFO:teuthology.orchestra.run.smithi171.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 c-n-f Metadata [472 B] 2024-03-28T13:53:57.058 INFO:teuthology.orchestra.run.smithi171.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 Packages [67.1 kB] 2024-03-28T13:53:57.059 INFO:teuthology.orchestra.run.smithi171.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy-backports/main i386 Packages [59.2 kB] 2024-03-28T13:53:57.061 INFO:teuthology.orchestra.run.smithi171.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy-backports/main Translation-en [11.0 kB] 2024-03-28T13:53:57.061 INFO:teuthology.orchestra.run.smithi171.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 c-n-f Metadata [388 B] 2024-03-28T13:53:57.063 INFO:teuthology.orchestra.run.smithi171.stdout:Get:41 http://archive.ubuntu.com/ubuntu jammy-backports/universe i386 Packages [17.2 kB] 2024-03-28T13:53:57.064 INFO:teuthology.orchestra.run.smithi171.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 Packages [28.4 kB] 2024-03-28T13:53:57.105 INFO:teuthology.orchestra.run.smithi171.stdout:Get:43 http://archive.ubuntu.com/ubuntu jammy-backports/universe Translation-en [16.2 kB] 2024-03-28T13:53:57.105 INFO:teuthology.orchestra.run.smithi171.stdout:Get:44 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 c-n-f Metadata [644 B] 2024-03-28T13:53:57.318 INFO:teuthology.orchestra.run.smithi104.stdout:Get:30 http://security.ubuntu.com/ubuntu jammy-security/main i386 Packages [435 kB] 2024-03-28T13:53:57.336 INFO:teuthology.orchestra.run.smithi104.stdout:Get:31 http://security.ubuntu.com/ubuntu jammy-security/main Translation-en [232 kB] 2024-03-28T13:53:57.345 INFO:teuthology.orchestra.run.smithi104.stdout:Get:32 http://security.ubuntu.com/ubuntu jammy-security/main amd64 c-n-f Metadata [11.4 kB] 2024-03-28T13:53:57.346 INFO:teuthology.orchestra.run.smithi104.stdout:Get:33 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 Packages [1,605 kB] 2024-03-28T13:53:57.438 INFO:teuthology.orchestra.run.smithi104.stdout:Get:34 http://security.ubuntu.com/ubuntu jammy-security/restricted i386 Packages [33.7 kB] 2024-03-28T13:53:57.439 INFO:teuthology.orchestra.run.smithi104.stdout:Get:35 http://security.ubuntu.com/ubuntu jammy-security/restricted Translation-en [270 kB] 2024-03-28T13:53:57.450 INFO:teuthology.orchestra.run.smithi104.stdout:Get:36 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 c-n-f Metadata [520 B] 2024-03-28T13:53:57.450 INFO:teuthology.orchestra.run.smithi104.stdout:Get:37 http://security.ubuntu.com/ubuntu jammy-security/universe i386 Packages [599 kB] 2024-03-28T13:53:57.472 INFO:teuthology.orchestra.run.smithi104.stdout:Get:38 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 Packages [852 kB] 2024-03-28T13:53:57.658 INFO:teuthology.orchestra.run.smithi104.stdout:Get:39 http://security.ubuntu.com/ubuntu jammy-security/universe Translation-en [163 kB] 2024-03-28T13:53:58.065 INFO:teuthology.orchestra.run.smithi104.stdout:Get:40 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 c-n-f Metadata [16.8 kB] 2024-03-28T13:53:58.071 INFO:teuthology.orchestra.run.smithi104.stdout:Get:41 http://security.ubuntu.com/ubuntu jammy-security/multiverse i386 Packages [1,356 B] 2024-03-28T13:53:58.072 INFO:teuthology.orchestra.run.smithi104.stdout:Get:42 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 Packages [37.1 kB] 2024-03-28T13:53:58.085 INFO:teuthology.orchestra.run.smithi104.stdout:Get:43 http://security.ubuntu.com/ubuntu jammy-security/multiverse Translation-en [7,476 B] 2024-03-28T13:53:58.088 INFO:teuthology.orchestra.run.smithi104.stdout:Get:44 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 c-n-f Metadata [260 B] 2024-03-28T13:54:02.303 INFO:teuthology.orchestra.run.smithi104.stdout:Fetched 12.6 MB in 2s (7,029 kB/s) 2024-03-28T13:54:02.460 INFO:teuthology.orchestra.run.smithi171.stdout:Fetched 12.6 MB in 2s (6,924 kB/s) 2024-03-28T13:54:03.608 INFO:teuthology.orchestra.run.smithi104.stdout:Reading package lists... 2024-03-28T13:54:03.625 DEBUG:teuthology.orchestra.run.smithi104:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-03-28T13:54:03.687 INFO:teuthology.orchestra.run.smithi104.stdout:Reading package lists... 2024-03-28T13:54:03.885 INFO:teuthology.orchestra.run.smithi104.stdout:Building dependency tree... 2024-03-28T13:54:03.886 INFO:teuthology.orchestra.run.smithi104.stdout:Reading state information... 2024-03-28T13:54:03.888 INFO:teuthology.orchestra.run.smithi171.stdout:Reading package lists... 2024-03-28T13:54:03.906 DEBUG:teuthology.orchestra.run.smithi171:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-03-28T13:54:03.967 INFO:teuthology.orchestra.run.smithi171.stdout:Reading package lists... 2024-03-28T13:54:04.061 INFO:teuthology.orchestra.run.smithi104.stdout:The following packages were automatically installed and are no longer required: 2024-03-28T13:54:04.061 INFO:teuthology.orchestra.run.smithi104.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-03-28T13:54:04.061 INFO:teuthology.orchestra.run.smithi104.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-03-28T13:54:04.061 INFO:teuthology.orchestra.run.smithi104.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-03-28T13:54:04.062 INFO:teuthology.orchestra.run.smithi104.stdout: python2 python2-dev python2-minimal python2.7 python2.7-dev 2024-03-28T13:54:04.062 INFO:teuthology.orchestra.run.smithi104.stdout: python2.7-minimal 2024-03-28T13:54:04.062 INFO:teuthology.orchestra.run.smithi104.stdout:Use 'sudo apt autoremove' to remove them. 2024-03-28T13:54:04.063 INFO:teuthology.orchestra.run.smithi104.stdout:The following additional packages will be installed: 2024-03-28T13:54:04.063 INFO:teuthology.orchestra.run.smithi104.stdout: linux-generic linux-headers-5.15.0-101 linux-headers-5.15.0-101-generic 2024-03-28T13:54:04.063 INFO:teuthology.orchestra.run.smithi104.stdout: linux-headers-generic linux-image-5.15.0-101-generic 2024-03-28T13:54:04.063 INFO:teuthology.orchestra.run.smithi104.stdout: linux-modules-5.15.0-101-generic linux-modules-extra-5.15.0-101-generic 2024-03-28T13:54:04.064 INFO:teuthology.orchestra.run.smithi104.stdout:Suggested packages: 2024-03-28T13:54:04.065 INFO:teuthology.orchestra.run.smithi104.stdout: fdutils linux-doc | linux-source-5.15.0 linux-tools 2024-03-28T13:54:04.065 INFO:teuthology.orchestra.run.smithi104.stdout:Recommended packages: 2024-03-28T13:54:04.065 INFO:teuthology.orchestra.run.smithi104.stdout: thermald 2024-03-28T13:54:04.108 INFO:teuthology.orchestra.run.smithi104.stdout:The following NEW packages will be installed: 2024-03-28T13:54:04.108 INFO:teuthology.orchestra.run.smithi104.stdout: linux-headers-5.15.0-101 linux-headers-5.15.0-101-generic 2024-03-28T13:54:04.108 INFO:teuthology.orchestra.run.smithi104.stdout: linux-image-5.15.0-101-generic linux-modules-5.15.0-101-generic 2024-03-28T13:54:04.109 INFO:teuthology.orchestra.run.smithi104.stdout: linux-modules-extra-5.15.0-101-generic 2024-03-28T13:54:04.109 INFO:teuthology.orchestra.run.smithi104.stdout:The following packages will be upgraded: 2024-03-28T13:54:04.110 INFO:teuthology.orchestra.run.smithi104.stdout: linux-generic linux-headers-generic linux-image-generic 2024-03-28T13:54:04.166 INFO:teuthology.orchestra.run.smithi104.stdout:3 upgraded, 5 newly installed, 0 to remove and 311 not upgraded. 2024-03-28T13:54:04.166 INFO:teuthology.orchestra.run.smithi104.stdout:Need to get 113 MB of archives. 2024-03-28T13:54:04.166 INFO:teuthology.orchestra.run.smithi104.stdout:After this operation, 582 MB of additional disk space will be used. 2024-03-28T13:54:04.166 INFO:teuthology.orchestra.run.smithi104.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-5.15.0-101-generic amd64 5.15.0-101.111 [22.6 MB] 2024-03-28T13:54:04.173 INFO:teuthology.orchestra.run.smithi171.stdout:Building dependency tree... 2024-03-28T13:54:04.174 INFO:teuthology.orchestra.run.smithi171.stdout:Reading state information... 2024-03-28T13:54:04.331 INFO:teuthology.orchestra.run.smithi171.stdout:The following packages were automatically installed and are no longer required: 2024-03-28T13:54:04.331 INFO:teuthology.orchestra.run.smithi171.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-03-28T13:54:04.331 INFO:teuthology.orchestra.run.smithi171.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-03-28T13:54:04.332 INFO:teuthology.orchestra.run.smithi171.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-03-28T13:54:04.332 INFO:teuthology.orchestra.run.smithi171.stdout: python2 python2-dev python2-minimal python2.7 python2.7-dev 2024-03-28T13:54:04.332 INFO:teuthology.orchestra.run.smithi171.stdout: python2.7-minimal 2024-03-28T13:54:04.332 INFO:teuthology.orchestra.run.smithi171.stdout:Use 'sudo apt autoremove' to remove them. 2024-03-28T13:54:04.333 INFO:teuthology.orchestra.run.smithi171.stdout:The following additional packages will be installed: 2024-03-28T13:54:04.333 INFO:teuthology.orchestra.run.smithi171.stdout: linux-generic linux-headers-5.15.0-101 linux-headers-5.15.0-101-generic 2024-03-28T13:54:04.333 INFO:teuthology.orchestra.run.smithi171.stdout: linux-headers-generic linux-image-5.15.0-101-generic 2024-03-28T13:54:04.333 INFO:teuthology.orchestra.run.smithi171.stdout: linux-modules-5.15.0-101-generic linux-modules-extra-5.15.0-101-generic 2024-03-28T13:54:04.334 INFO:teuthology.orchestra.run.smithi171.stdout:Suggested packages: 2024-03-28T13:54:04.335 INFO:teuthology.orchestra.run.smithi171.stdout: fdutils linux-doc | linux-source-5.15.0 linux-tools 2024-03-28T13:54:04.335 INFO:teuthology.orchestra.run.smithi171.stdout:Recommended packages: 2024-03-28T13:54:04.335 INFO:teuthology.orchestra.run.smithi171.stdout: thermald 2024-03-28T13:54:04.380 INFO:teuthology.orchestra.run.smithi171.stdout:The following NEW packages will be installed: 2024-03-28T13:54:04.380 INFO:teuthology.orchestra.run.smithi171.stdout: linux-headers-5.15.0-101 linux-headers-5.15.0-101-generic 2024-03-28T13:54:04.380 INFO:teuthology.orchestra.run.smithi171.stdout: linux-image-5.15.0-101-generic linux-modules-5.15.0-101-generic 2024-03-28T13:54:04.381 INFO:teuthology.orchestra.run.smithi171.stdout: linux-modules-extra-5.15.0-101-generic 2024-03-28T13:54:04.382 INFO:teuthology.orchestra.run.smithi171.stdout:The following packages will be upgraded: 2024-03-28T13:54:04.383 INFO:teuthology.orchestra.run.smithi171.stdout: linux-generic linux-headers-generic linux-image-generic 2024-03-28T13:54:04.561 INFO:teuthology.orchestra.run.smithi171.stdout:3 upgraded, 5 newly installed, 0 to remove and 311 not upgraded. 2024-03-28T13:54:04.562 INFO:teuthology.orchestra.run.smithi171.stdout:Need to get 113 MB of archives. 2024-03-28T13:54:04.562 INFO:teuthology.orchestra.run.smithi171.stdout:After this operation, 582 MB of additional disk space will be used. 2024-03-28T13:54:04.562 INFO:teuthology.orchestra.run.smithi171.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-5.15.0-101-generic amd64 5.15.0-101.111 [22.6 MB] 2024-03-28T13:54:04.927 INFO:teuthology.orchestra.run.smithi104.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-5.15.0-101-generic amd64 5.15.0-101.111 [11.5 MB] 2024-03-28T13:54:05.125 INFO:teuthology.orchestra.run.smithi104.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-extra-5.15.0-101-generic amd64 5.15.0-101.111 [63.9 MB] 2024-03-28T13:54:05.975 INFO:teuthology.orchestra.run.smithi171.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-5.15.0-101-generic amd64 5.15.0-101.111 [11.5 MB] 2024-03-28T13:54:06.326 INFO:teuthology.orchestra.run.smithi171.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-extra-5.15.0-101-generic amd64 5.15.0-101.111 [63.9 MB] 2024-03-28T13:54:06.401 INFO:teuthology.orchestra.run.smithi104.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-generic amd64 5.15.0.101.98 [1,698 B] 2024-03-28T13:54:06.407 INFO:teuthology.orchestra.run.smithi104.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-generic amd64 5.15.0.101.98 [2,486 B] 2024-03-28T13:54:06.408 INFO:teuthology.orchestra.run.smithi104.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-101 all 5.15.0-101.111 [12.3 MB] 2024-03-28T13:54:06.735 INFO:teuthology.orchestra.run.smithi104.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-101-generic amd64 5.15.0-101.111 [2,876 kB] 2024-03-28T13:54:06.774 INFO:teuthology.orchestra.run.smithi104.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-generic amd64 5.15.0.101.98 [2,332 B] 2024-03-28T13:54:07.145 INFO:teuthology.orchestra.run.smithi104.stdout:Fetched 113 MB in 3s (42.6 MB/s) 2024-03-28T13:54:07.304 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package linux-modules-5.15.0-101-generic. 2024-03-28T13:54:08.599 INFO:teuthology.orchestra.run.smithi171.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-generic amd64 5.15.0.101.98 [1,698 B] 2024-03-28T13:54:08.599 INFO:teuthology.orchestra.run.smithi171.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-generic amd64 5.15.0.101.98 [2,486 B] 2024-03-28T13:54:08.599 INFO:teuthology.orchestra.run.smithi171.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-101 all 5.15.0-101.111 [12.3 MB] 2024-03-28T13:54:09.207 INFO:teuthology.orchestra.run.smithi104.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 129542 files and directories currently installed.) 2024-03-28T13:54:09.212 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../0-linux-modules-5.15.0-101-generic_5.15.0-101.111_amd64.deb ... 2024-03-28T13:54:09.228 INFO:teuthology.orchestra.run.smithi171.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-101-generic amd64 5.15.0-101.111 [2,876 kB] 2024-03-28T13:54:09.293 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking linux-modules-5.15.0-101-generic (5.15.0-101.111) ... 2024-03-28T13:54:09.336 INFO:teuthology.orchestra.run.smithi171.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-generic amd64 5.15.0.101.98 [2,332 B] 2024-03-28T13:54:09.717 INFO:teuthology.orchestra.run.smithi171.stdout:Fetched 113 MB in 5s (22.9 MB/s) 2024-03-28T13:54:09.891 INFO:teuthology.orchestra.run.smithi171.stdout:Selecting previously unselected package linux-modules-5.15.0-101-generic. 2024-03-28T13:54:11.385 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package linux-image-5.15.0-101-generic. 2024-03-28T13:54:11.406 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../1-linux-image-5.15.0-101-generic_5.15.0-101.111_amd64.deb ... 2024-03-28T13:54:11.498 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking linux-image-5.15.0-101-generic (5.15.0-101.111) ... 2024-03-28T13:54:11.913 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package linux-modules-extra-5.15.0-101-generic. 2024-03-28T13:54:11.934 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../2-linux-modules-extra-5.15.0-101-generic_5.15.0-101.111_amd64.deb ... 2024-03-28T13:54:11.952 INFO:teuthology.orchestra.run.smithi171.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 129542 files and directories currently installed.) 2024-03-28T13:54:11.957 INFO:teuthology.orchestra.run.smithi171.stdout:Preparing to unpack .../0-linux-modules-5.15.0-101-generic_5.15.0-101.111_amd64.deb ... 2024-03-28T13:54:11.971 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking linux-modules-extra-5.15.0-101-generic (5.15.0-101.111) ... 2024-03-28T13:54:12.042 INFO:teuthology.orchestra.run.smithi171.stdout:Unpacking linux-modules-5.15.0-101-generic (5.15.0-101.111) ... 2024-03-28T13:54:14.542 INFO:teuthology.orchestra.run.smithi171.stdout:Selecting previously unselected package linux-image-5.15.0-101-generic. 2024-03-28T13:54:14.563 INFO:teuthology.orchestra.run.smithi171.stdout:Preparing to unpack .../1-linux-image-5.15.0-101-generic_5.15.0-101.111_amd64.deb ... 2024-03-28T13:54:14.644 INFO:teuthology.orchestra.run.smithi171.stdout:Unpacking linux-image-5.15.0-101-generic (5.15.0-101.111) ... 2024-03-28T13:54:15.112 INFO:teuthology.orchestra.run.smithi171.stdout:Selecting previously unselected package linux-modules-extra-5.15.0-101-generic. 2024-03-28T13:54:15.133 INFO:teuthology.orchestra.run.smithi171.stdout:Preparing to unpack .../2-linux-modules-extra-5.15.0-101-generic_5.15.0-101.111_amd64.deb ... 2024-03-28T13:54:15.170 INFO:teuthology.orchestra.run.smithi171.stdout:Unpacking linux-modules-extra-5.15.0-101-generic (5.15.0-101.111) ... 2024-03-28T13:54:18.079 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../3-linux-generic_5.15.0.101.98_amd64.deb ... 2024-03-28T13:54:18.207 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking linux-generic (5.15.0.101.98) over (5.15.0.56.54) ... 2024-03-28T13:54:18.531 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../4-linux-image-generic_5.15.0.101.98_amd64.deb ... 2024-03-28T13:54:18.660 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking linux-image-generic (5.15.0.101.98) over (5.15.0.56.54) ... 2024-03-28T13:54:18.955 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package linux-headers-5.15.0-101. 2024-03-28T13:54:18.968 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../5-linux-headers-5.15.0-101_5.15.0-101.111_all.deb ... 2024-03-28T13:54:19.013 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking linux-headers-5.15.0-101 (5.15.0-101.111) ... 2024-03-28T13:54:21.588 INFO:teuthology.orchestra.run.smithi171.stdout:Preparing to unpack .../3-linux-generic_5.15.0.101.98_amd64.deb ... 2024-03-28T13:54:21.713 INFO:teuthology.orchestra.run.smithi171.stdout:Unpacking linux-generic (5.15.0.101.98) over (5.15.0.56.54) ... 2024-03-28T13:54:22.063 INFO:teuthology.orchestra.run.smithi171.stdout:Preparing to unpack .../4-linux-image-generic_5.15.0.101.98_amd64.deb ... 2024-03-28T13:54:22.167 INFO:teuthology.orchestra.run.smithi171.stdout:Unpacking linux-image-generic (5.15.0.101.98) over (5.15.0.56.54) ... 2024-03-28T13:54:22.454 INFO:teuthology.orchestra.run.smithi171.stdout:Selecting previously unselected package linux-headers-5.15.0-101. 2024-03-28T13:54:22.468 INFO:teuthology.orchestra.run.smithi171.stdout:Preparing to unpack .../5-linux-headers-5.15.0-101_5.15.0-101.111_all.deb ... 2024-03-28T13:54:22.521 INFO:teuthology.orchestra.run.smithi171.stdout:Unpacking linux-headers-5.15.0-101 (5.15.0-101.111) ... 2024-03-28T13:54:25.051 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package linux-headers-5.15.0-101-generic. 2024-03-28T13:54:25.087 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../6-linux-headers-5.15.0-101-generic_5.15.0-101.111_amd64.deb ... 2024-03-28T13:54:25.126 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking linux-headers-5.15.0-101-generic (5.15.0-101.111) ... 2024-03-28T13:54:27.669 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../7-linux-headers-generic_5.15.0.101.98_amd64.deb ... 2024-03-28T13:54:27.793 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking linux-headers-generic (5.15.0.101.98) over (5.15.0.56.54) ... 2024-03-28T13:54:27.828 INFO:teuthology.orchestra.run.smithi171.stdout:Selecting previously unselected package linux-headers-5.15.0-101-generic. 2024-03-28T13:54:27.865 INFO:teuthology.orchestra.run.smithi171.stdout:Preparing to unpack .../6-linux-headers-5.15.0-101-generic_5.15.0-101.111_amd64.deb ... 2024-03-28T13:54:27.902 INFO:teuthology.orchestra.run.smithi171.stdout:Unpacking linux-headers-5.15.0-101-generic (5.15.0-101.111) ... 2024-03-28T13:54:28.126 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up linux-headers-5.15.0-101 (5.15.0-101.111) ... 2024-03-28T13:54:28.256 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up linux-headers-5.15.0-101-generic (5.15.0-101.111) ... 2024-03-28T13:54:28.382 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up linux-headers-generic (5.15.0.101.98) ... 2024-03-28T13:54:28.511 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up linux-image-5.15.0-101-generic (5.15.0-101.111) ... 2024-03-28T13:54:29.843 INFO:teuthology.orchestra.run.smithi104.stdout:I: /vmlinuz.old is now a symlink to boot/vmlinuz-5.15.0-56-generic 2024-03-28T13:54:29.843 INFO:teuthology.orchestra.run.smithi104.stdout:I: /initrd.img.old is now a symlink to boot/initrd.img-5.15.0-56-generic 2024-03-28T13:54:29.843 INFO:teuthology.orchestra.run.smithi104.stdout:I: /vmlinuz is now a symlink to boot/vmlinuz-5.15.0-101-generic 2024-03-28T13:54:29.843 INFO:teuthology.orchestra.run.smithi104.stdout:I: /initrd.img is now a symlink to boot/initrd.img-5.15.0-101-generic 2024-03-28T13:54:30.014 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up linux-modules-extra-5.15.0-101-generic (5.15.0-101.111) ... 2024-03-28T13:54:30.907 INFO:teuthology.orchestra.run.smithi171.stdout:Preparing to unpack .../7-linux-headers-generic_5.15.0.101.98_amd64.deb ... 2024-03-28T13:54:31.046 INFO:teuthology.orchestra.run.smithi171.stdout:Unpacking linux-headers-generic (5.15.0.101.98) over (5.15.0.56.54) ... 2024-03-28T13:54:31.389 INFO:teuthology.orchestra.run.smithi171.stdout:Setting up linux-headers-5.15.0-101 (5.15.0-101.111) ... 2024-03-28T13:54:31.415 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up linux-modules-5.15.0-101-generic (5.15.0-101.111) ... 2024-03-28T13:54:31.510 INFO:teuthology.orchestra.run.smithi171.stdout:Setting up linux-headers-5.15.0-101-generic (5.15.0-101.111) ... 2024-03-28T13:54:31.644 INFO:teuthology.orchestra.run.smithi171.stdout:Setting up linux-headers-generic (5.15.0.101.98) ... 2024-03-28T13:54:31.773 INFO:teuthology.orchestra.run.smithi171.stdout:Setting up linux-image-5.15.0-101-generic (5.15.0-101.111) ... 2024-03-28T13:54:32.891 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up linux-image-generic (5.15.0.101.98) ... 2024-03-28T13:54:33.025 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up linux-generic (5.15.0.101.98) ... 2024-03-28T13:54:33.108 INFO:teuthology.orchestra.run.smithi171.stdout:I: /vmlinuz.old is now a symlink to boot/vmlinuz-5.15.0-56-generic 2024-03-28T13:54:33.108 INFO:teuthology.orchestra.run.smithi171.stdout:I: /initrd.img.old is now a symlink to boot/initrd.img-5.15.0-56-generic 2024-03-28T13:54:33.108 INFO:teuthology.orchestra.run.smithi171.stdout:I: /vmlinuz is now a symlink to boot/vmlinuz-5.15.0-101-generic 2024-03-28T13:54:33.108 INFO:teuthology.orchestra.run.smithi171.stdout:I: /initrd.img is now a symlink to boot/initrd.img-5.15.0-101-generic 2024-03-28T13:54:33.152 INFO:teuthology.orchestra.run.smithi104.stdout:Processing triggers for linux-image-5.15.0-101-generic (5.15.0-101.111) ... 2024-03-28T13:54:33.213 INFO:teuthology.orchestra.run.smithi104.stdout:/etc/kernel/postinst.d/initramfs-tools: 2024-03-28T13:54:33.213 INFO:teuthology.orchestra.run.smithi104.stdout:update-initramfs: Generating /boot/initrd.img-5.15.0-101-generic 2024-03-28T13:54:33.267 INFO:teuthology.orchestra.run.smithi171.stdout:Setting up linux-modules-extra-5.15.0-101-generic (5.15.0-101.111) ... 2024-03-28T13:54:34.676 INFO:teuthology.orchestra.run.smithi171.stdout:Setting up linux-modules-5.15.0-101-generic (5.15.0-101.111) ... 2024-03-28T13:54:36.102 INFO:teuthology.orchestra.run.smithi171.stdout:Setting up linux-image-generic (5.15.0.101.98) ... 2024-03-28T13:54:36.228 INFO:teuthology.orchestra.run.smithi171.stdout:Setting up linux-generic (5.15.0.101.98) ... 2024-03-28T13:54:36.354 INFO:teuthology.orchestra.run.smithi171.stdout:Processing triggers for linux-image-5.15.0-101-generic (5.15.0-101.111) ... 2024-03-28T13:54:36.413 INFO:teuthology.orchestra.run.smithi171.stdout:/etc/kernel/postinst.d/initramfs-tools: 2024-03-28T13:54:36.414 INFO:teuthology.orchestra.run.smithi171.stdout:update-initramfs: Generating /boot/initrd.img-5.15.0-101-generic 2024-03-28T13:54:47.841 INFO:teuthology.orchestra.run.smithi104.stdout:/etc/kernel/postinst.d/zz-update-grub: 2024-03-28T13:54:47.841 INFO:teuthology.orchestra.run.smithi104.stdout:Sourcing file `/etc/default/grub' 2024-03-28T13:54:47.851 INFO:teuthology.orchestra.run.smithi104.stdout:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-03-28T13:54:47.875 INFO:teuthology.orchestra.run.smithi104.stdout:Generating grub configuration file ... 2024-03-28T13:54:48.143 INFO:teuthology.orchestra.run.smithi104.stdout:Found linux image: /boot/vmlinuz-5.15.0-101-generic 2024-03-28T13:54:48.160 INFO:teuthology.orchestra.run.smithi104.stdout:Found initrd image: /boot/initrd.img-5.15.0-101-generic 2024-03-28T13:54:48.487 INFO:teuthology.orchestra.run.smithi104.stdout:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2024-03-28T13:54:48.492 INFO:teuthology.orchestra.run.smithi104.stdout:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2024-03-28T13:54:48.566 INFO:teuthology.orchestra.run.smithi104.stdout:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2024-03-28T13:54:48.571 INFO:teuthology.orchestra.run.smithi104.stdout:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2024-03-28T13:54:48.731 INFO:teuthology.orchestra.run.smithi104.stdout:Warning: os-prober will not be executed to detect other bootable partitions. 2024-03-28T13:54:48.731 INFO:teuthology.orchestra.run.smithi104.stdout:Systems on them will not be added to the GRUB boot configuration. 2024-03-28T13:54:48.731 INFO:teuthology.orchestra.run.smithi104.stdout:Check GRUB_DISABLE_OS_PROBER documentation entry. 2024-03-28T13:54:48.773 INFO:teuthology.orchestra.run.smithi104.stdout:done 2024-03-28T13:54:49.230 DEBUG:teuthology.orchestra.run.smithi104:> dpkg -s linux-image-generic 2024-03-28T13:54:49.252 INFO:teuthology.orchestra.run.smithi104.stdout:Package: linux-image-generic 2024-03-28T13:54:49.252 INFO:teuthology.orchestra.run.smithi104.stdout:Status: install ok installed 2024-03-28T13:54:49.252 INFO:teuthology.orchestra.run.smithi104.stdout:Priority: optional 2024-03-28T13:54:49.252 INFO:teuthology.orchestra.run.smithi104.stdout:Section: kernel 2024-03-28T13:54:49.252 INFO:teuthology.orchestra.run.smithi104.stdout:Installed-Size: 21 2024-03-28T13:54:49.252 INFO:teuthology.orchestra.run.smithi104.stdout:Maintainer: Ubuntu Kernel Team 2024-03-28T13:54:49.252 INFO:teuthology.orchestra.run.smithi104.stdout:Architecture: amd64 2024-03-28T13:54:49.252 INFO:teuthology.orchestra.run.smithi104.stdout:Source: linux-meta 2024-03-28T13:54:49.253 INFO:teuthology.orchestra.run.smithi104.stdout:Version: 5.15.0.101.98 2024-03-28T13:54:49.253 INFO:teuthology.orchestra.run.smithi104.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.3), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-101), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.3) 2024-03-28T13:54:49.253 INFO:teuthology.orchestra.run.smithi104.stdout:Depends: linux-image-5.15.0-101-generic, linux-modules-extra-5.15.0-101-generic, linux-firmware, intel-microcode, amd64-microcode 2024-03-28T13:54:49.253 INFO:teuthology.orchestra.run.smithi104.stdout:Recommends: thermald 2024-03-28T13:54:49.253 INFO:teuthology.orchestra.run.smithi104.stdout:Description: Generic Linux kernel image 2024-03-28T13:54:49.253 INFO:teuthology.orchestra.run.smithi104.stdout: This package will always depend on the latest generic kernel image 2024-03-28T13:54:49.253 INFO:teuthology.orchestra.run.smithi104.stdout: available. 2024-03-28T13:54:49.253 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.15.0-56-generic Expected: 5.15.0-101-generic 2024-03-28T13:54:49.253 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-28T13:54:49.254 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-28T13:54:49.254 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-03-28T13:54:49.254 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi104.front.sepia.ceph.com, path=None, version=distro) 2024-03-28T13:54:49.254 DEBUG:teuthology.orchestra.run.smithi104:> sudo apt-get clean 2024-03-28T13:54:49.435 DEBUG:teuthology.orchestra.run.smithi104:> sudo apt-get update 2024-03-28T13:54:49.616 INFO:teuthology.orchestra.run.smithi104.stdout:Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-03-28T13:54:49.616 INFO:teuthology.orchestra.run.smithi104.stdout:Hit:2 http://security.ubuntu.com/ubuntu jammy-security InRelease 2024-03-28T13:54:49.653 INFO:teuthology.orchestra.run.smithi104.stdout:Hit:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2024-03-28T13:54:49.698 INFO:teuthology.orchestra.run.smithi104.stdout:Hit:4 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2024-03-28T13:54:50.509 INFO:teuthology.orchestra.run.smithi171.stdout:/etc/kernel/postinst.d/zz-update-grub: 2024-03-28T13:54:50.509 INFO:teuthology.orchestra.run.smithi171.stdout:Sourcing file `/etc/default/grub' 2024-03-28T13:54:50.523 INFO:teuthology.orchestra.run.smithi171.stdout:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-03-28T13:54:50.551 INFO:teuthology.orchestra.run.smithi171.stdout:Generating grub configuration file ... 2024-03-28T13:54:50.861 INFO:teuthology.orchestra.run.smithi171.stdout:Found linux image: /boot/vmlinuz-5.15.0-101-generic 2024-03-28T13:54:50.877 INFO:teuthology.orchestra.run.smithi171.stdout:Found initrd image: /boot/initrd.img-5.15.0-101-generic 2024-03-28T13:54:51.023 INFO:teuthology.orchestra.run.smithi104.stdout:Reading package lists... 2024-03-28T13:54:51.041 DEBUG:teuthology.orchestra.run.smithi104:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-03-28T13:54:51.105 INFO:teuthology.orchestra.run.smithi104.stdout:Reading package lists... 2024-03-28T13:54:51.201 INFO:teuthology.orchestra.run.smithi171.stdout:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2024-03-28T13:54:51.207 INFO:teuthology.orchestra.run.smithi171.stdout:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2024-03-28T13:54:51.283 INFO:teuthology.orchestra.run.smithi171.stdout:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2024-03-28T13:54:51.288 INFO:teuthology.orchestra.run.smithi171.stdout:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2024-03-28T13:54:51.302 INFO:teuthology.orchestra.run.smithi104.stdout:Building dependency tree... 2024-03-28T13:54:51.303 INFO:teuthology.orchestra.run.smithi104.stdout:Reading state information... 2024-03-28T13:54:51.451 INFO:teuthology.orchestra.run.smithi171.stdout:Warning: os-prober will not be executed to detect other bootable partitions. 2024-03-28T13:54:51.451 INFO:teuthology.orchestra.run.smithi171.stdout:Systems on them will not be added to the GRUB boot configuration. 2024-03-28T13:54:51.451 INFO:teuthology.orchestra.run.smithi171.stdout:Check GRUB_DISABLE_OS_PROBER documentation entry. 2024-03-28T13:54:51.480 INFO:teuthology.orchestra.run.smithi104.stdout:linux-image-generic is already the newest version (5.15.0.101.98). 2024-03-28T13:54:51.480 INFO:teuthology.orchestra.run.smithi104.stdout:The following packages were automatically installed and are no longer required: 2024-03-28T13:54:51.480 INFO:teuthology.orchestra.run.smithi104.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-03-28T13:54:51.480 INFO:teuthology.orchestra.run.smithi104.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-03-28T13:54:51.480 INFO:teuthology.orchestra.run.smithi104.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-03-28T13:54:51.480 INFO:teuthology.orchestra.run.smithi104.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2024-03-28T13:54:51.481 INFO:teuthology.orchestra.run.smithi104.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2024-03-28T13:54:51.481 INFO:teuthology.orchestra.run.smithi104.stdout:Use 'sudo apt autoremove' to remove them. 2024-03-28T13:54:51.484 INFO:teuthology.orchestra.run.smithi171.stdout:done 2024-03-28T13:54:51.524 INFO:teuthology.orchestra.run.smithi104.stdout:0 upgraded, 0 newly installed, 0 to remove and 311 not upgraded. 2024-03-28T13:54:51.525 DEBUG:teuthology.orchestra.run.smithi104:> dpkg -s linux-image-generic 2024-03-28T13:54:51.541 INFO:teuthology.orchestra.run.smithi104.stdout:Package: linux-image-generic 2024-03-28T13:54:51.541 INFO:teuthology.orchestra.run.smithi104.stdout:Status: install ok installed 2024-03-28T13:54:51.541 INFO:teuthology.orchestra.run.smithi104.stdout:Priority: optional 2024-03-28T13:54:51.541 INFO:teuthology.orchestra.run.smithi104.stdout:Section: kernel 2024-03-28T13:54:51.542 INFO:teuthology.orchestra.run.smithi104.stdout:Installed-Size: 21 2024-03-28T13:54:51.542 INFO:teuthology.orchestra.run.smithi104.stdout:Maintainer: Ubuntu Kernel Team 2024-03-28T13:54:51.542 INFO:teuthology.orchestra.run.smithi104.stdout:Architecture: amd64 2024-03-28T13:54:51.542 INFO:teuthology.orchestra.run.smithi104.stdout:Source: linux-meta 2024-03-28T13:54:51.542 INFO:teuthology.orchestra.run.smithi104.stdout:Version: 5.15.0.101.98 2024-03-28T13:54:51.542 INFO:teuthology.orchestra.run.smithi104.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.3), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-101), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.3) 2024-03-28T13:54:51.542 INFO:teuthology.orchestra.run.smithi104.stdout:Depends: linux-image-5.15.0-101-generic, linux-modules-extra-5.15.0-101-generic, linux-firmware, intel-microcode, amd64-microcode 2024-03-28T13:54:51.542 INFO:teuthology.orchestra.run.smithi104.stdout:Recommends: thermald 2024-03-28T13:54:51.542 INFO:teuthology.orchestra.run.smithi104.stdout:Description: Generic Linux kernel image 2024-03-28T13:54:51.542 INFO:teuthology.orchestra.run.smithi104.stdout: This package will always depend on the latest generic kernel image 2024-03-28T13:54:51.542 INFO:teuthology.orchestra.run.smithi104.stdout: available. 2024-03-28T13:54:51.543 DEBUG:teuthology.orchestra.run.smithi104:> mktemp 2024-03-28T13:54:51.589 INFO:teuthology.orchestra.run.smithi104.stdout:/tmp/tmp.ZFhVFxwpQG 2024-03-28T13:54:51.589 DEBUG:teuthology.orchestra.run.smithi104:> sudo cp /boot/grub/grub.cfg /tmp/tmp.ZFhVFxwpQG 2024-03-28T13:54:51.646 DEBUG:teuthology.orchestra.run.smithi104:> sudo chmod 0666 /tmp/tmp.ZFhVFxwpQG 2024-03-28T13:54:51.763 DEBUG:teuthology.orchestra.remote:smithi104:/tmp/tmp.ZFhVFxwpQG is 10KB 2024-03-28T13:54:51.776 DEBUG:teuthology.orchestra.run.smithi104:> rm -fr /tmp/tmp.ZFhVFxwpQG 2024-03-28T13:54:51.782 DEBUG:teuthology.orchestra.run.smithi104:> sudo rm -f -- /etc/grub.d/01_ceph_kernel 2024-03-28T13:54:51.839 DEBUG:teuthology.orchestra.run.smithi104:> set -ex 2024-03-28T13:54:51.839 DEBUG:teuthology.orchestra.run.smithi104:> sudo dd of=/etc/grub.d/01_ceph_kernel 2024-03-28T13:54:51.839 DEBUG:teuthology.orchestra.run.smithi104:> sudo chmod 755 /etc/grub.d/01_ceph_kernel 2024-03-28T13:54:51.885 DEBUG:teuthology.orchestra.run.smithi171:> dpkg -s linux-image-generic 2024-03-28T13:54:51.905 INFO:teuthology.task.kernel:Distro Kernel Version: 5.15.0-101-generic 2024-03-28T13:54:51.905 DEBUG:teuthology.orchestra.run.smithi104:> sudo update-grub 2024-03-28T13:54:51.911 INFO:teuthology.orchestra.run.smithi171.stdout:Package: linux-image-generic 2024-03-28T13:54:51.911 INFO:teuthology.orchestra.run.smithi171.stdout:Status: install ok installed 2024-03-28T13:54:51.911 INFO:teuthology.orchestra.run.smithi171.stdout:Priority: optional 2024-03-28T13:54:51.911 INFO:teuthology.orchestra.run.smithi171.stdout:Section: kernel 2024-03-28T13:54:51.911 INFO:teuthology.orchestra.run.smithi171.stdout:Installed-Size: 21 2024-03-28T13:54:51.911 INFO:teuthology.orchestra.run.smithi171.stdout:Maintainer: Ubuntu Kernel Team 2024-03-28T13:54:51.911 INFO:teuthology.orchestra.run.smithi171.stdout:Architecture: amd64 2024-03-28T13:54:51.911 INFO:teuthology.orchestra.run.smithi171.stdout:Source: linux-meta 2024-03-28T13:54:51.911 INFO:teuthology.orchestra.run.smithi171.stdout:Version: 5.15.0.101.98 2024-03-28T13:54:51.911 INFO:teuthology.orchestra.run.smithi171.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.3), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-101), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.3) 2024-03-28T13:54:51.912 INFO:teuthology.orchestra.run.smithi171.stdout:Depends: linux-image-5.15.0-101-generic, linux-modules-extra-5.15.0-101-generic, linux-firmware, intel-microcode, amd64-microcode 2024-03-28T13:54:51.912 INFO:teuthology.orchestra.run.smithi171.stdout:Recommends: thermald 2024-03-28T13:54:51.912 INFO:teuthology.orchestra.run.smithi171.stdout:Description: Generic Linux kernel image 2024-03-28T13:54:51.912 INFO:teuthology.orchestra.run.smithi171.stdout: This package will always depend on the latest generic kernel image 2024-03-28T13:54:51.912 INFO:teuthology.orchestra.run.smithi171.stdout: available. 2024-03-28T13:54:51.912 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.15.0-56-generic Expected: 5.15.0-101-generic 2024-03-28T13:54:51.912 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-28T13:54:51.912 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-28T13:54:51.912 INFO:teuthology.task.kernel:Installing distro kernel on mon.b... 2024-03-28T13:54:51.912 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi171.front.sepia.ceph.com, path=None, version=distro) 2024-03-28T13:54:51.913 DEBUG:teuthology.orchestra.run.smithi171:> sudo apt-get clean 2024-03-28T13:54:52.106 DEBUG:teuthology.orchestra.run.smithi171:> sudo apt-get update 2024-03-28T13:54:52.252 INFO:teuthology.orchestra.run.smithi171.stdout:Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-03-28T13:54:52.253 INFO:teuthology.orchestra.run.smithi171.stdout:Hit:2 http://security.ubuntu.com/ubuntu jammy-security InRelease 2024-03-28T13:54:52.263 INFO:teuthology.orchestra.run.smithi171.stdout:Hit:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2024-03-28T13:54:52.283 INFO:teuthology.orchestra.run.smithi171.stdout:Hit:4 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2024-03-28T13:54:52.997 INFO:teuthology.orchestra.run.smithi104.stderr:Sourcing file `/etc/default/grub' 2024-03-28T13:54:52.998 INFO:teuthology.orchestra.run.smithi104.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-03-28T13:54:53.000 INFO:teuthology.orchestra.run.smithi104.stderr:Generating grub configuration file ... 2024-03-28T13:54:53.187 INFO:teuthology.orchestra.run.smithi104.stderr:Found linux image: /boot/vmlinuz-5.15.0-101-generic 2024-03-28T13:54:53.193 INFO:teuthology.orchestra.run.smithi104.stderr:Found initrd image: /boot/initrd.img-5.15.0-101-generic 2024-03-28T13:54:53.446 INFO:teuthology.orchestra.run.smithi104.stderr:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2024-03-28T13:54:53.451 INFO:teuthology.orchestra.run.smithi104.stderr:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2024-03-28T13:54:53.491 INFO:teuthology.orchestra.run.smithi104.stderr:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2024-03-28T13:54:53.496 INFO:teuthology.orchestra.run.smithi104.stderr:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2024-03-28T13:54:53.594 INFO:teuthology.orchestra.run.smithi171.stdout:Reading package lists... 2024-03-28T13:54:53.613 DEBUG:teuthology.orchestra.run.smithi171:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-03-28T13:54:53.644 INFO:teuthology.orchestra.run.smithi104.stderr:Warning: os-prober will not be executed to detect other bootable partitions. 2024-03-28T13:54:53.644 INFO:teuthology.orchestra.run.smithi104.stderr:Systems on them will not be added to the GRUB boot configuration. 2024-03-28T13:54:53.644 INFO:teuthology.orchestra.run.smithi104.stderr:Check GRUB_DISABLE_OS_PROBER documentation entry. 2024-03-28T13:54:53.657 INFO:teuthology.orchestra.run.smithi104.stderr:done 2024-03-28T13:54:53.663 INFO:teuthology.orchestra.run.smithi171.stdout:Reading package lists... 2024-03-28T13:54:53.673 DEBUG:teuthology.orchestra.run.smithi104:> sudo shutdown -r now 2024-03-28T13:54:53.851 INFO:teuthology.orchestra.run.smithi171.stdout:Building dependency tree... 2024-03-28T13:54:53.851 INFO:teuthology.orchestra.run.smithi171.stdout:Reading state information... 2024-03-28T13:54:54.004 INFO:teuthology.orchestra.run.smithi171.stdout:linux-image-generic is already the newest version (5.15.0.101.98). 2024-03-28T13:54:54.004 INFO:teuthology.orchestra.run.smithi171.stdout:The following packages were automatically installed and are no longer required: 2024-03-28T13:54:54.004 INFO:teuthology.orchestra.run.smithi171.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-03-28T13:54:54.004 INFO:teuthology.orchestra.run.smithi171.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-03-28T13:54:54.004 INFO:teuthology.orchestra.run.smithi171.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-03-28T13:54:54.004 INFO:teuthology.orchestra.run.smithi171.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2024-03-28T13:54:54.005 INFO:teuthology.orchestra.run.smithi171.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2024-03-28T13:54:54.005 INFO:teuthology.orchestra.run.smithi171.stdout:Use 'sudo apt autoremove' to remove them. 2024-03-28T13:54:54.042 INFO:teuthology.orchestra.run.smithi171.stdout:0 upgraded, 0 newly installed, 0 to remove and 311 not upgraded. 2024-03-28T13:54:54.044 DEBUG:teuthology.orchestra.run.smithi171:> dpkg -s linux-image-generic 2024-03-28T13:54:54.102 INFO:teuthology.orchestra.run.smithi171.stdout:Package: linux-image-generic 2024-03-28T13:54:54.102 INFO:teuthology.orchestra.run.smithi171.stdout:Status: install ok installed 2024-03-28T13:54:54.102 INFO:teuthology.orchestra.run.smithi171.stdout:Priority: optional 2024-03-28T13:54:54.102 INFO:teuthology.orchestra.run.smithi171.stdout:Section: kernel 2024-03-28T13:54:54.102 INFO:teuthology.orchestra.run.smithi171.stdout:Installed-Size: 21 2024-03-28T13:54:54.102 INFO:teuthology.orchestra.run.smithi171.stdout:Maintainer: Ubuntu Kernel Team 2024-03-28T13:54:54.102 INFO:teuthology.orchestra.run.smithi171.stdout:Architecture: amd64 2024-03-28T13:54:54.102 INFO:teuthology.orchestra.run.smithi171.stdout:Source: linux-meta 2024-03-28T13:54:54.102 INFO:teuthology.orchestra.run.smithi171.stdout:Version: 5.15.0.101.98 2024-03-28T13:54:54.102 INFO:teuthology.orchestra.run.smithi171.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.3), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-101), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.3) 2024-03-28T13:54:54.103 INFO:teuthology.orchestra.run.smithi171.stdout:Depends: linux-image-5.15.0-101-generic, linux-modules-extra-5.15.0-101-generic, linux-firmware, intel-microcode, amd64-microcode 2024-03-28T13:54:54.103 INFO:teuthology.orchestra.run.smithi171.stdout:Recommends: thermald 2024-03-28T13:54:54.103 INFO:teuthology.orchestra.run.smithi171.stdout:Description: Generic Linux kernel image 2024-03-28T13:54:54.103 INFO:teuthology.orchestra.run.smithi171.stdout: This package will always depend on the latest generic kernel image 2024-03-28T13:54:54.103 INFO:teuthology.orchestra.run.smithi171.stdout: available. 2024-03-28T13:54:54.103 DEBUG:teuthology.orchestra.run.smithi171:> mktemp 2024-03-28T13:54:54.146 INFO:teuthology.orchestra.run.smithi171.stdout:/tmp/tmp.i9gpbvp5nn 2024-03-28T13:54:54.147 DEBUG:teuthology.orchestra.run.smithi171:> sudo cp /boot/grub/grub.cfg /tmp/tmp.i9gpbvp5nn 2024-03-28T13:54:54.203 DEBUG:teuthology.orchestra.run.smithi171:> sudo chmod 0666 /tmp/tmp.i9gpbvp5nn 2024-03-28T13:54:54.329 DEBUG:teuthology.orchestra.remote:smithi171:/tmp/tmp.i9gpbvp5nn is 10KB 2024-03-28T13:54:54.341 DEBUG:teuthology.orchestra.run.smithi171:> rm -fr /tmp/tmp.i9gpbvp5nn 2024-03-28T13:54:54.346 DEBUG:teuthology.orchestra.run.smithi171:> sudo rm -f -- /etc/grub.d/01_ceph_kernel 2024-03-28T13:54:54.406 DEBUG:teuthology.orchestra.run.smithi171:> set -ex 2024-03-28T13:54:54.407 DEBUG:teuthology.orchestra.run.smithi171:> sudo dd of=/etc/grub.d/01_ceph_kernel 2024-03-28T13:54:54.407 DEBUG:teuthology.orchestra.run.smithi171:> sudo chmod 755 /etc/grub.d/01_ceph_kernel 2024-03-28T13:54:54.474 INFO:teuthology.task.kernel:Distro Kernel Version: 5.15.0-101-generic 2024-03-28T13:54:54.474 DEBUG:teuthology.orchestra.run.smithi171:> sudo update-grub 2024-03-28T13:54:55.662 INFO:teuthology.orchestra.run.smithi171.stderr:Sourcing file `/etc/default/grub' 2024-03-28T13:54:55.663 INFO:teuthology.orchestra.run.smithi171.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-03-28T13:54:55.665 INFO:teuthology.orchestra.run.smithi171.stderr:Generating grub configuration file ... 2024-03-28T13:54:55.863 INFO:teuthology.orchestra.run.smithi171.stderr:Found linux image: /boot/vmlinuz-5.15.0-101-generic 2024-03-28T13:54:55.869 INFO:teuthology.orchestra.run.smithi171.stderr:Found initrd image: /boot/initrd.img-5.15.0-101-generic 2024-03-28T13:54:56.089 INFO:teuthology.orchestra.run.smithi171.stderr:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2024-03-28T13:54:56.094 INFO:teuthology.orchestra.run.smithi171.stderr:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2024-03-28T13:54:56.132 INFO:teuthology.orchestra.run.smithi171.stderr:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2024-03-28T13:54:56.137 INFO:teuthology.orchestra.run.smithi171.stderr:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2024-03-28T13:54:56.272 INFO:teuthology.orchestra.run.smithi171.stderr:Warning: os-prober will not be executed to detect other bootable partitions. 2024-03-28T13:54:56.272 INFO:teuthology.orchestra.run.smithi171.stderr:Systems on them will not be added to the GRUB boot configuration. 2024-03-28T13:54:56.273 INFO:teuthology.orchestra.run.smithi171.stderr:Check GRUB_DISABLE_OS_PROBER documentation entry. 2024-03-28T13:54:56.298 INFO:teuthology.orchestra.run.smithi171.stderr:done 2024-03-28T13:54:56.301 DEBUG:teuthology.orchestra.run.smithi171:> sudo shutdown -r now 2024-03-28T13:55:23.703 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-03-28T13:55:23.703 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi104.front.sepia.ceph.com' 2024-03-28T13:55:23.704 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi104.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-28T13:55:26.306 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2024-03-28T13:55:26.307 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi171.front.sepia.ceph.com' 2024-03-28T13:55:26.307 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi171.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-28T13:55:58.367 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.104 2024-03-28T13:56:00.926 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.171 2024-03-28T13:56:07.374 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi104.front.sepia.ceph.com' 2024-03-28T13:56:07.375 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi104.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-28T13:56:09.927 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi171.front.sepia.ceph.com' 2024-03-28T13:56:09.928 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi171.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-28T13:56:25.762 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.104 2024-03-28T13:56:28.322 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.171 2024-03-28T13:56:37.768 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi104.front.sepia.ceph.com' 2024-03-28T13:56:37.768 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi104.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-28T13:56:38.085 DEBUG:teuthology.orchestra.run.smithi104:> true 2024-03-28T13:56:38.611 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi104.front.sepia.ceph.com' 2024-03-28T13:56:38.611 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.15.0-101-generic"... 2024-03-28T13:56:38.611 DEBUG:teuthology.orchestra.run.smithi104:> uname -r 2024-03-28T13:56:38.659 INFO:teuthology.orchestra.run.smithi104.stdout:5.15.0-101-generic 2024-03-28T13:56:38.659 DEBUG:teuthology.task.kernel:current kernel version is 5.15.0-101-generic vs 5.15.0-101-generic 2024-03-28T13:56:38.659 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-28T13:56:38.659 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2024-03-28T13:56:39.660 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-03-28T13:56:39.660 DEBUG:teuthology.orchestra.run.smithi104:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-28T13:56:39.742 INFO:teuthology.orchestra.run.smithi104.stdout:ttyS1 2024-03-28T13:56:39.765 DEBUG:teuthology.parallel:result is None 2024-03-28T13:56:40.323 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi171.front.sepia.ceph.com' 2024-03-28T13:56:40.324 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi171.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-28T13:56:42.367 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.171 2024-03-28T13:56:57.368 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi171.front.sepia.ceph.com' 2024-03-28T13:56:57.369 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi171.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-28T13:56:57.674 DEBUG:teuthology.orchestra.run.smithi171:> true 2024-03-28T13:56:58.280 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi171.front.sepia.ceph.com' 2024-03-28T13:56:58.280 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "5.15.0-101-generic"... 2024-03-28T13:56:58.280 DEBUG:teuthology.orchestra.run.smithi171:> uname -r 2024-03-28T13:56:58.326 INFO:teuthology.orchestra.run.smithi171.stdout:5.15.0-101-generic 2024-03-28T13:56:58.326 DEBUG:teuthology.task.kernel:current kernel version is 5.15.0-101-generic vs 5.15.0-101-generic 2024-03-28T13:56:58.326 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-28T13:56:58.327 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2024-03-28T13:56:59.327 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2024-03-28T13:56:59.328 DEBUG:teuthology.orchestra.run.smithi171:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-28T13:56:59.402 INFO:teuthology.orchestra.run.smithi171.stdout:ttyS1 2024-03-28T13:56:59.428 DEBUG:teuthology.parallel:result is None 2024-03-28T13:56:59.429 INFO:teuthology.run_tasks:Running task internal.base... 2024-03-28T13:56:59.438 INFO:teuthology.task.internal:Creating test directory... 2024-03-28T13:56:59.439 DEBUG:teuthology.orchestra.run.smithi104:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-28T13:56:59.441 DEBUG:teuthology.orchestra.run.smithi171:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-28T13:56:59.447 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-03-28T13:56:59.454 INFO:teuthology.run_tasks:Running task internal.archive... 2024-03-28T13:56:59.461 INFO:teuthology.task.internal:Creating archive directory... 2024-03-28T13:56:59.461 DEBUG:teuthology.orchestra.run.smithi104:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-28T13:56:59.489 DEBUG:teuthology.orchestra.run.smithi171:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-28T13:56:59.514 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-03-28T13:56:59.521 INFO:teuthology.task.internal:Enabling coredump saving... 2024-03-28T13:56:59.521 DEBUG:teuthology.orchestra.run.smithi104:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-28T13:56:59.553 DEBUG:teuthology.orchestra.run.smithi171:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-28T13:56:59.573 INFO:teuthology.orchestra.run.smithi171.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-28T13:56:59.573 INFO:teuthology.orchestra.run.smithi104.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-28T13:56:59.580 INFO:teuthology.orchestra.run.smithi104.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-28T13:56:59.581 INFO:teuthology.orchestra.run.smithi171.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-28T13:56:59.583 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-03-28T13:56:59.591 INFO:teuthology.task.internal:Configuring sudo... 2024-03-28T13:56:59.591 DEBUG:teuthology.orchestra.run.smithi104:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-28T13:56:59.628 DEBUG:teuthology.orchestra.run.smithi171:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-28T13:56:59.646 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-03-28T13:56:59.656 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-03-28T13:56:59.656 DEBUG:teuthology.orchestra.run.smithi104:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-28T13:56:59.688 DEBUG:teuthology.orchestra.run.smithi171:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-28T13:56:59.694 DEBUG:teuthology.orchestra.run.smithi104:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-28T13:56:59.739 DEBUG:teuthology.orchestra.run.smithi104:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-28T13:56:59.790 DEBUG:teuthology.orchestra.run.smithi104:> set -ex 2024-03-28T13:56:59.791 DEBUG:teuthology.orchestra.run.smithi104:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-28T13:56:59.851 DEBUG:teuthology.orchestra.run.smithi171:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-28T13:56:59.856 DEBUG:teuthology.orchestra.run.smithi171:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-28T13:56:59.903 DEBUG:teuthology.orchestra.run.smithi171:> set -ex 2024-03-28T13:56:59.903 DEBUG:teuthology.orchestra.run.smithi171:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-28T13:56:59.964 DEBUG:teuthology.orchestra.run.smithi104:> sudo service rsyslog restart 2024-03-28T13:56:59.966 DEBUG:teuthology.orchestra.run.smithi171:> sudo service rsyslog restart 2024-03-28T13:57:00.036 INFO:teuthology.run_tasks:Running task internal.timer... 2024-03-28T13:57:00.043 INFO:teuthology.task.internal:Starting timer... 2024-03-28T13:57:00.044 INFO:teuthology.run_tasks:Running task pcp... 2024-03-28T13:57:00.054 INFO:teuthology.run_tasks:Running task selinux... 2024-03-28T13:57:00.063 DEBUG:teuthology.task.selinux:Excluding smithi104: OS 'ubuntu' does not support SELinux 2024-03-28T13:57:00.063 DEBUG:teuthology.task.selinux:Excluding smithi171: OS 'ubuntu' does not support SELinux 2024-03-28T13:57:00.063 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-03-28T13:57:00.063 DEBUG:teuthology.task.selinux:Existing SELinux modes: {} 2024-03-28T13:57:00.063 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-03-28T13:57:00.063 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-03-28T13:57:00.072 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-03-28T13:57:00.090 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-03-28T13:57:00.091 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi104.front.sepia.ceph.com,smithi171.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-03-28T14:04:01.003 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi104.front.sepia.ceph.com'), Remote(name='ubuntu@smithi171.front.sepia.ceph.com')] 2024-03-28T14:04:01.005 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi104.front.sepia.ceph.com' 2024-03-28T14:04:01.006 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi104.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-28T14:04:01.087 DEBUG:teuthology.orchestra.run.smithi104:> true 2024-03-28T14:04:01.163 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi104.front.sepia.ceph.com' 2024-03-28T14:04:01.163 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi171.front.sepia.ceph.com' 2024-03-28T14:04:01.164 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi171.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-28T14:04:01.247 DEBUG:teuthology.orchestra.run.smithi171:> true 2024-03-28T14:04:01.326 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi171.front.sepia.ceph.com' 2024-03-28T14:04:01.327 INFO:teuthology.run_tasks:Running task clock... 2024-03-28T14:04:01.337 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-03-28T14:04:01.337 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-28T14:04:01.338 DEBUG:teuthology.orchestra.run.smithi104:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-28T14:04:01.340 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-28T14:04:01.341 DEBUG:teuthology.orchestra.run.smithi171:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-28T14:04:01.365 INFO:teuthology.orchestra.run.smithi104.stdout:28 Mar 14:04:01 ntpd[17165]: ntpd 4.2.8p15@1.3728-o Wed Feb 16 17:13:02 UTC 2022 (1): Starting 2024-03-28T14:04:01.366 INFO:teuthology.orchestra.run.smithi104.stdout:28 Mar 14:04:01 ntpd[17165]: Command line: ntpd -gq 2024-03-28T14:04:01.366 INFO:teuthology.orchestra.run.smithi104.stdout:28 Mar 14:04:01 ntpd[17165]: ---------------------------------------------------- 2024-03-28T14:04:01.366 INFO:teuthology.orchestra.run.smithi104.stdout:28 Mar 14:04:01 ntpd[17165]: ntp-4 is maintained by Network Time Foundation, 2024-03-28T14:04:01.366 INFO:teuthology.orchestra.run.smithi104.stdout:28 Mar 14:04:01 ntpd[17165]: Inc. (NTF), a non-profit 501(c)(3) public-benefit 2024-03-28T14:04:01.366 INFO:teuthology.orchestra.run.smithi104.stdout:28 Mar 14:04:01 ntpd[17165]: corporation. Support and training for ntp-4 are 2024-03-28T14:04:01.366 INFO:teuthology.orchestra.run.smithi104.stdout:28 Mar 14:04:01 ntpd[17165]: available at https://www.nwtime.org/support 2024-03-28T14:04:01.366 INFO:teuthology.orchestra.run.smithi104.stdout:28 Mar 14:04:01 ntpd[17165]: ---------------------------------------------------- 2024-03-28T14:04:01.366 INFO:teuthology.orchestra.run.smithi104.stdout:28 Mar 14:04:01 ntpd[17165]: proto: precision = 0.063 usec (-24) 2024-03-28T14:04:01.367 INFO:teuthology.orchestra.run.smithi104.stdout:28 Mar 14:04:01 ntpd[17165]: basedate set to 2022-02-04 2024-03-28T14:04:01.367 INFO:teuthology.orchestra.run.smithi104.stdout:28 Mar 14:04:01 ntpd[17165]: gps base set to 2022-02-06 (week 2196) 2024-03-28T14:04:01.367 INFO:teuthology.orchestra.run.smithi104.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-03-28T14:04:01.367 INFO:teuthology.orchestra.run.smithi104.stdout:28 Mar 14:04:01 ntpd[17165]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-03-28T14:04:01.367 INFO:teuthology.orchestra.run.smithi104.stdout:28 Mar 14:04:01 ntpd[17165]: restrict ::: KOD does nothing without LIMITED. 2024-03-28T14:04:01.368 INFO:teuthology.orchestra.run.smithi104.stderr:restrict ::: KOD does nothing without LIMITED. 2024-03-28T14:04:01.369 INFO:teuthology.orchestra.run.smithi104.stdout:28 Mar 14:04:01 ntpd[17165]: Listen and drop on 0 v6wildcard [::]:123 2024-03-28T14:04:01.369 INFO:teuthology.orchestra.run.smithi104.stdout:28 Mar 14:04:01 ntpd[17165]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2024-03-28T14:04:01.369 INFO:teuthology.orchestra.run.smithi104.stdout:28 Mar 14:04:01 ntpd[17165]: Listen normally on 2 lo 127.0.0.1:123 2024-03-28T14:04:01.369 INFO:teuthology.orchestra.run.smithi104.stdout:28 Mar 14:04:01 ntpd[17165]: Listen normally on 3 enp3s0f1 172.21.15.104:123 2024-03-28T14:04:01.369 INFO:teuthology.orchestra.run.smithi104.stdout:28 Mar 14:04:01 ntpd[17165]: Listen normally on 4 lo [::1]:123 2024-03-28T14:04:01.369 INFO:teuthology.orchestra.run.smithi104.stdout:28 Mar 14:04:01 ntpd[17165]: Listen normally on 5 enp3s0f1 [fe80::ec4:7aff:fe8f:ce89%5]:123 2024-03-28T14:04:01.369 INFO:teuthology.orchestra.run.smithi104.stdout:28 Mar 14:04:01 ntpd[17165]: Listening on routing socket on fd #22 for interface updates 2024-03-28T14:04:01.392 INFO:teuthology.orchestra.run.smithi171.stdout:28 Mar 14:04:01 ntpd[17191]: ntpd 4.2.8p15@1.3728-o Wed Feb 16 17:13:02 UTC 2022 (1): Starting 2024-03-28T14:04:01.393 INFO:teuthology.orchestra.run.smithi171.stdout:28 Mar 14:04:01 ntpd[17191]: Command line: ntpd -gq 2024-03-28T14:04:01.393 INFO:teuthology.orchestra.run.smithi171.stdout:28 Mar 14:04:01 ntpd[17191]: ---------------------------------------------------- 2024-03-28T14:04:01.393 INFO:teuthology.orchestra.run.smithi171.stdout:28 Mar 14:04:01 ntpd[17191]: ntp-4 is maintained by Network Time Foundation, 2024-03-28T14:04:01.393 INFO:teuthology.orchestra.run.smithi171.stdout:28 Mar 14:04:01 ntpd[17191]: Inc. (NTF), a non-profit 501(c)(3) public-benefit 2024-03-28T14:04:01.393 INFO:teuthology.orchestra.run.smithi171.stdout:28 Mar 14:04:01 ntpd[17191]: corporation. Support and training for ntp-4 are 2024-03-28T14:04:01.393 INFO:teuthology.orchestra.run.smithi171.stdout:28 Mar 14:04:01 ntpd[17191]: available at https://www.nwtime.org/support 2024-03-28T14:04:01.393 INFO:teuthology.orchestra.run.smithi171.stdout:28 Mar 14:04:01 ntpd[17191]: ---------------------------------------------------- 2024-03-28T14:04:01.393 INFO:teuthology.orchestra.run.smithi171.stdout:28 Mar 14:04:01 ntpd[17191]: proto: precision = 0.045 usec (-24) 2024-03-28T14:04:01.394 INFO:teuthology.orchestra.run.smithi171.stdout:28 Mar 14:04:01 ntpd[17191]: basedate set to 2022-02-04 2024-03-28T14:04:01.394 INFO:teuthology.orchestra.run.smithi171.stdout:28 Mar 14:04:01 ntpd[17191]: gps base set to 2022-02-06 (week 2196) 2024-03-28T14:04:01.394 INFO:teuthology.orchestra.run.smithi171.stdout:28 Mar 14:04:01 ntpd[17191]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-03-28T14:04:01.394 INFO:teuthology.orchestra.run.smithi171.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-03-28T14:04:01.394 INFO:teuthology.orchestra.run.smithi171.stderr:restrict ::: KOD does nothing without LIMITED. 2024-03-28T14:04:01.396 INFO:teuthology.orchestra.run.smithi171.stdout:28 Mar 14:04:01 ntpd[17191]: restrict ::: KOD does nothing without LIMITED. 2024-03-28T14:04:01.396 INFO:teuthology.orchestra.run.smithi171.stdout:28 Mar 14:04:01 ntpd[17191]: Listen and drop on 0 v6wildcard [::]:123 2024-03-28T14:04:01.396 INFO:teuthology.orchestra.run.smithi171.stdout:28 Mar 14:04:01 ntpd[17191]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2024-03-28T14:04:01.396 INFO:teuthology.orchestra.run.smithi171.stdout:28 Mar 14:04:01 ntpd[17191]: Listen normally on 2 lo 127.0.0.1:123 2024-03-28T14:04:01.396 INFO:teuthology.orchestra.run.smithi171.stdout:28 Mar 14:04:01 ntpd[17191]: Listen normally on 3 enp3s0f1 172.21.15.171:123 2024-03-28T14:04:01.396 INFO:teuthology.orchestra.run.smithi171.stdout:28 Mar 14:04:01 ntpd[17191]: Listen normally on 4 lo [::1]:123 2024-03-28T14:04:01.396 INFO:teuthology.orchestra.run.smithi171.stdout:28 Mar 14:04:01 ntpd[17191]: Listen normally on 5 enp3s0f1 [fe80::ec4:7aff:fe88:6f6d%5]:123 2024-03-28T14:04:01.396 INFO:teuthology.orchestra.run.smithi171.stdout:28 Mar 14:04:01 ntpd[17191]: Listening on routing socket on fd #22 for interface updates 2024-03-28T14:04:02.366 INFO:teuthology.orchestra.run.smithi104.stderr:28 Mar 14:04:02 ntpd[17165]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-28T14:04:02.366 INFO:teuthology.orchestra.run.smithi104.stderr:28 Mar 14:04:02 ntpd[17165]: can't open /var/log/ntpstats/rawstats.20240328: Permission denied 2024-03-28T14:04:02.367 INFO:teuthology.orchestra.run.smithi104.stderr:28 Mar 14:04:02 ntpd[17165]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-28T14:04:02.367 INFO:teuthology.orchestra.run.smithi104.stderr:28 Mar 14:04:02 ntpd[17165]: can't open /var/log/ntpstats/rawstats.20240328: Permission denied 2024-03-28T14:04:02.367 INFO:teuthology.orchestra.run.smithi104.stderr:28 Mar 14:04:02 ntpd[17165]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-28T14:04:02.367 INFO:teuthology.orchestra.run.smithi104.stderr:28 Mar 14:04:02 ntpd[17165]: can't open /var/log/ntpstats/rawstats.20240328: Permission denied 2024-03-28T14:04:02.367 INFO:teuthology.orchestra.run.smithi104.stderr:28 Mar 14:04:02 ntpd[17165]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-28T14:04:02.367 INFO:teuthology.orchestra.run.smithi104.stderr:28 Mar 14:04:02 ntpd[17165]: can't open /var/log/ntpstats/rawstats.20240328: Permission denied 2024-03-28T14:04:02.367 INFO:teuthology.orchestra.run.smithi104.stderr:28 Mar 14:04:02 ntpd[17165]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-03-28T14:04:02.367 INFO:teuthology.orchestra.run.smithi104.stderr:28 Mar 14:04:02 ntpd[17165]: can't open /var/log/ntpstats/peerstats.20240328: Permission denied 2024-03-28T14:04:02.367 INFO:teuthology.orchestra.run.smithi104.stderr:28 Mar 14:04:02 ntpd[17165]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-28T14:04:02.367 INFO:teuthology.orchestra.run.smithi104.stderr:28 Mar 14:04:02 ntpd[17165]: can't open /var/log/ntpstats/rawstats.20240328: Permission denied 2024-03-28T14:04:02.367 INFO:teuthology.orchestra.run.smithi104.stderr:28 Mar 14:04:02 ntpd[17165]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-03-28T14:04:02.367 INFO:teuthology.orchestra.run.smithi104.stderr:28 Mar 14:04:02 ntpd[17165]: can't open /var/log/ntpstats/peerstats.20240328: Permission denied 2024-03-28T14:04:02.393 INFO:teuthology.orchestra.run.smithi171.stderr:28 Mar 14:04:02 ntpd[17191]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-28T14:04:02.393 INFO:teuthology.orchestra.run.smithi171.stderr:28 Mar 14:04:02 ntpd[17191]: can't open /var/log/ntpstats/rawstats.20240328: Permission denied 2024-03-28T14:04:02.394 INFO:teuthology.orchestra.run.smithi171.stderr:28 Mar 14:04:02 ntpd[17191]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-28T14:04:02.394 INFO:teuthology.orchestra.run.smithi171.stderr:28 Mar 14:04:02 ntpd[17191]: can't open /var/log/ntpstats/rawstats.20240328: Permission denied 2024-03-28T14:04:02.394 INFO:teuthology.orchestra.run.smithi171.stderr:28 Mar 14:04:02 ntpd[17191]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-28T14:04:02.394 INFO:teuthology.orchestra.run.smithi171.stderr:28 Mar 14:04:02 ntpd[17191]: can't open /var/log/ntpstats/rawstats.20240328: Permission denied 2024-03-28T14:04:02.394 INFO:teuthology.orchestra.run.smithi171.stderr:28 Mar 14:04:02 ntpd[17191]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-03-28T14:04:02.394 INFO:teuthology.orchestra.run.smithi171.stderr:28 Mar 14:04:02 ntpd[17191]: can't open /var/log/ntpstats/peerstats.20240328: Permission denied 2024-03-28T14:04:02.394 INFO:teuthology.orchestra.run.smithi171.stderr:28 Mar 14:04:02 ntpd[17191]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-28T14:04:02.394 INFO:teuthology.orchestra.run.smithi171.stderr:28 Mar 14:04:02 ntpd[17191]: can't open /var/log/ntpstats/rawstats.20240328: Permission denied 2024-03-28T14:04:02.394 INFO:teuthology.orchestra.run.smithi171.stderr:28 Mar 14:04:02 ntpd[17191]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-03-28T14:04:02.394 INFO:teuthology.orchestra.run.smithi171.stderr:28 Mar 14:04:02 ntpd[17191]: can't open /var/log/ntpstats/peerstats.20240328: Permission denied 2024-03-28T14:04:04.366 INFO:teuthology.orchestra.run.smithi104.stderr:28 Mar 14:04:04 ntpd[17165]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-28T14:04:04.367 INFO:teuthology.orchestra.run.smithi104.stderr:28 Mar 14:04:04 ntpd[17165]: can't open /var/log/ntpstats/rawstats.20240328: Permission denied 2024-03-28T14:04:04.367 INFO:teuthology.orchestra.run.smithi104.stderr:28 Mar 14:04:04 ntpd[17165]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-28T14:04:04.367 INFO:teuthology.orchestra.run.smithi104.stderr:28 Mar 14:04:04 ntpd[17165]: can't open /var/log/ntpstats/rawstats.20240328: Permission denied 2024-03-28T14:04:04.367 INFO:teuthology.orchestra.run.smithi104.stderr:28 Mar 14:04:04 ntpd[17165]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-28T14:04:04.367 INFO:teuthology.orchestra.run.smithi104.stderr:28 Mar 14:04:04 ntpd[17165]: can't open /var/log/ntpstats/rawstats.20240328: Permission denied 2024-03-28T14:04:04.367 INFO:teuthology.orchestra.run.smithi104.stderr:28 Mar 14:04:04 ntpd[17165]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-03-28T14:04:04.367 INFO:teuthology.orchestra.run.smithi104.stderr:28 Mar 14:04:04 ntpd[17165]: can't open /var/log/ntpstats/peerstats.20240328: Permission denied 2024-03-28T14:04:04.367 INFO:teuthology.orchestra.run.smithi104.stderr:28 Mar 14:04:04 ntpd[17165]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-28T14:04:04.367 INFO:teuthology.orchestra.run.smithi104.stderr:28 Mar 14:04:04 ntpd[17165]: can't open /var/log/ntpstats/rawstats.20240328: Permission denied 2024-03-28T14:04:04.367 INFO:teuthology.orchestra.run.smithi104.stderr:28 Mar 14:04:04 ntpd[17165]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-03-28T14:04:04.367 INFO:teuthology.orchestra.run.smithi104.stderr:28 Mar 14:04:04 ntpd[17165]: can't open /var/log/ntpstats/peerstats.20240328: Permission denied 2024-03-28T14:04:04.393 INFO:teuthology.orchestra.run.smithi171.stderr:28 Mar 14:04:04 ntpd[17191]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-28T14:04:04.393 INFO:teuthology.orchestra.run.smithi171.stderr:28 Mar 14:04:04 ntpd[17191]: can't open /var/log/ntpstats/rawstats.20240328: Permission denied 2024-03-28T14:04:04.394 INFO:teuthology.orchestra.run.smithi171.stderr:28 Mar 14:04:04 ntpd[17191]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-28T14:04:04.394 INFO:teuthology.orchestra.run.smithi171.stderr:28 Mar 14:04:04 ntpd[17191]: can't open /var/log/ntpstats/rawstats.20240328: Permission denied 2024-03-28T14:04:04.394 INFO:teuthology.orchestra.run.smithi171.stderr:28 Mar 14:04:04 ntpd[17191]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-28T14:04:04.394 INFO:teuthology.orchestra.run.smithi171.stderr:28 Mar 14:04:04 ntpd[17191]: can't open /var/log/ntpstats/rawstats.20240328: Permission denied 2024-03-28T14:04:04.394 INFO:teuthology.orchestra.run.smithi171.stderr:28 Mar 14:04:04 ntpd[17191]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-28T14:04:04.394 INFO:teuthology.orchestra.run.smithi171.stderr:28 Mar 14:04:04 ntpd[17191]: can't open /var/log/ntpstats/rawstats.20240328: Permission denied 2024-03-28T14:04:04.394 INFO:teuthology.orchestra.run.smithi171.stderr:28 Mar 14:04:04 ntpd[17191]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-03-28T14:04:04.394 INFO:teuthology.orchestra.run.smithi171.stderr:28 Mar 14:04:04 ntpd[17191]: can't open /var/log/ntpstats/peerstats.20240328: Permission denied 2024-03-28T14:04:04.394 INFO:teuthology.orchestra.run.smithi171.stderr:28 Mar 14:04:04 ntpd[17191]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-28T14:04:04.394 INFO:teuthology.orchestra.run.smithi171.stderr:28 Mar 14:04:04 ntpd[17191]: can't open /var/log/ntpstats/rawstats.20240328: Permission denied 2024-03-28T14:04:04.394 INFO:teuthology.orchestra.run.smithi171.stderr:28 Mar 14:04:04 ntpd[17191]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-03-28T14:04:04.394 INFO:teuthology.orchestra.run.smithi171.stderr:28 Mar 14:04:04 ntpd[17191]: can't open /var/log/ntpstats/peerstats.20240328: Permission denied 2024-03-28T14:04:05.366 INFO:teuthology.orchestra.run.smithi104.stderr:28 Mar 14:04:05 ntpd[17165]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-28T14:04:05.366 INFO:teuthology.orchestra.run.smithi104.stderr:28 Mar 14:04:05 ntpd[17165]: can't open /var/log/ntpstats/rawstats.20240328: Permission denied 2024-03-28T14:04:05.366 INFO:teuthology.orchestra.run.smithi104.stderr:28 Mar 14:04:05 ntpd[17165]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-28T14:04:05.366 INFO:teuthology.orchestra.run.smithi104.stderr:28 Mar 14:04:05 ntpd[17165]: can't open /var/log/ntpstats/rawstats.20240328: Permission denied 2024-03-28T14:04:05.366 INFO:teuthology.orchestra.run.smithi104.stderr:28 Mar 14:04:05 ntpd[17165]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-03-28T14:04:05.366 INFO:teuthology.orchestra.run.smithi104.stderr:28 Mar 14:04:05 ntpd[17165]: can't open /var/log/ntpstats/peerstats.20240328: Permission denied 2024-03-28T14:04:06.366 INFO:teuthology.orchestra.run.smithi104.stderr:28 Mar 14:04:06 ntpd[17165]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-28T14:04:06.367 INFO:teuthology.orchestra.run.smithi104.stderr:28 Mar 14:04:06 ntpd[17165]: can't open /var/log/ntpstats/rawstats.20240328: Permission denied 2024-03-28T14:04:06.367 INFO:teuthology.orchestra.run.smithi104.stderr:28 Mar 14:04:06 ntpd[17165]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-28T14:04:06.367 INFO:teuthology.orchestra.run.smithi104.stderr:28 Mar 14:04:06 ntpd[17165]: can't open /var/log/ntpstats/rawstats.20240328: Permission denied 2024-03-28T14:04:06.367 INFO:teuthology.orchestra.run.smithi104.stderr:28 Mar 14:04:06 ntpd[17165]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-28T14:04:06.367 INFO:teuthology.orchestra.run.smithi104.stderr:28 Mar 14:04:06 ntpd[17165]: can't open /var/log/ntpstats/rawstats.20240328: Permission denied 2024-03-28T14:04:06.367 INFO:teuthology.orchestra.run.smithi104.stderr:28 Mar 14:04:06 ntpd[17165]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-03-28T14:04:06.367 INFO:teuthology.orchestra.run.smithi104.stderr:28 Mar 14:04:06 ntpd[17165]: can't open /var/log/ntpstats/peerstats.20240328: Permission denied 2024-03-28T14:04:06.367 INFO:teuthology.orchestra.run.smithi104.stderr:28 Mar 14:04:06 ntpd[17165]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-28T14:04:06.367 INFO:teuthology.orchestra.run.smithi104.stderr:28 Mar 14:04:06 ntpd[17165]: can't open /var/log/ntpstats/rawstats.20240328: Permission denied 2024-03-28T14:04:06.367 INFO:teuthology.orchestra.run.smithi104.stderr:28 Mar 14:04:06 ntpd[17165]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-03-28T14:04:06.367 INFO:teuthology.orchestra.run.smithi104.stderr:28 Mar 14:04:06 ntpd[17165]: can't open /var/log/ntpstats/peerstats.20240328: Permission denied 2024-03-28T14:04:06.394 INFO:teuthology.orchestra.run.smithi171.stderr:28 Mar 14:04:06 ntpd[17191]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-28T14:04:06.394 INFO:teuthology.orchestra.run.smithi171.stderr:28 Mar 14:04:06 ntpd[17191]: can't open /var/log/ntpstats/rawstats.20240328: Permission denied 2024-03-28T14:04:06.394 INFO:teuthology.orchestra.run.smithi171.stderr:28 Mar 14:04:06 ntpd[17191]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-28T14:04:06.394 INFO:teuthology.orchestra.run.smithi171.stderr:28 Mar 14:04:06 ntpd[17191]: can't open /var/log/ntpstats/rawstats.20240328: Permission denied 2024-03-28T14:04:06.394 INFO:teuthology.orchestra.run.smithi171.stderr:28 Mar 14:04:06 ntpd[17191]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-28T14:04:06.394 INFO:teuthology.orchestra.run.smithi171.stderr:28 Mar 14:04:06 ntpd[17191]: can't open /var/log/ntpstats/rawstats.20240328: Permission denied 2024-03-28T14:04:06.394 INFO:teuthology.orchestra.run.smithi171.stderr:28 Mar 14:04:06 ntpd[17191]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-28T14:04:06.394 INFO:teuthology.orchestra.run.smithi171.stderr:28 Mar 14:04:06 ntpd[17191]: can't open /var/log/ntpstats/rawstats.20240328: Permission denied 2024-03-28T14:04:06.394 INFO:teuthology.orchestra.run.smithi171.stderr:28 Mar 14:04:06 ntpd[17191]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-03-28T14:04:06.394 INFO:teuthology.orchestra.run.smithi171.stderr:28 Mar 14:04:06 ntpd[17191]: can't open /var/log/ntpstats/peerstats.20240328: Permission denied 2024-03-28T14:04:06.395 INFO:teuthology.orchestra.run.smithi171.stderr:28 Mar 14:04:06 ntpd[17191]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-28T14:04:06.395 INFO:teuthology.orchestra.run.smithi171.stderr:28 Mar 14:04:06 ntpd[17191]: can't open /var/log/ntpstats/rawstats.20240328: Permission denied 2024-03-28T14:04:06.395 INFO:teuthology.orchestra.run.smithi171.stderr:28 Mar 14:04:06 ntpd[17191]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-03-28T14:04:06.395 INFO:teuthology.orchestra.run.smithi171.stderr:28 Mar 14:04:06 ntpd[17191]: can't open /var/log/ntpstats/peerstats.20240328: Permission denied 2024-03-28T14:04:06.395 INFO:teuthology.orchestra.run.smithi171.stderr:28 Mar 14:04:06 ntpd[17191]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-28T14:04:06.395 INFO:teuthology.orchestra.run.smithi171.stderr:28 Mar 14:04:06 ntpd[17191]: can't open /var/log/ntpstats/rawstats.20240328: Permission denied 2024-03-28T14:04:06.395 INFO:teuthology.orchestra.run.smithi171.stderr:28 Mar 14:04:06 ntpd[17191]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-03-28T14:04:06.395 INFO:teuthology.orchestra.run.smithi171.stderr:28 Mar 14:04:06 ntpd[17191]: can't open /var/log/ntpstats/peerstats.20240328: Permission denied 2024-03-28T14:04:07.366 INFO:teuthology.orchestra.run.smithi104.stderr:28 Mar 14:04:07 ntpd[17165]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-28T14:04:07.366 INFO:teuthology.orchestra.run.smithi104.stderr:28 Mar 14:04:07 ntpd[17165]: can't open /var/log/ntpstats/rawstats.20240328: Permission denied 2024-03-28T14:04:07.366 INFO:teuthology.orchestra.run.smithi104.stderr:28 Mar 14:04:07 ntpd[17165]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-28T14:04:07.366 INFO:teuthology.orchestra.run.smithi104.stderr:28 Mar 14:04:07 ntpd[17165]: can't open /var/log/ntpstats/rawstats.20240328: Permission denied 2024-03-28T14:04:07.366 INFO:teuthology.orchestra.run.smithi104.stderr:28 Mar 14:04:07 ntpd[17165]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-03-28T14:04:07.366 INFO:teuthology.orchestra.run.smithi104.stderr:28 Mar 14:04:07 ntpd[17165]: can't open /var/log/ntpstats/peerstats.20240328: Permission denied 2024-03-28T14:04:08.366 INFO:teuthology.orchestra.run.smithi104.stderr:28 Mar 14:04:08 ntpd[17165]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-28T14:04:08.366 INFO:teuthology.orchestra.run.smithi104.stderr:28 Mar 14:04:08 ntpd[17165]: can't open /var/log/ntpstats/rawstats.20240328: Permission denied 2024-03-28T14:04:08.366 INFO:teuthology.orchestra.run.smithi104.stderr:28 Mar 14:04:08 ntpd[17165]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-28T14:04:08.366 INFO:teuthology.orchestra.run.smithi104.stderr:28 Mar 14:04:08 ntpd[17165]: can't open /var/log/ntpstats/rawstats.20240328: Permission denied 2024-03-28T14:04:08.366 INFO:teuthology.orchestra.run.smithi104.stderr:28 Mar 14:04:08 ntpd[17165]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-28T14:04:08.367 INFO:teuthology.orchestra.run.smithi104.stderr:28 Mar 14:04:08 ntpd[17165]: can't open /var/log/ntpstats/rawstats.20240328: Permission denied 2024-03-28T14:04:08.367 INFO:teuthology.orchestra.run.smithi104.stderr:28 Mar 14:04:08 ntpd[17165]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-03-28T14:04:08.367 INFO:teuthology.orchestra.run.smithi104.stderr:28 Mar 14:04:08 ntpd[17165]: can't open /var/log/ntpstats/peerstats.20240328: Permission denied 2024-03-28T14:04:08.367 INFO:teuthology.orchestra.run.smithi104.stdout:28 Mar 14:04:08 ntpd[17165]: ntpd: time slew +0.000651 s 2024-03-28T14:04:08.367 INFO:teuthology.orchestra.run.smithi104.stdout:ntpd: time slew +0.000651s 2024-03-28T14:04:08.367 INFO:teuthology.orchestra.run.smithi104.stderr:28 Mar 14:04:08 ntpd[17165]: couldn't unlink /var/log/ntpstats/loopstats: Permission denied 2024-03-28T14:04:08.368 INFO:teuthology.orchestra.run.smithi104.stderr:28 Mar 14:04:08 ntpd[17165]: can't open /var/log/ntpstats/loopstats.20240328: Permission denied 2024-03-28T14:04:08.393 INFO:teuthology.orchestra.run.smithi171.stderr:28 Mar 14:04:08 ntpd[17191]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-28T14:04:08.394 INFO:teuthology.orchestra.run.smithi171.stderr:28 Mar 14:04:08 ntpd[17191]: can't open /var/log/ntpstats/rawstats.20240328: Permission denied 2024-03-28T14:04:08.394 INFO:teuthology.orchestra.run.smithi171.stderr:28 Mar 14:04:08 ntpd[17191]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-28T14:04:08.394 INFO:teuthology.orchestra.run.smithi171.stderr:28 Mar 14:04:08 ntpd[17191]: can't open /var/log/ntpstats/rawstats.20240328: Permission denied 2024-03-28T14:04:08.394 INFO:teuthology.orchestra.run.smithi171.stderr:28 Mar 14:04:08 ntpd[17191]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-28T14:04:08.394 INFO:teuthology.orchestra.run.smithi171.stderr:28 Mar 14:04:08 ntpd[17191]: can't open /var/log/ntpstats/rawstats.20240328: Permission denied 2024-03-28T14:04:08.394 INFO:teuthology.orchestra.run.smithi171.stderr:28 Mar 14:04:08 ntpd[17191]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-28T14:04:08.394 INFO:teuthology.orchestra.run.smithi171.stderr:28 Mar 14:04:08 ntpd[17191]: can't open /var/log/ntpstats/rawstats.20240328: Permission denied 2024-03-28T14:04:08.394 INFO:teuthology.orchestra.run.smithi171.stderr:28 Mar 14:04:08 ntpd[17191]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-03-28T14:04:08.394 INFO:teuthology.orchestra.run.smithi171.stderr:28 Mar 14:04:08 ntpd[17191]: can't open /var/log/ntpstats/peerstats.20240328: Permission denied 2024-03-28T14:04:08.395 INFO:teuthology.orchestra.run.smithi171.stdout:28 Mar 14:04:08 ntpd[17191]: ntpd: time slew -0.001247 s 2024-03-28T14:04:08.395 INFO:teuthology.orchestra.run.smithi171.stdout:ntpd: time slew -0.001247s 2024-03-28T14:04:08.395 INFO:teuthology.orchestra.run.smithi171.stderr:28 Mar 14:04:08 ntpd[17191]: couldn't unlink /var/log/ntpstats/loopstats: Permission denied 2024-03-28T14:04:08.395 INFO:teuthology.orchestra.run.smithi171.stderr:28 Mar 14:04:08 ntpd[17191]: can't open /var/log/ntpstats/loopstats.20240328: Permission denied 2024-03-28T14:04:08.441 INFO:teuthology.orchestra.run.smithi171.stdout: remote refid st t when poll reach delay offset jitter 2024-03-28T14:04:08.441 INFO:teuthology.orchestra.run.smithi171.stdout:============================================================================== 2024-03-28T14:04:08.441 INFO:teuthology.orchestra.run.smithi171.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-03-28T14:04:08.441 INFO:teuthology.orchestra.run.smithi171.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-03-28T14:04:08.441 INFO:teuthology.orchestra.run.smithi171.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-03-28T14:04:08.441 INFO:teuthology.orchestra.run.smithi171.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-03-28T14:04:08.487 INFO:teuthology.orchestra.run.smithi104.stdout: remote refid st t when poll reach delay offset jitter 2024-03-28T14:04:08.487 INFO:teuthology.orchestra.run.smithi104.stdout:============================================================================== 2024-03-28T14:04:08.487 INFO:teuthology.orchestra.run.smithi104.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-03-28T14:04:08.487 INFO:teuthology.orchestra.run.smithi104.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-03-28T14:04:08.488 INFO:teuthology.orchestra.run.smithi104.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-03-28T14:04:08.488 INFO:teuthology.orchestra.run.smithi104.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-03-28T14:04:08.489 INFO:teuthology.run_tasks:Running task install... 2024-03-28T14:04:08.497 DEBUG:teuthology.task.install:project ceph 2024-03-28T14:04:08.497 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': 'e0e48ba6ee8c8b4c8f38d3b879d6eafbbb1efc15'}} 2024-03-28T14:04:08.497 DEBUG:teuthology.task.install:config {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'flavor': 'default', 'sha1': 'e0e48ba6ee8c8b4c8f38d3b879d6eafbbb1efc15'} 2024-03-28T14:04:08.497 INFO:teuthology.task.install:Using flavor: default 2024-03-28T14:04:08.503 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-03-28T14:04:08.503 INFO:teuthology.task.install:extra packages: {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']} 2024-03-28T14:04:08.503 DEBUG:teuthology.orchestra.run.smithi104:> sudo apt-key list | grep Ceph 2024-03-28T14:04:08.504 DEBUG:teuthology.orchestra.run.smithi171:> sudo apt-key list | grep Ceph 2024-03-28T14:04:08.560 INFO:teuthology.orchestra.run.smithi171.stderr:Warning: apt-key is deprecated. Manage keyring files in trusted.gpg.d instead (see apt-key(8)). 2024-03-28T14:04:08.589 INFO:teuthology.orchestra.run.smithi171.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2024-03-28T14:04:08.589 INFO:teuthology.orchestra.run.smithi171.stdout:uid [ unknown] Ceph.com (release key) 2024-03-28T14:04:08.589 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, ceph-volume, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror on remote deb x86_64 2024-03-28T14:04:08.590 INFO:teuthology.task.install.deb:Installing system (non-project) packages: bison, flex, libelf-dev, libssl-dev, network-manager, iproute2, util-linux, dump, indent, libaio-dev, libtool-bin, uuid-dev, xfslibs-dev, postgresql, postgresql-client, postgresql-common, postgresql-contrib on remote deb x86_64 2024-03-28T14:04:08.590 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=e0e48ba6ee8c8b4c8f38d3b879d6eafbbb1efc15 2024-03-28T14:04:08.594 INFO:teuthology.orchestra.run.smithi104.stderr:Warning: apt-key is deprecated. Manage keyring files in trusted.gpg.d instead (see apt-key(8)). 2024-03-28T14:04:08.624 INFO:teuthology.orchestra.run.smithi104.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2024-03-28T14:04:08.624 INFO:teuthology.orchestra.run.smithi104.stdout:uid [ unknown] Ceph.com (release key) 2024-03-28T14:04:08.624 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, ceph-volume, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror on remote deb x86_64 2024-03-28T14:04:08.624 INFO:teuthology.task.install.deb:Installing system (non-project) packages: bison, flex, libelf-dev, libssl-dev, network-manager, iproute2, util-linux, dump, indent, libaio-dev, libtool-bin, uuid-dev, xfslibs-dev, postgresql, postgresql-client, postgresql-common, postgresql-contrib on remote deb x86_64 2024-03-28T14:04:08.624 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=e0e48ba6ee8c8b4c8f38d3b879d6eafbbb1efc15 2024-03-28T14:04:08.733 INFO:teuthology.task.install.deb:Pulling from https://4.chacra.ceph.com/r/ceph/wip-batrick-testing-20240327.230800/e0e48ba6ee8c8b4c8f38d3b879d6eafbbb1efc15/ubuntu/jammy/flavors/default/ 2024-03-28T14:04:08.733 INFO:teuthology.task.install.deb:Package version is 19.0.0-2499-ge0e48ba6-1jammy 2024-03-28T14:04:08.762 INFO:teuthology.task.install.deb:Pulling from https://4.chacra.ceph.com/r/ceph/wip-batrick-testing-20240327.230800/e0e48ba6ee8c8b4c8f38d3b879d6eafbbb1efc15/ubuntu/jammy/flavors/default/ 2024-03-28T14:04:08.762 INFO:teuthology.task.install.deb:Package version is 19.0.0-2499-ge0e48ba6-1jammy 2024-03-28T14:04:08.872 DEBUG:teuthology.orchestra.run.smithi104:> set -ex 2024-03-28T14:04:08.873 DEBUG:teuthology.orchestra.run.smithi104:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2024-03-28T14:04:08.886 DEBUG:teuthology.orchestra.run.smithi104:> sudo apt-get update 2024-03-28T14:04:08.904 DEBUG:teuthology.orchestra.run.smithi171:> set -ex 2024-03-28T14:04:08.904 DEBUG:teuthology.orchestra.run.smithi171:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2024-03-28T14:04:08.918 DEBUG:teuthology.orchestra.run.smithi171:> sudo apt-get update 2024-03-28T14:04:09.089 INFO:teuthology.orchestra.run.smithi104.stdout:Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-03-28T14:04:09.090 INFO:teuthology.orchestra.run.smithi104.stdout:Hit:2 http://security.ubuntu.com/ubuntu jammy-security InRelease 2024-03-28T14:04:09.103 INFO:teuthology.orchestra.run.smithi104.stdout:Hit:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2024-03-28T14:04:09.123 INFO:teuthology.orchestra.run.smithi104.stdout:Hit:4 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2024-03-28T14:04:09.178 INFO:teuthology.orchestra.run.smithi104.stdout:Ign:5 https://4.chacra.ceph.com/r/ceph/wip-batrick-testing-20240327.230800/e0e48ba6ee8c8b4c8f38d3b879d6eafbbb1efc15/ubuntu/jammy/flavors/default jammy InRelease 2024-03-28T14:04:09.200 INFO:teuthology.orchestra.run.smithi171.stdout:Ign:1 https://4.chacra.ceph.com/r/ceph/wip-batrick-testing-20240327.230800/e0e48ba6ee8c8b4c8f38d3b879d6eafbbb1efc15/ubuntu/jammy/flavors/default jammy InRelease 2024-03-28T14:04:09.211 INFO:teuthology.orchestra.run.smithi171.stdout:Hit:2 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-03-28T14:04:09.211 INFO:teuthology.orchestra.run.smithi171.stdout:Hit:3 http://security.ubuntu.com/ubuntu jammy-security InRelease 2024-03-28T14:04:09.220 INFO:teuthology.orchestra.run.smithi104.stdout:Get:6 https://4.chacra.ceph.com/r/ceph/wip-batrick-testing-20240327.230800/e0e48ba6ee8c8b4c8f38d3b879d6eafbbb1efc15/ubuntu/jammy/flavors/default jammy Release [7,662 B] 2024-03-28T14:04:09.238 INFO:teuthology.orchestra.run.smithi171.stdout:Get:4 https://4.chacra.ceph.com/r/ceph/wip-batrick-testing-20240327.230800/e0e48ba6ee8c8b4c8f38d3b879d6eafbbb1efc15/ubuntu/jammy/flavors/default jammy Release [7,662 B] 2024-03-28T14:04:09.257 INFO:teuthology.orchestra.run.smithi104.stdout:Ign:7 https://4.chacra.ceph.com/r/ceph/wip-batrick-testing-20240327.230800/e0e48ba6ee8c8b4c8f38d3b879d6eafbbb1efc15/ubuntu/jammy/flavors/default jammy Release.gpg 2024-03-28T14:04:09.276 INFO:teuthology.orchestra.run.smithi171.stdout:Ign:5 https://4.chacra.ceph.com/r/ceph/wip-batrick-testing-20240327.230800/e0e48ba6ee8c8b4c8f38d3b879d6eafbbb1efc15/ubuntu/jammy/flavors/default jammy Release.gpg 2024-03-28T14:04:09.284 INFO:teuthology.orchestra.run.smithi171.stdout:Hit:6 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2024-03-28T14:04:09.294 INFO:teuthology.orchestra.run.smithi104.stdout:Get:8 https://4.chacra.ceph.com/r/ceph/wip-batrick-testing-20240327.230800/e0e48ba6ee8c8b4c8f38d3b879d6eafbbb1efc15/ubuntu/jammy/flavors/default jammy/main i386 Packages [3,412 B] 2024-03-28T14:04:09.312 INFO:teuthology.orchestra.run.smithi171.stdout:Get:7 https://4.chacra.ceph.com/r/ceph/wip-batrick-testing-20240327.230800/e0e48ba6ee8c8b4c8f38d3b879d6eafbbb1efc15/ubuntu/jammy/flavors/default jammy/main i386 Packages [3,412 B] 2024-03-28T14:04:09.338 INFO:teuthology.orchestra.run.smithi104.stdout:Get:9 https://4.chacra.ceph.com/r/ceph/wip-batrick-testing-20240327.230800/e0e48ba6ee8c8b4c8f38d3b879d6eafbbb1efc15/ubuntu/jammy/flavors/default jammy/main amd64 Packages [18.1 kB] 2024-03-28T14:04:09.354 INFO:teuthology.orchestra.run.smithi171.stdout:Get:8 https://4.chacra.ceph.com/r/ceph/wip-batrick-testing-20240327.230800/e0e48ba6ee8c8b4c8f38d3b879d6eafbbb1efc15/ubuntu/jammy/flavors/default jammy/main amd64 Packages [18.1 kB] 2024-03-28T14:04:09.366 INFO:teuthology.orchestra.run.smithi171.stdout:Hit:9 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2024-03-28T14:04:09.460 INFO:teuthology.orchestra.run.smithi104.stdout:Fetched 29.1 kB in 0s (72.9 kB/s) 2024-03-28T14:04:09.576 INFO:teuthology.orchestra.run.smithi171.stdout:Fetched 29.1 kB in 0s (60.2 kB/s) 2024-03-28T14:04:10.388 INFO:teuthology.orchestra.run.smithi104.stdout:Reading package lists... 2024-03-28T14:04:10.404 DEBUG:teuthology.orchestra.run.smithi104:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=19.0.0-2499-ge0e48ba6-1jammy cephadm=19.0.0-2499-ge0e48ba6-1jammy ceph-mds=19.0.0-2499-ge0e48ba6-1jammy ceph-mgr=19.0.0-2499-ge0e48ba6-1jammy ceph-common=19.0.0-2499-ge0e48ba6-1jammy ceph-fuse=19.0.0-2499-ge0e48ba6-1jammy ceph-test=19.0.0-2499-ge0e48ba6-1jammy ceph-volume=19.0.0-2499-ge0e48ba6-1jammy radosgw=19.0.0-2499-ge0e48ba6-1jammy python3-rados=19.0.0-2499-ge0e48ba6-1jammy python3-rgw=19.0.0-2499-ge0e48ba6-1jammy python3-cephfs=19.0.0-2499-ge0e48ba6-1jammy python3-rbd=19.0.0-2499-ge0e48ba6-1jammy libcephfs2=19.0.0-2499-ge0e48ba6-1jammy libcephfs-dev=19.0.0-2499-ge0e48ba6-1jammy librados2=19.0.0-2499-ge0e48ba6-1jammy librbd1=19.0.0-2499-ge0e48ba6-1jammy rbd-fuse=19.0.0-2499-ge0e48ba6-1jammy python3-cephfs=19.0.0-2499-ge0e48ba6-1jammy cephfs-shell=19.0.0-2499-ge0e48ba6-1jammy cephfs-top=19.0.0-2499-ge0e48ba6-1jammy cephfs-mirror=19.0.0-2499-ge0e48ba6-1jammy 2024-03-28T14:04:10.463 INFO:teuthology.orchestra.run.smithi104.stdout:Reading package lists... 2024-03-28T14:04:10.482 INFO:teuthology.orchestra.run.smithi171.stdout:Reading package lists... 2024-03-28T14:04:10.498 DEBUG:teuthology.orchestra.run.smithi171:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=19.0.0-2499-ge0e48ba6-1jammy cephadm=19.0.0-2499-ge0e48ba6-1jammy ceph-mds=19.0.0-2499-ge0e48ba6-1jammy ceph-mgr=19.0.0-2499-ge0e48ba6-1jammy ceph-common=19.0.0-2499-ge0e48ba6-1jammy ceph-fuse=19.0.0-2499-ge0e48ba6-1jammy ceph-test=19.0.0-2499-ge0e48ba6-1jammy ceph-volume=19.0.0-2499-ge0e48ba6-1jammy radosgw=19.0.0-2499-ge0e48ba6-1jammy python3-rados=19.0.0-2499-ge0e48ba6-1jammy python3-rgw=19.0.0-2499-ge0e48ba6-1jammy python3-cephfs=19.0.0-2499-ge0e48ba6-1jammy python3-rbd=19.0.0-2499-ge0e48ba6-1jammy libcephfs2=19.0.0-2499-ge0e48ba6-1jammy libcephfs-dev=19.0.0-2499-ge0e48ba6-1jammy librados2=19.0.0-2499-ge0e48ba6-1jammy librbd1=19.0.0-2499-ge0e48ba6-1jammy rbd-fuse=19.0.0-2499-ge0e48ba6-1jammy python3-cephfs=19.0.0-2499-ge0e48ba6-1jammy cephfs-shell=19.0.0-2499-ge0e48ba6-1jammy cephfs-top=19.0.0-2499-ge0e48ba6-1jammy cephfs-mirror=19.0.0-2499-ge0e48ba6-1jammy 2024-03-28T14:04:10.558 INFO:teuthology.orchestra.run.smithi171.stdout:Reading package lists... 2024-03-28T14:04:10.659 INFO:teuthology.orchestra.run.smithi104.stdout:Building dependency tree... 2024-03-28T14:04:10.659 INFO:teuthology.orchestra.run.smithi104.stdout:Reading state information... 2024-03-28T14:04:10.747 INFO:teuthology.orchestra.run.smithi171.stdout:Building dependency tree... 2024-03-28T14:04:10.748 INFO:teuthology.orchestra.run.smithi171.stdout:Reading state information... 2024-03-28T14:04:10.842 INFO:teuthology.orchestra.run.smithi104.stdout:The following packages were automatically installed and are no longer required: 2024-03-28T14:04:10.842 INFO:teuthology.orchestra.run.smithi104.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-03-28T14:04:10.842 INFO:teuthology.orchestra.run.smithi104.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-03-28T14:04:10.843 INFO:teuthology.orchestra.run.smithi104.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-03-28T14:04:10.843 INFO:teuthology.orchestra.run.smithi104.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2024-03-28T14:04:10.843 INFO:teuthology.orchestra.run.smithi104.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2024-03-28T14:04:10.843 INFO:teuthology.orchestra.run.smithi104.stdout:Use 'sudo apt autoremove' to remove them. 2024-03-28T14:04:10.844 INFO:teuthology.orchestra.run.smithi104.stdout:The following additional packages will be installed: 2024-03-28T14:04:10.845 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-base ceph-mgr-cephadm ceph-mgr-dashboard ceph-mgr-diskprediction-local 2024-03-28T14:04:10.845 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-mgr-k8sevents ceph-mgr-modules-core ceph-mon ceph-osd jq 2024-03-28T14:04:10.845 INFO:teuthology.orchestra.run.smithi104.stdout: libdouble-conversion3 libjq1 liblttng-ust1 liblua5.3-dev liboath0 libonig5 2024-03-28T14:04:10.845 INFO:teuthology.orchestra.run.smithi104.stdout: libpcre2-16-0 libqt5core5a libqt5dbus5 libqt5network5 libradosstriper1 2024-03-28T14:04:10.845 INFO:teuthology.orchestra.run.smithi104.stdout: librdkafka1 libreadline-dev librgw2 libsqlite3-mod-ceph libthrift-0.16.0 2024-03-28T14:04:10.845 INFO:teuthology.orchestra.run.smithi104.stdout: lua-any lua-sec lua-socket lua5.1 luarocks nvme-cli pkg-config 2024-03-28T14:04:10.845 INFO:teuthology.orchestra.run.smithi104.stdout: python-asyncssh-doc python-babel-localedata python-pastedeploy-tpl 2024-03-28T14:04:10.846 INFO:teuthology.orchestra.run.smithi104.stdout: python3-asyncssh python3-babel python3-bcrypt python3-cachetools 2024-03-28T14:04:10.846 INFO:teuthology.orchestra.run.smithi104.stdout: python3-ceph-argparse python3-ceph-common python3-certifi python3-cheroot 2024-03-28T14:04:10.846 INFO:teuthology.orchestra.run.smithi104.stdout: python3-cherrypy3 python3-cmd2 python3-colorama python3-google-auth 2024-03-28T14:04:10.846 INFO:teuthology.orchestra.run.smithi104.stdout: python3-idna python3-iniconfig python3-jaraco.classes 2024-03-28T14:04:10.846 INFO:teuthology.orchestra.run.smithi104.stdout: python3-jaraco.collections python3-jaraco.functools python3-jaraco.text 2024-03-28T14:04:10.846 INFO:teuthology.orchestra.run.smithi104.stdout: python3-jinja2 python3-joblib python3-kubernetes python3-logutils 2024-03-28T14:04:10.846 INFO:teuthology.orchestra.run.smithi104.stdout: python3-mako python3-markupsafe python3-natsort python3-openssl 2024-03-28T14:04:10.846 INFO:teuthology.orchestra.run.smithi104.stdout: python3-paste python3-pastedeploy python3-pastescript python3-pecan 2024-03-28T14:04:10.847 INFO:teuthology.orchestra.run.smithi104.stdout: python3-pluggy python3-portend python3-prettytable python3-psutil python3-py 2024-03-28T14:04:10.847 INFO:teuthology.orchestra.run.smithi104.stdout: python3-pyasn1 python3-pyasn1-modules python3-pygments python3-pyinotify 2024-03-28T14:04:10.847 INFO:teuthology.orchestra.run.smithi104.stdout: python3-pyperclip python3-pytest python3-repoze.lru python3-requests 2024-03-28T14:04:10.847 INFO:teuthology.orchestra.run.smithi104.stdout: python3-requests-oauthlib python3-routes python3-rsa python3-simplegeneric 2024-03-28T14:04:10.847 INFO:teuthology.orchestra.run.smithi104.stdout: python3-simplejson python3-singledispatch python3-sklearn 2024-03-28T14:04:10.847 INFO:teuthology.orchestra.run.smithi104.stdout: python3-sklearn-lib python3-tempita python3-tempora python3-threadpoolctl 2024-03-28T14:04:10.847 INFO:teuthology.orchestra.run.smithi104.stdout: python3-toml python3-urllib3 python3-waitress python3-wcwidth python3-webob 2024-03-28T14:04:10.847 INFO:teuthology.orchestra.run.smithi104.stdout: python3-websocket python3-webtest python3-werkzeug python3-zc.lockfile 2024-03-28T14:04:10.847 INFO:teuthology.orchestra.run.smithi104.stdout: qttranslations5-l10n socat unzip xmlstarlet zip 2024-03-28T14:04:10.847 INFO:teuthology.orchestra.run.smithi104.stdout:Suggested packages: 2024-03-28T14:04:10.848 INFO:teuthology.orchestra.run.smithi104.stdout: python3-influxdb readline-doc python-jinja2-doc python3-beaker 2024-03-28T14:04:10.848 INFO:teuthology.orchestra.run.smithi104.stdout: python-mako-doc python-natsort-doc python-openssl-doc python3-openssl-dbg 2024-03-28T14:04:10.848 INFO:teuthology.orchestra.run.smithi104.stdout: httpd-wsgi libapache2-mod-python libapache2-mod-scgi libjs-mochikit 2024-03-28T14:04:10.848 INFO:teuthology.orchestra.run.smithi104.stdout: python-pecan-doc python-psutil-doc subversion python-pygments-doc 2024-03-28T14:04:10.848 INFO:teuthology.orchestra.run.smithi104.stdout: python-pyinotify-doc python3-socks python-requests-doc python3-dap 2024-03-28T14:04:10.848 INFO:teuthology.orchestra.run.smithi104.stdout: python-sklearn-doc ipython3 python-waitress-doc python-webob-doc 2024-03-28T14:04:10.848 INFO:teuthology.orchestra.run.smithi104.stdout: python-webtest-doc python-werkzeug-doc python3-watchdog 2024-03-28T14:04:10.848 INFO:teuthology.orchestra.run.smithi104.stdout:Recommended packages: 2024-03-28T14:04:10.848 INFO:teuthology.orchestra.run.smithi104.stdout: btrfs-tools 2024-03-28T14:04:10.929 INFO:teuthology.orchestra.run.smithi171.stdout:The following packages were automatically installed and are no longer required: 2024-03-28T14:04:10.929 INFO:teuthology.orchestra.run.smithi171.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-03-28T14:04:10.929 INFO:teuthology.orchestra.run.smithi171.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-03-28T14:04:10.929 INFO:teuthology.orchestra.run.smithi171.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-03-28T14:04:10.930 INFO:teuthology.orchestra.run.smithi171.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2024-03-28T14:04:10.930 INFO:teuthology.orchestra.run.smithi171.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2024-03-28T14:04:10.930 INFO:teuthology.orchestra.run.smithi171.stdout:Use 'sudo apt autoremove' to remove them. 2024-03-28T14:04:10.931 INFO:teuthology.orchestra.run.smithi171.stdout:The following additional packages will be installed: 2024-03-28T14:04:10.931 INFO:teuthology.orchestra.run.smithi171.stdout: ceph-base ceph-mgr-cephadm ceph-mgr-dashboard ceph-mgr-diskprediction-local 2024-03-28T14:04:10.931 INFO:teuthology.orchestra.run.smithi171.stdout: ceph-mgr-k8sevents ceph-mgr-modules-core ceph-mon ceph-osd jq 2024-03-28T14:04:10.931 INFO:teuthology.orchestra.run.smithi171.stdout: libdouble-conversion3 libjq1 liblttng-ust1 liblua5.3-dev liboath0 libonig5 2024-03-28T14:04:10.931 INFO:teuthology.orchestra.run.smithi171.stdout: libpcre2-16-0 libqt5core5a libqt5dbus5 libqt5network5 libradosstriper1 2024-03-28T14:04:10.932 INFO:teuthology.orchestra.run.smithi171.stdout: librdkafka1 libreadline-dev librgw2 libsqlite3-mod-ceph libthrift-0.16.0 2024-03-28T14:04:10.932 INFO:teuthology.orchestra.run.smithi171.stdout: lua-any lua-sec lua-socket lua5.1 luarocks nvme-cli pkg-config 2024-03-28T14:04:10.932 INFO:teuthology.orchestra.run.smithi171.stdout: python-asyncssh-doc python-babel-localedata python-pastedeploy-tpl 2024-03-28T14:04:10.932 INFO:teuthology.orchestra.run.smithi171.stdout: python3-asyncssh python3-babel python3-bcrypt python3-cachetools 2024-03-28T14:04:10.932 INFO:teuthology.orchestra.run.smithi171.stdout: python3-ceph-argparse python3-ceph-common python3-certifi python3-cheroot 2024-03-28T14:04:10.932 INFO:teuthology.orchestra.run.smithi171.stdout: python3-cherrypy3 python3-cmd2 python3-colorama python3-google-auth 2024-03-28T14:04:10.932 INFO:teuthology.orchestra.run.smithi171.stdout: python3-idna python3-iniconfig python3-jaraco.classes 2024-03-28T14:04:10.933 INFO:teuthology.orchestra.run.smithi171.stdout: python3-jaraco.collections python3-jaraco.functools python3-jaraco.text 2024-03-28T14:04:10.933 INFO:teuthology.orchestra.run.smithi171.stdout: python3-jinja2 python3-joblib python3-kubernetes python3-logutils 2024-03-28T14:04:10.933 INFO:teuthology.orchestra.run.smithi171.stdout: python3-mako python3-markupsafe python3-natsort python3-openssl 2024-03-28T14:04:10.933 INFO:teuthology.orchestra.run.smithi171.stdout: python3-paste python3-pastedeploy python3-pastescript python3-pecan 2024-03-28T14:04:10.933 INFO:teuthology.orchestra.run.smithi171.stdout: python3-pluggy python3-portend python3-prettytable python3-psutil python3-py 2024-03-28T14:04:10.933 INFO:teuthology.orchestra.run.smithi171.stdout: python3-pyasn1 python3-pyasn1-modules python3-pygments python3-pyinotify 2024-03-28T14:04:10.933 INFO:teuthology.orchestra.run.smithi171.stdout: python3-pyperclip python3-pytest python3-repoze.lru python3-requests 2024-03-28T14:04:10.933 INFO:teuthology.orchestra.run.smithi171.stdout: python3-requests-oauthlib python3-routes python3-rsa python3-simplegeneric 2024-03-28T14:04:10.933 INFO:teuthology.orchestra.run.smithi171.stdout: python3-simplejson python3-singledispatch python3-sklearn 2024-03-28T14:04:10.933 INFO:teuthology.orchestra.run.smithi171.stdout: python3-sklearn-lib python3-tempita python3-tempora python3-threadpoolctl 2024-03-28T14:04:10.933 INFO:teuthology.orchestra.run.smithi171.stdout: python3-toml python3-urllib3 python3-waitress python3-wcwidth python3-webob 2024-03-28T14:04:10.933 INFO:teuthology.orchestra.run.smithi171.stdout: python3-websocket python3-webtest python3-werkzeug python3-zc.lockfile 2024-03-28T14:04:10.933 INFO:teuthology.orchestra.run.smithi171.stdout: qttranslations5-l10n socat unzip xmlstarlet zip 2024-03-28T14:04:10.934 INFO:teuthology.orchestra.run.smithi171.stdout:Suggested packages: 2024-03-28T14:04:10.934 INFO:teuthology.orchestra.run.smithi171.stdout: python3-influxdb readline-doc python-jinja2-doc python3-beaker 2024-03-28T14:04:10.934 INFO:teuthology.orchestra.run.smithi171.stdout: python-mako-doc python-natsort-doc python-openssl-doc python3-openssl-dbg 2024-03-28T14:04:10.934 INFO:teuthology.orchestra.run.smithi171.stdout: httpd-wsgi libapache2-mod-python libapache2-mod-scgi libjs-mochikit 2024-03-28T14:04:10.934 INFO:teuthology.orchestra.run.smithi171.stdout: python-pecan-doc python-psutil-doc subversion python-pygments-doc 2024-03-28T14:04:10.934 INFO:teuthology.orchestra.run.smithi171.stdout: python-pyinotify-doc python3-socks python-requests-doc python3-dap 2024-03-28T14:04:10.934 INFO:teuthology.orchestra.run.smithi171.stdout: python-sklearn-doc ipython3 python-waitress-doc python-webob-doc 2024-03-28T14:04:10.934 INFO:teuthology.orchestra.run.smithi171.stdout: python-webtest-doc python-werkzeug-doc python3-watchdog 2024-03-28T14:04:10.934 INFO:teuthology.orchestra.run.smithi171.stdout:Recommended packages: 2024-03-28T14:04:10.934 INFO:teuthology.orchestra.run.smithi171.stdout: btrfs-tools 2024-03-28T14:04:10.953 INFO:teuthology.orchestra.run.smithi104.stdout:The following NEW packages will be installed: 2024-03-28T14:04:10.953 INFO:teuthology.orchestra.run.smithi104.stdout: ceph ceph-base ceph-common ceph-fuse ceph-mds ceph-mgr ceph-mgr-cephadm 2024-03-28T14:04:10.953 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-k8sevents 2024-03-28T14:04:10.953 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-mgr-modules-core ceph-mon ceph-osd ceph-test ceph-volume cephadm 2024-03-28T14:04:10.954 INFO:teuthology.orchestra.run.smithi104.stdout: cephfs-mirror cephfs-shell cephfs-top jq libcephfs-dev libcephfs2 2024-03-28T14:04:10.954 INFO:teuthology.orchestra.run.smithi104.stdout: libdouble-conversion3 libjq1 liblttng-ust1 liblua5.3-dev liboath0 libonig5 2024-03-28T14:04:10.954 INFO:teuthology.orchestra.run.smithi104.stdout: libpcre2-16-0 libqt5core5a libqt5dbus5 libqt5network5 librados2 2024-03-28T14:04:10.955 INFO:teuthology.orchestra.run.smithi104.stdout: libradosstriper1 librbd1 librdkafka1 libreadline-dev librgw2 2024-03-28T14:04:10.955 INFO:teuthology.orchestra.run.smithi104.stdout: libsqlite3-mod-ceph libthrift-0.16.0 lua-any lua-sec lua-socket lua5.1 2024-03-28T14:04:10.955 INFO:teuthology.orchestra.run.smithi104.stdout: luarocks nvme-cli pkg-config python-asyncssh-doc python-babel-localedata 2024-03-28T14:04:10.955 INFO:teuthology.orchestra.run.smithi104.stdout: python-pastedeploy-tpl python3-asyncssh python3-babel python3-bcrypt 2024-03-28T14:04:10.955 INFO:teuthology.orchestra.run.smithi104.stdout: python3-cachetools python3-ceph-argparse python3-ceph-common python3-cephfs 2024-03-28T14:04:10.955 INFO:teuthology.orchestra.run.smithi104.stdout: python3-certifi python3-cheroot python3-cherrypy3 python3-cmd2 2024-03-28T14:04:10.955 INFO:teuthology.orchestra.run.smithi104.stdout: python3-colorama python3-google-auth python3-idna python3-iniconfig 2024-03-28T14:04:10.955 INFO:teuthology.orchestra.run.smithi104.stdout: python3-jaraco.classes python3-jaraco.collections python3-jaraco.functools 2024-03-28T14:04:10.955 INFO:teuthology.orchestra.run.smithi104.stdout: python3-jaraco.text python3-jinja2 python3-joblib python3-kubernetes 2024-03-28T14:04:10.955 INFO:teuthology.orchestra.run.smithi104.stdout: python3-logutils python3-mako python3-markupsafe python3-natsort 2024-03-28T14:04:10.955 INFO:teuthology.orchestra.run.smithi104.stdout: python3-openssl python3-paste python3-pastedeploy python3-pastescript 2024-03-28T14:04:10.955 INFO:teuthology.orchestra.run.smithi104.stdout: python3-pecan python3-pluggy python3-portend python3-prettytable 2024-03-28T14:04:10.955 INFO:teuthology.orchestra.run.smithi104.stdout: python3-psutil python3-py python3-pyasn1 python3-pyasn1-modules 2024-03-28T14:04:10.955 INFO:teuthology.orchestra.run.smithi104.stdout: python3-pygments python3-pyinotify python3-pyperclip python3-pytest 2024-03-28T14:04:10.956 INFO:teuthology.orchestra.run.smithi104.stdout: python3-rados python3-rbd python3-repoze.lru python3-requests 2024-03-28T14:04:10.956 INFO:teuthology.orchestra.run.smithi104.stdout: python3-requests-oauthlib python3-rgw python3-routes python3-rsa 2024-03-28T14:04:10.956 INFO:teuthology.orchestra.run.smithi104.stdout: python3-simplegeneric python3-simplejson python3-singledispatch 2024-03-28T14:04:10.956 INFO:teuthology.orchestra.run.smithi104.stdout: python3-sklearn python3-sklearn-lib python3-tempita python3-tempora 2024-03-28T14:04:10.956 INFO:teuthology.orchestra.run.smithi104.stdout: python3-threadpoolctl python3-toml python3-urllib3 python3-waitress 2024-03-28T14:04:10.956 INFO:teuthology.orchestra.run.smithi104.stdout: python3-wcwidth python3-webob python3-websocket python3-webtest 2024-03-28T14:04:10.956 INFO:teuthology.orchestra.run.smithi104.stdout: python3-werkzeug python3-zc.lockfile qttranslations5-l10n radosgw rbd-fuse 2024-03-28T14:04:10.956 INFO:teuthology.orchestra.run.smithi104.stdout: socat unzip xmlstarlet zip 2024-03-28T14:04:11.041 INFO:teuthology.orchestra.run.smithi171.stdout:The following NEW packages will be installed: 2024-03-28T14:04:11.041 INFO:teuthology.orchestra.run.smithi171.stdout: ceph ceph-base ceph-common ceph-fuse ceph-mds ceph-mgr ceph-mgr-cephadm 2024-03-28T14:04:11.041 INFO:teuthology.orchestra.run.smithi171.stdout: ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-k8sevents 2024-03-28T14:04:11.041 INFO:teuthology.orchestra.run.smithi171.stdout: ceph-mgr-modules-core ceph-mon ceph-osd ceph-test ceph-volume cephadm 2024-03-28T14:04:11.041 INFO:teuthology.orchestra.run.smithi171.stdout: cephfs-mirror cephfs-shell cephfs-top jq libcephfs-dev libcephfs2 2024-03-28T14:04:11.041 INFO:teuthology.orchestra.run.smithi171.stdout: libdouble-conversion3 libjq1 liblttng-ust1 liblua5.3-dev liboath0 libonig5 2024-03-28T14:04:11.041 INFO:teuthology.orchestra.run.smithi171.stdout: libpcre2-16-0 libqt5core5a libqt5dbus5 libqt5network5 librados2 2024-03-28T14:04:11.041 INFO:teuthology.orchestra.run.smithi171.stdout: libradosstriper1 librbd1 librdkafka1 libreadline-dev librgw2 2024-03-28T14:04:11.041 INFO:teuthology.orchestra.run.smithi171.stdout: libsqlite3-mod-ceph libthrift-0.16.0 lua-any lua-sec lua-socket lua5.1 2024-03-28T14:04:11.042 INFO:teuthology.orchestra.run.smithi171.stdout: luarocks nvme-cli pkg-config python-asyncssh-doc python-babel-localedata 2024-03-28T14:04:11.043 INFO:teuthology.orchestra.run.smithi171.stdout: python-pastedeploy-tpl python3-asyncssh python3-babel python3-bcrypt 2024-03-28T14:04:11.043 INFO:teuthology.orchestra.run.smithi171.stdout: python3-cachetools python3-ceph-argparse python3-ceph-common python3-cephfs 2024-03-28T14:04:11.043 INFO:teuthology.orchestra.run.smithi171.stdout: python3-certifi python3-cheroot python3-cherrypy3 python3-cmd2 2024-03-28T14:04:11.043 INFO:teuthology.orchestra.run.smithi171.stdout: python3-colorama python3-google-auth python3-idna python3-iniconfig 2024-03-28T14:04:11.043 INFO:teuthology.orchestra.run.smithi171.stdout: python3-jaraco.classes python3-jaraco.collections python3-jaraco.functools 2024-03-28T14:04:11.043 INFO:teuthology.orchestra.run.smithi171.stdout: python3-jaraco.text python3-jinja2 python3-joblib python3-kubernetes 2024-03-28T14:04:11.043 INFO:teuthology.orchestra.run.smithi171.stdout: python3-logutils python3-mako python3-markupsafe python3-natsort 2024-03-28T14:04:11.043 INFO:teuthology.orchestra.run.smithi171.stdout: python3-openssl python3-paste python3-pastedeploy python3-pastescript 2024-03-28T14:04:11.043 INFO:teuthology.orchestra.run.smithi171.stdout: python3-pecan python3-pluggy python3-portend python3-prettytable 2024-03-28T14:04:11.043 INFO:teuthology.orchestra.run.smithi171.stdout: python3-psutil python3-py python3-pyasn1 python3-pyasn1-modules 2024-03-28T14:04:11.043 INFO:teuthology.orchestra.run.smithi171.stdout: python3-pygments python3-pyinotify python3-pyperclip python3-pytest 2024-03-28T14:04:11.043 INFO:teuthology.orchestra.run.smithi171.stdout: python3-rados python3-rbd python3-repoze.lru python3-requests 2024-03-28T14:04:11.043 INFO:teuthology.orchestra.run.smithi171.stdout: python3-requests-oauthlib python3-rgw python3-routes python3-rsa 2024-03-28T14:04:11.043 INFO:teuthology.orchestra.run.smithi171.stdout: python3-simplegeneric python3-simplejson python3-singledispatch 2024-03-28T14:04:11.044 INFO:teuthology.orchestra.run.smithi171.stdout: python3-sklearn python3-sklearn-lib python3-tempita python3-tempora 2024-03-28T14:04:11.044 INFO:teuthology.orchestra.run.smithi171.stdout: python3-threadpoolctl python3-toml python3-urllib3 python3-waitress 2024-03-28T14:04:11.044 INFO:teuthology.orchestra.run.smithi171.stdout: python3-wcwidth python3-webob python3-websocket python3-webtest 2024-03-28T14:04:11.044 INFO:teuthology.orchestra.run.smithi171.stdout: python3-werkzeug python3-zc.lockfile qttranslations5-l10n radosgw rbd-fuse 2024-03-28T14:04:11.044 INFO:teuthology.orchestra.run.smithi171.stdout: socat unzip xmlstarlet zip 2024-03-28T14:04:11.109 INFO:teuthology.orchestra.run.smithi171.stdout:0 upgraded, 124 newly installed, 0 to remove and 298 not upgraded. 2024-03-28T14:04:11.109 INFO:teuthology.orchestra.run.smithi171.stdout:Need to get 179 MB of archives. 2024-03-28T14:04:11.110 INFO:teuthology.orchestra.run.smithi171.stdout:After this operation, 821 MB of additional disk space will be used. 2024-03-28T14:04:11.110 INFO:teuthology.orchestra.run.smithi171.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyperclip all 1.8.2-2 [11.5 kB] 2024-03-28T14:04:11.131 INFO:teuthology.orchestra.run.smithi171.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-wcwidth all 0.2.5+dfsg1-1 [21.9 kB] 2024-03-28T14:04:11.141 INFO:teuthology.orchestra.run.smithi104.stdout:0 upgraded, 124 newly installed, 0 to remove and 298 not upgraded. 2024-03-28T14:04:11.141 INFO:teuthology.orchestra.run.smithi104.stdout:Need to get 179 MB of archives. 2024-03-28T14:04:11.141 INFO:teuthology.orchestra.run.smithi104.stdout:After this operation, 821 MB of additional disk space will be used. 2024-03-28T14:04:11.141 INFO:teuthology.orchestra.run.smithi104.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyperclip all 1.8.2-2 [11.5 kB] 2024-03-28T14:04:11.144 INFO:teuthology.orchestra.run.smithi171.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cmd2 all 2.3.3-1 [124 kB] 2024-03-28T14:04:11.171 INFO:teuthology.orchestra.run.smithi104.stdout:Get:2 https://4.chacra.ceph.com/r/ceph/wip-batrick-testing-20240327.230800/e0e48ba6ee8c8b4c8f38d3b879d6eafbbb1efc15/ubuntu/jammy/flavors/default jammy/main amd64 librados2 amd64 19.0.0-2499-ge0e48ba6-1jammy [3,553 kB] 2024-03-28T14:04:11.181 INFO:teuthology.orchestra.run.smithi171.stdout:Get:4 https://4.chacra.ceph.com/r/ceph/wip-batrick-testing-20240327.230800/e0e48ba6ee8c8b4c8f38d3b879d6eafbbb1efc15/ubuntu/jammy/flavors/default jammy/main amd64 librados2 amd64 19.0.0-2499-ge0e48ba6-1jammy [3,553 kB] 2024-03-28T14:04:11.196 INFO:teuthology.orchestra.run.smithi171.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblttng-ust1 amd64 2.13.1-1ubuntu1 [190 kB] 2024-03-28T14:04:11.219 INFO:teuthology.orchestra.run.smithi171.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libdouble-conversion3 amd64 3.1.7-4 [39.0 kB] 2024-03-28T14:04:11.223 INFO:teuthology.orchestra.run.smithi171.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libpcre2-16-0 amd64 10.39-3ubuntu0.1 [203 kB] 2024-03-28T14:04:11.224 INFO:teuthology.orchestra.run.smithi104.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-wcwidth all 0.2.5+dfsg1-1 [21.9 kB] 2024-03-28T14:04:11.240 INFO:teuthology.orchestra.run.smithi171.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5core5a amd64 5.15.3+dfsg-2ubuntu0.2 [2,006 kB] 2024-03-28T14:04:11.269 INFO:teuthology.orchestra.run.smithi104.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cmd2 all 2.3.3-1 [124 kB] 2024-03-28T14:04:11.301 INFO:teuthology.orchestra.run.smithi171.stdout:Get:9 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5dbus5 amd64 5.15.3+dfsg-2ubuntu0.2 [222 kB] 2024-03-28T14:04:11.306 INFO:teuthology.orchestra.run.smithi171.stdout:Get:10 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5network5 amd64 5.15.3+dfsg-2ubuntu0.2 [731 kB] 2024-03-28T14:04:11.321 INFO:teuthology.orchestra.run.smithi171.stdout:Get:11 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libthrift-0.16.0 amd64 0.16.0-2 [267 kB] 2024-03-28T14:04:11.326 INFO:teuthology.orchestra.run.smithi171.stdout:Get:12 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-prettytable all 2.5.0-2 [31.3 kB] 2024-03-28T14:04:11.327 INFO:teuthology.orchestra.run.smithi171.stdout:Get:13 http://archive.ubuntu.com/ubuntu jammy/universe amd64 librdkafka1 amd64 1.8.0-1build1 [633 kB] 2024-03-28T14:04:11.337 INFO:teuthology.orchestra.run.smithi171.stdout:Get:14 http://archive.ubuntu.com/ubuntu jammy/main amd64 libreadline-dev amd64 8.1.2-1 [166 kB] 2024-03-28T14:04:11.339 INFO:teuthology.orchestra.run.smithi171.stdout:Get:15 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblua5.3-dev amd64 5.3.6-1build1 [167 kB] 2024-03-28T14:04:11.342 INFO:teuthology.orchestra.run.smithi171.stdout:Get:16 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua5.1 amd64 5.1.5-8.1build4 [94.6 kB] 2024-03-28T14:04:11.343 INFO:teuthology.orchestra.run.smithi171.stdout:Get:17 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-any all 27ubuntu1 [5,034 B] 2024-03-28T14:04:11.344 INFO:teuthology.orchestra.run.smithi171.stdout:Get:18 http://archive.ubuntu.com/ubuntu jammy/main amd64 zip amd64 3.0-12build2 [176 kB] 2024-03-28T14:04:11.370 INFO:teuthology.orchestra.run.smithi171.stdout:Get:19 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 unzip amd64 6.0-26ubuntu3.2 [175 kB] 2024-03-28T14:04:11.373 INFO:teuthology.orchestra.run.smithi171.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy/universe amd64 luarocks all 3.8.0+dfsg1-1 [140 kB] 2024-03-28T14:04:11.375 INFO:teuthology.orchestra.run.smithi171.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy/main amd64 liboath0 amd64 2.6.7-3build1 [41.1 kB] 2024-03-28T14:04:11.376 INFO:teuthology.orchestra.run.smithi171.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.functools all 3.4.0-2 [9,030 B] 2024-03-28T14:04:11.376 INFO:teuthology.orchestra.run.smithi171.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-cheroot all 8.5.2+ds1-1ubuntu3.1 [71.1 kB] 2024-03-28T14:04:11.378 INFO:teuthology.orchestra.run.smithi171.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.classes all 3.2.1-3 [6,452 B] 2024-03-28T14:04:11.378 INFO:teuthology.orchestra.run.smithi171.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.text all 3.6.0-2 [8,716 B] 2024-03-28T14:04:11.402 INFO:teuthology.orchestra.run.smithi171.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.collections all 3.4.0-2 [11.4 kB] 2024-03-28T14:04:11.403 INFO:teuthology.orchestra.run.smithi171.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempora all 4.1.2-1 [14.8 kB] 2024-03-28T14:04:11.403 INFO:teuthology.orchestra.run.smithi171.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-portend all 3.0.0-1 [7,240 B] 2024-03-28T14:04:11.412 INFO:teuthology.orchestra.run.smithi171.stdout:Get:29 https://4.chacra.ceph.com/r/ceph/wip-batrick-testing-20240327.230800/e0e48ba6ee8c8b4c8f38d3b879d6eafbbb1efc15/ubuntu/jammy/flavors/default jammy/main amd64 librbd1 amd64 19.0.0-2499-ge0e48ba6-1jammy [3,226 kB] 2024-03-28T14:04:11.439 INFO:teuthology.orchestra.run.smithi171.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-zc.lockfile all 2.0-1 [8,980 B] 2024-03-28T14:04:11.440 INFO:teuthology.orchestra.run.smithi171.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cherrypy3 all 18.6.1-4 [208 kB] 2024-03-28T14:04:11.445 INFO:teuthology.orchestra.run.smithi171.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-natsort all 8.0.2-1 [35.3 kB] 2024-03-28T14:04:11.446 INFO:teuthology.orchestra.run.smithi171.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-logutils all 0.3.3-8 [17.6 kB] 2024-03-28T14:04:11.447 INFO:teuthology.orchestra.run.smithi171.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-markupsafe amd64 2.0.1-2build1 [12.7 kB] 2024-03-28T14:04:11.448 INFO:teuthology.orchestra.run.smithi171.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-mako all 1.1.3+ds1-2ubuntu0.1 [60.5 kB] 2024-03-28T14:04:11.449 INFO:teuthology.orchestra.run.smithi171.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplegeneric all 0.8.1-3 [11.3 kB] 2024-03-28T14:04:11.472 INFO:teuthology.orchestra.run.smithi104.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblttng-ust1 amd64 2.13.1-1ubuntu1 [190 kB] 2024-03-28T14:04:11.473 INFO:teuthology.orchestra.run.smithi171.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-singledispatch all 3.4.0.3-3 [7,320 B] 2024-03-28T14:04:11.474 INFO:teuthology.orchestra.run.smithi171.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webob all 1:1.8.6-1.1 [85.2 kB] 2024-03-28T14:04:11.476 INFO:teuthology.orchestra.run.smithi171.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-waitress all 1.4.4-1.1ubuntu1 [47.3 kB] 2024-03-28T14:04:11.516 INFO:teuthology.orchestra.run.smithi171.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempita all 0.5.2-6ubuntu1 [15.1 kB] 2024-03-28T14:04:11.517 INFO:teuthology.orchestra.run.smithi171.stdout:Get:41 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-paste all 3.5.0+dfsg1-1 [456 kB] 2024-03-28T14:04:11.527 INFO:teuthology.orchestra.run.smithi171.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-pastedeploy-tpl all 2.1.1-1 [4,892 B] 2024-03-28T14:04:11.528 INFO:teuthology.orchestra.run.smithi171.stdout:Get:43 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastedeploy all 2.1.1-1 [26.6 kB] 2024-03-28T14:04:11.529 INFO:teuthology.orchestra.run.smithi171.stdout:Get:44 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webtest all 2.0.35-1 [28.5 kB] 2024-03-28T14:04:11.530 INFO:teuthology.orchestra.run.smithi171.stdout:Get:45 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pecan all 1.3.3-4ubuntu2 [87.3 kB] 2024-03-28T14:04:11.532 INFO:teuthology.orchestra.run.smithi171.stdout:Get:46 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-certifi all 2020.6.20-1 [150 kB] 2024-03-28T14:04:11.540 INFO:teuthology.orchestra.run.smithi171.stdout:Get:47 https://4.chacra.ceph.com/r/ceph/wip-batrick-testing-20240327.230800/e0e48ba6ee8c8b4c8f38d3b879d6eafbbb1efc15/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs2 amd64 19.0.0-2499-ge0e48ba6-1jammy [712 kB] 2024-03-28T14:04:11.563 INFO:teuthology.orchestra.run.smithi171.stdout:Get:48 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-idna all 3.3-1 [49.3 kB] 2024-03-28T14:04:11.564 INFO:teuthology.orchestra.run.smithi171.stdout:Get:49 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-urllib3 all 1.26.5-1~exp1ubuntu0.1 [98.2 kB] 2024-03-28T14:04:11.567 INFO:teuthology.orchestra.run.smithi171.stdout:Get:50 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-requests all 2.25.1+dfsg-2ubuntu0.1 [48.8 kB] 2024-03-28T14:04:11.567 INFO:teuthology.orchestra.run.smithi104.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libdouble-conversion3 amd64 3.1.7-4 [39.0 kB] 2024-03-28T14:04:11.574 INFO:teuthology.orchestra.run.smithi171.stdout:Get:51 https://4.chacra.ceph.com/r/ceph/wip-batrick-testing-20240327.230800/e0e48ba6ee8c8b4c8f38d3b879d6eafbbb1efc15/ubuntu/jammy/flavors/default jammy/main amd64 python3-rados amd64 19.0.0-2499-ge0e48ba6-1jammy [356 kB] 2024-03-28T14:04:11.581 INFO:teuthology.orchestra.run.smithi104.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libpcre2-16-0 amd64 10.39-3ubuntu0.1 [203 kB] 2024-03-28T14:04:11.589 INFO:teuthology.orchestra.run.smithi171.stdout:Get:52 https://4.chacra.ceph.com/r/ceph/wip-batrick-testing-20240327.230800/e0e48ba6ee8c8b4c8f38d3b879d6eafbbb1efc15/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-argparse all 19.0.0-2499-ge0e48ba6-1jammy [32.8 kB] 2024-03-28T14:04:11.590 INFO:teuthology.orchestra.run.smithi171.stdout:Get:53 https://4.chacra.ceph.com/r/ceph/wip-batrick-testing-20240327.230800/e0e48ba6ee8c8b4c8f38d3b879d6eafbbb1efc15/ubuntu/jammy/flavors/default jammy/main amd64 python3-cephfs amd64 19.0.0-2499-ge0e48ba6-1jammy [180 kB] 2024-03-28T14:04:11.597 INFO:teuthology.orchestra.run.smithi171.stdout:Get:54 https://4.chacra.ceph.com/r/ceph/wip-batrick-testing-20240327.230800/e0e48ba6ee8c8b4c8f38d3b879d6eafbbb1efc15/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-common all 19.0.0-2499-ge0e48ba6-1jammy [66.6 kB] 2024-03-28T14:04:11.600 INFO:teuthology.orchestra.run.smithi171.stdout:Get:55 https://4.chacra.ceph.com/r/ceph/wip-batrick-testing-20240327.230800/e0e48ba6ee8c8b4c8f38d3b879d6eafbbb1efc15/ubuntu/jammy/flavors/default jammy/main amd64 python3-rbd amd64 19.0.0-2499-ge0e48ba6-1jammy [330 kB] 2024-03-28T14:04:11.603 INFO:teuthology.orchestra.run.smithi171.stdout:Get:56 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-werkzeug all 2.0.2+dfsg1-1ubuntu0.22.04.1 [181 kB] 2024-03-28T14:04:11.608 INFO:teuthology.orchestra.run.smithi171.stdout:Get:57 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-bcrypt amd64 3.2.0-1build1 [32.7 kB] 2024-03-28T14:04:11.608 INFO:teuthology.orchestra.run.smithi171.stdout:Get:58 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-openssl all 21.0.0-1 [45.2 kB] 2024-03-28T14:04:11.610 INFO:teuthology.orchestra.run.smithi171.stdout:Get:59 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-asyncssh all 2.5.0-1 [185 kB] 2024-03-28T14:04:11.615 INFO:teuthology.orchestra.run.smithi171.stdout:Get:60 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-babel-localedata all 2.8.0+dfsg.1-7 [4,982 kB] 2024-03-28T14:04:11.616 INFO:teuthology.orchestra.run.smithi171.stdout:Get:61 https://4.chacra.ceph.com/r/ceph/wip-batrick-testing-20240327.230800/e0e48ba6ee8c8b4c8f38d3b879d6eafbbb1efc15/ubuntu/jammy/flavors/default jammy/main amd64 librgw2 amd64 19.0.0-2499-ge0e48ba6-1jammy [6,660 kB] 2024-03-28T14:04:11.633 INFO:teuthology.orchestra.run.smithi104.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5core5a amd64 5.15.3+dfsg-2ubuntu0.2 [2,006 kB] 2024-03-28T14:04:11.763 INFO:teuthology.orchestra.run.smithi171.stdout:Get:62 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-babel all 2.8.0+dfsg.1-7 [85.1 kB] 2024-03-28T14:04:11.764 INFO:teuthology.orchestra.run.smithi171.stdout:Get:63 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-jinja2 all 3.0.3-1ubuntu0.1 [108 kB] 2024-03-28T14:04:11.766 INFO:teuthology.orchestra.run.smithi171.stdout:Get:64 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-repoze.lru all 0.7-2 [12.1 kB] 2024-03-28T14:04:11.767 INFO:teuthology.orchestra.run.smithi171.stdout:Get:65 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-routes all 2.5.1-1ubuntu1 [89.0 kB] 2024-03-28T14:04:11.768 INFO:teuthology.orchestra.run.smithi171.stdout:Get:66 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn-lib amd64 0.23.2-5ubuntu6 [2,058 kB] 2024-03-28T14:04:11.793 INFO:teuthology.orchestra.run.smithi171.stdout:Get:67 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-joblib all 0.17.0-4ubuntu1 [204 kB] 2024-03-28T14:04:11.795 INFO:teuthology.orchestra.run.smithi171.stdout:Get:68 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-threadpoolctl all 3.1.0-1 [21.3 kB] 2024-03-28T14:04:11.795 INFO:teuthology.orchestra.run.smithi171.stdout:Get:69 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn all 0.23.2-5ubuntu6 [1,829 kB] 2024-03-28T14:04:11.808 INFO:teuthology.orchestra.run.smithi104.stdout:Get:9 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5dbus5 amd64 5.15.3+dfsg-2ubuntu0.2 [222 kB] 2024-03-28T14:04:11.816 INFO:teuthology.orchestra.run.smithi104.stdout:Get:10 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5network5 amd64 5.15.3+dfsg-2ubuntu0.2 [731 kB] 2024-03-28T14:04:11.833 INFO:teuthology.orchestra.run.smithi171.stdout:Get:70 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cachetools all 5.0.0-1 [9,722 B] 2024-03-28T14:04:11.834 INFO:teuthology.orchestra.run.smithi171.stdout:Get:71 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1 all 0.4.8-1 [50.9 kB] 2024-03-28T14:04:11.835 INFO:teuthology.orchestra.run.smithi171.stdout:Get:72 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1-modules all 0.2.1-1 [33.0 kB] 2024-03-28T14:04:11.836 INFO:teuthology.orchestra.run.smithi171.stdout:Get:73 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-rsa all 4.8-1 [28.4 kB] 2024-03-28T14:04:11.837 INFO:teuthology.orchestra.run.smithi171.stdout:Get:74 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-google-auth all 1.5.1-3 [35.7 kB] 2024-03-28T14:04:11.838 INFO:teuthology.orchestra.run.smithi171.stdout:Get:75 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-requests-oauthlib all 1.3.0+ds-0.1 [18.7 kB] 2024-03-28T14:04:11.838 INFO:teuthology.orchestra.run.smithi171.stdout:Get:76 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-websocket all 1.2.3-1 [34.7 kB] 2024-03-28T14:04:11.840 INFO:teuthology.orchestra.run.smithi171.stdout:Get:77 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-kubernetes all 12.0.1-1ubuntu1 [353 kB] 2024-03-28T14:04:11.844 INFO:teuthology.orchestra.run.smithi104.stdout:Get:11 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libthrift-0.16.0 amd64 0.16.0-2 [267 kB] 2024-03-28T14:04:11.845 INFO:teuthology.orchestra.run.smithi171.stdout:Get:78 http://archive.ubuntu.com/ubuntu jammy/main amd64 libonig5 amd64 6.9.7.1-2build1 [172 kB] 2024-03-28T14:04:11.854 INFO:teuthology.orchestra.run.smithi104.stdout:Get:12 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-prettytable all 2.5.0-2 [31.3 kB] 2024-03-28T14:04:11.858 INFO:teuthology.orchestra.run.smithi171.stdout:Get:79 http://archive.ubuntu.com/ubuntu jammy/main amd64 libjq1 amd64 1.6-2.1ubuntu3 [133 kB] 2024-03-28T14:04:11.869 INFO:teuthology.orchestra.run.smithi171.stdout:Get:80 http://archive.ubuntu.com/ubuntu jammy/main amd64 jq amd64 1.6-2.1ubuntu3 [52.5 kB] 2024-03-28T14:04:11.870 INFO:teuthology.orchestra.run.smithi171.stdout:Get:81 http://archive.ubuntu.com/ubuntu jammy/main amd64 socat amd64 1.7.4.1-3ubuntu4 [349 kB] 2024-03-28T14:04:11.877 INFO:teuthology.orchestra.run.smithi171.stdout:Get:82 http://archive.ubuntu.com/ubuntu jammy/universe amd64 xmlstarlet amd64 1.6.1-2.1 [265 kB] 2024-03-28T14:04:11.883 INFO:teuthology.orchestra.run.smithi171.stdout:Get:83 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-colorama all 0.4.4-1 [24.5 kB] 2024-03-28T14:04:11.883 INFO:teuthology.orchestra.run.smithi171.stdout:Get:84 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-socket amd64 3.0~rc1+git+ac3201d-6 [78.9 kB] 2024-03-28T14:04:11.885 INFO:teuthology.orchestra.run.smithi171.stdout:Get:85 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-sec amd64 1.0.2-1 [37.6 kB] 2024-03-28T14:04:11.886 INFO:teuthology.orchestra.run.smithi171.stdout:Get:86 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 nvme-cli amd64 1.16-3ubuntu0.1 [474 kB] 2024-03-28T14:04:11.886 INFO:teuthology.orchestra.run.smithi171.stdout:Get:87 https://4.chacra.ceph.com/r/ceph/wip-batrick-testing-20240327.230800/e0e48ba6ee8c8b4c8f38d3b879d6eafbbb1efc15/ubuntu/jammy/flavors/default jammy/main amd64 python3-rgw amd64 19.0.0-2499-ge0e48ba6-1jammy [112 kB] 2024-03-28T14:04:11.890 INFO:teuthology.orchestra.run.smithi171.stdout:Get:88 https://4.chacra.ceph.com/r/ceph/wip-batrick-testing-20240327.230800/e0e48ba6ee8c8b4c8f38d3b879d6eafbbb1efc15/ubuntu/jammy/flavors/default jammy/main amd64 libradosstriper1 amd64 19.0.0-2499-ge0e48ba6-1jammy [466 kB] 2024-03-28T14:04:11.896 INFO:teuthology.orchestra.run.smithi171.stdout:Get:89 http://archive.ubuntu.com/ubuntu jammy/main amd64 pkg-config amd64 0.29.2-1ubuntu3 [48.2 kB] 2024-03-28T14:04:11.897 INFO:teuthology.orchestra.run.smithi171.stdout:Get:90 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python-asyncssh-doc all 2.5.0-1 [318 kB] 2024-03-28T14:04:11.908 INFO:teuthology.orchestra.run.smithi171.stdout:Get:91 https://4.chacra.ceph.com/r/ceph/wip-batrick-testing-20240327.230800/e0e48ba6ee8c8b4c8f38d3b879d6eafbbb1efc15/ubuntu/jammy/flavors/default jammy/main amd64 ceph-common amd64 19.0.0-2499-ge0e48ba6-1jammy [25.3 MB] 2024-03-28T14:04:11.909 INFO:teuthology.orchestra.run.smithi171.stdout:Get:92 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-iniconfig all 1.1.1-2 [6,024 B] 2024-03-28T14:04:11.910 INFO:teuthology.orchestra.run.smithi171.stdout:Get:93 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastescript all 2.0.2-4 [54.6 kB] 2024-03-28T14:04:11.911 INFO:teuthology.orchestra.run.smithi171.stdout:Get:94 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pluggy all 0.13.0-7.1 [19.0 kB] 2024-03-28T14:04:11.931 INFO:teuthology.orchestra.run.smithi171.stdout:Get:95 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-psutil amd64 5.9.0-1build1 [158 kB] 2024-03-28T14:04:11.932 INFO:teuthology.orchestra.run.smithi171.stdout:Get:96 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-py all 1.10.0-1 [71.9 kB] 2024-03-28T14:04:11.935 INFO:teuthology.orchestra.run.smithi171.stdout:Get:97 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pygments all 2.11.2+dfsg-2 [750 kB] 2024-03-28T14:04:11.950 INFO:teuthology.orchestra.run.smithi171.stdout:Get:98 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyinotify all 0.9.6-1.3 [24.8 kB] 2024-03-28T14:04:11.950 INFO:teuthology.orchestra.run.smithi171.stdout:Get:99 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-toml all 0.10.2-1 [16.5 kB] 2024-03-28T14:04:11.951 INFO:teuthology.orchestra.run.smithi171.stdout:Get:100 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pytest all 6.2.5-1ubuntu2 [214 kB] 2024-03-28T14:04:11.954 INFO:teuthology.orchestra.run.smithi171.stdout:Get:101 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplejson amd64 3.17.6-1build1 [54.7 kB] 2024-03-28T14:04:11.957 INFO:teuthology.orchestra.run.smithi171.stdout:Get:102 http://archive.ubuntu.com/ubuntu jammy/universe amd64 qttranslations5-l10n all 5.15.3-1 [1,983 kB] 2024-03-28T14:04:12.022 INFO:teuthology.orchestra.run.smithi104.stdout:Get:13 http://archive.ubuntu.com/ubuntu jammy/universe amd64 librdkafka1 amd64 1.8.0-1build1 [633 kB] 2024-03-28T14:04:12.162 INFO:teuthology.orchestra.run.smithi104.stdout:Get:14 https://4.chacra.ceph.com/r/ceph/wip-batrick-testing-20240327.230800/e0e48ba6ee8c8b4c8f38d3b879d6eafbbb1efc15/ubuntu/jammy/flavors/default jammy/main amd64 librbd1 amd64 19.0.0-2499-ge0e48ba6-1jammy [3,226 kB] 2024-03-28T14:04:12.586 INFO:teuthology.orchestra.run.smithi104.stdout:Get:15 http://archive.ubuntu.com/ubuntu jammy/main amd64 libreadline-dev amd64 8.1.2-1 [166 kB] 2024-03-28T14:04:12.601 INFO:teuthology.orchestra.run.smithi104.stdout:Get:16 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblua5.3-dev amd64 5.3.6-1build1 [167 kB] 2024-03-28T14:04:12.617 INFO:teuthology.orchestra.run.smithi104.stdout:Get:17 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua5.1 amd64 5.1.5-8.1build4 [94.6 kB] 2024-03-28T14:04:12.625 INFO:teuthology.orchestra.run.smithi104.stdout:Get:18 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-any all 27ubuntu1 [5,034 B] 2024-03-28T14:04:12.626 INFO:teuthology.orchestra.run.smithi104.stdout:Get:19 http://archive.ubuntu.com/ubuntu jammy/main amd64 zip amd64 3.0-12build2 [176 kB] 2024-03-28T14:04:12.642 INFO:teuthology.orchestra.run.smithi104.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 unzip amd64 6.0-26ubuntu3.2 [175 kB] 2024-03-28T14:04:12.680 INFO:teuthology.orchestra.run.smithi104.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy/universe amd64 luarocks all 3.8.0+dfsg1-1 [140 kB] 2024-03-28T14:04:12.693 INFO:teuthology.orchestra.run.smithi104.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy/main amd64 liboath0 amd64 2.6.7-3build1 [41.1 kB] 2024-03-28T14:04:12.697 INFO:teuthology.orchestra.run.smithi104.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.functools all 3.4.0-2 [9,030 B] 2024-03-28T14:04:12.698 INFO:teuthology.orchestra.run.smithi104.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-cheroot all 8.5.2+ds1-1ubuntu3.1 [71.1 kB] 2024-03-28T14:04:12.825 INFO:teuthology.orchestra.run.smithi104.stdout:Get:25 https://4.chacra.ceph.com/r/ceph/wip-batrick-testing-20240327.230800/e0e48ba6ee8c8b4c8f38d3b879d6eafbbb1efc15/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs2 amd64 19.0.0-2499-ge0e48ba6-1jammy [712 kB] 2024-03-28T14:04:12.868 INFO:teuthology.orchestra.run.smithi104.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.classes all 3.2.1-3 [6,452 B] 2024-03-28T14:04:12.951 INFO:teuthology.orchestra.run.smithi104.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.text all 3.6.0-2 [8,716 B] 2024-03-28T14:04:12.974 INFO:teuthology.orchestra.run.smithi104.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.collections all 3.4.0-2 [11.4 kB] 2024-03-28T14:04:12.982 INFO:teuthology.orchestra.run.smithi104.stdout:Get:29 https://4.chacra.ceph.com/r/ceph/wip-batrick-testing-20240327.230800/e0e48ba6ee8c8b4c8f38d3b879d6eafbbb1efc15/ubuntu/jammy/flavors/default jammy/main amd64 python3-rados amd64 19.0.0-2499-ge0e48ba6-1jammy [356 kB] 2024-03-28T14:04:13.032 INFO:teuthology.orchestra.run.smithi104.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempora all 4.1.2-1 [14.8 kB] 2024-03-28T14:04:13.074 INFO:teuthology.orchestra.run.smithi104.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-portend all 3.0.0-1 [7,240 B] 2024-03-28T14:04:13.077 INFO:teuthology.orchestra.run.smithi104.stdout:Get:32 https://4.chacra.ceph.com/r/ceph/wip-batrick-testing-20240327.230800/e0e48ba6ee8c8b4c8f38d3b879d6eafbbb1efc15/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-argparse all 19.0.0-2499-ge0e48ba6-1jammy [32.8 kB] 2024-03-28T14:04:13.080 INFO:teuthology.orchestra.run.smithi171.stdout:Get:103 https://4.chacra.ceph.com/r/ceph/wip-batrick-testing-20240327.230800/e0e48ba6ee8c8b4c8f38d3b879d6eafbbb1efc15/ubuntu/jammy/flavors/default jammy/main amd64 ceph-base amd64 19.0.0-2499-ge0e48ba6-1jammy [5,086 kB] 2024-03-28T14:04:13.080 INFO:teuthology.orchestra.run.smithi104.stdout:Get:33 https://4.chacra.ceph.com/r/ceph/wip-batrick-testing-20240327.230800/e0e48ba6ee8c8b4c8f38d3b879d6eafbbb1efc15/ubuntu/jammy/flavors/default jammy/main amd64 python3-cephfs amd64 19.0.0-2499-ge0e48ba6-1jammy [180 kB] 2024-03-28T14:04:13.098 INFO:teuthology.orchestra.run.smithi104.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-zc.lockfile all 2.0-1 [8,980 B] 2024-03-28T14:04:13.125 INFO:teuthology.orchestra.run.smithi104.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cherrypy3 all 18.6.1-4 [208 kB] 2024-03-28T14:04:13.125 INFO:teuthology.orchestra.run.smithi104.stdout:Get:36 https://4.chacra.ceph.com/r/ceph/wip-batrick-testing-20240327.230800/e0e48ba6ee8c8b4c8f38d3b879d6eafbbb1efc15/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-common all 19.0.0-2499-ge0e48ba6-1jammy [66.6 kB] 2024-03-28T14:04:13.127 INFO:teuthology.orchestra.run.smithi104.stdout:Get:37 https://4.chacra.ceph.com/r/ceph/wip-batrick-testing-20240327.230800/e0e48ba6ee8c8b4c8f38d3b879d6eafbbb1efc15/ubuntu/jammy/flavors/default jammy/main amd64 python3-rbd amd64 19.0.0-2499-ge0e48ba6-1jammy [330 kB] 2024-03-28T14:04:13.201 INFO:teuthology.orchestra.run.smithi171.stdout:Get:104 https://4.chacra.ceph.com/r/ceph/wip-batrick-testing-20240327.230800/e0e48ba6ee8c8b4c8f38d3b879d6eafbbb1efc15/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-modules-core all 19.0.0-2499-ge0e48ba6-1jammy [244 kB] 2024-03-28T14:04:13.210 INFO:teuthology.orchestra.run.smithi171.stdout:Get:105 https://4.chacra.ceph.com/r/ceph/wip-batrick-testing-20240327.230800/e0e48ba6ee8c8b4c8f38d3b879d6eafbbb1efc15/ubuntu/jammy/flavors/default jammy/main amd64 libsqlite3-mod-ceph amd64 19.0.0-2499-ge0e48ba6-1jammy [124 kB] 2024-03-28T14:04:13.211 INFO:teuthology.orchestra.run.smithi104.stdout:Get:38 https://4.chacra.ceph.com/r/ceph/wip-batrick-testing-20240327.230800/e0e48ba6ee8c8b4c8f38d3b879d6eafbbb1efc15/ubuntu/jammy/flavors/default jammy/main amd64 librgw2 amd64 19.0.0-2499-ge0e48ba6-1jammy [6,660 kB] 2024-03-28T14:04:13.214 INFO:teuthology.orchestra.run.smithi171.stdout:Get:106 https://4.chacra.ceph.com/r/ceph/wip-batrick-testing-20240327.230800/e0e48ba6ee8c8b4c8f38d3b879d6eafbbb1efc15/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr amd64 19.0.0-2499-ge0e48ba6-1jammy [1,526 kB] 2024-03-28T14:04:13.281 INFO:teuthology.orchestra.run.smithi171.stdout:Get:107 https://4.chacra.ceph.com/r/ceph/wip-batrick-testing-20240327.230800/e0e48ba6ee8c8b4c8f38d3b879d6eafbbb1efc15/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mon amd64 19.0.0-2499-ge0e48ba6-1jammy [6,259 kB] 2024-03-28T14:04:13.309 INFO:teuthology.orchestra.run.smithi104.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-natsort all 8.0.2-1 [35.3 kB] 2024-03-28T14:04:13.325 INFO:teuthology.orchestra.run.smithi104.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-logutils all 0.3.3-8 [17.6 kB] 2024-03-28T14:04:13.333 INFO:teuthology.orchestra.run.smithi104.stdout:Get:41 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-markupsafe amd64 2.0.1-2build1 [12.7 kB] 2024-03-28T14:04:13.338 INFO:teuthology.orchestra.run.smithi104.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-mako all 1.1.3+ds1-2ubuntu0.1 [60.5 kB] 2024-03-28T14:04:13.522 INFO:teuthology.orchestra.run.smithi104.stdout:Get:43 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplegeneric all 0.8.1-3 [11.3 kB] 2024-03-28T14:04:13.556 INFO:teuthology.orchestra.run.smithi171.stdout:Get:108 https://4.chacra.ceph.com/r/ceph/wip-batrick-testing-20240327.230800/e0e48ba6ee8c8b4c8f38d3b879d6eafbbb1efc15/ubuntu/jammy/flavors/default jammy/main amd64 ceph-osd amd64 19.0.0-2499-ge0e48ba6-1jammy [22.8 MB] 2024-03-28T14:04:13.605 INFO:teuthology.orchestra.run.smithi104.stdout:Get:44 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-singledispatch all 3.4.0.3-3 [7,320 B] 2024-03-28T14:04:13.620 INFO:teuthology.orchestra.run.smithi104.stdout:Get:45 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webob all 1:1.8.6-1.1 [85.2 kB] 2024-03-28T14:04:13.803 INFO:teuthology.orchestra.run.smithi104.stdout:Get:46 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-waitress all 1.4.4-1.1ubuntu1 [47.3 kB] 2024-03-28T14:04:13.854 INFO:teuthology.orchestra.run.smithi104.stdout:Get:47 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempita all 0.5.2-6ubuntu1 [15.1 kB] 2024-03-28T14:04:13.868 INFO:teuthology.orchestra.run.smithi104.stdout:Get:48 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-paste all 3.5.0+dfsg1-1 [456 kB] 2024-03-28T14:04:14.026 INFO:teuthology.orchestra.run.smithi104.stdout:Get:49 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-pastedeploy-tpl all 2.1.1-1 [4,892 B] 2024-03-28T14:04:14.026 INFO:teuthology.orchestra.run.smithi104.stdout:Get:50 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastedeploy all 2.1.1-1 [26.6 kB] 2024-03-28T14:04:14.032 INFO:teuthology.orchestra.run.smithi104.stdout:Get:51 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webtest all 2.0.35-1 [28.5 kB] 2024-03-28T14:04:14.036 INFO:teuthology.orchestra.run.smithi104.stdout:Get:52 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pecan all 1.3.3-4ubuntu2 [87.3 kB] 2024-03-28T14:04:14.050 INFO:teuthology.orchestra.run.smithi104.stdout:Get:53 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-certifi all 2020.6.20-1 [150 kB] 2024-03-28T14:04:14.235 INFO:teuthology.orchestra.run.smithi104.stdout:Get:54 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-idna all 3.3-1 [49.3 kB] 2024-03-28T14:04:14.489 INFO:teuthology.orchestra.run.smithi104.stdout:Get:55 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-urllib3 all 1.26.5-1~exp1ubuntu0.1 [98.2 kB] 2024-03-28T14:04:14.591 INFO:teuthology.orchestra.run.smithi171.stdout:Get:109 https://4.chacra.ceph.com/r/ceph/wip-batrick-testing-20240327.230800/e0e48ba6ee8c8b4c8f38d3b879d6eafbbb1efc15/ubuntu/jammy/flavors/default jammy/main amd64 ceph amd64 19.0.0-2499-ge0e48ba6-1jammy [14.0 kB] 2024-03-28T14:04:14.591 INFO:teuthology.orchestra.run.smithi171.stdout:Get:110 https://4.chacra.ceph.com/r/ceph/wip-batrick-testing-20240327.230800/e0e48ba6ee8c8b4c8f38d3b879d6eafbbb1efc15/ubuntu/jammy/flavors/default jammy/main amd64 ceph-fuse amd64 19.0.0-2499-ge0e48ba6-1jammy [870 kB] 2024-03-28T14:04:14.602 INFO:teuthology.orchestra.run.smithi104.stdout:Get:56 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-requests all 2.25.1+dfsg-2ubuntu0.1 [48.8 kB] 2024-03-28T14:04:14.605 INFO:teuthology.orchestra.run.smithi104.stdout:Get:57 https://4.chacra.ceph.com/r/ceph/wip-batrick-testing-20240327.230800/e0e48ba6ee8c8b4c8f38d3b879d6eafbbb1efc15/ubuntu/jammy/flavors/default jammy/main amd64 python3-rgw amd64 19.0.0-2499-ge0e48ba6-1jammy [112 kB] 2024-03-28T14:04:14.608 INFO:teuthology.orchestra.run.smithi171.stdout:Get:111 https://4.chacra.ceph.com/r/ceph/wip-batrick-testing-20240327.230800/e0e48ba6ee8c8b4c8f38d3b879d6eafbbb1efc15/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mds amd64 19.0.0-2499-ge0e48ba6-1jammy [2,434 kB] 2024-03-28T14:04:14.641 INFO:teuthology.orchestra.run.smithi104.stdout:Get:58 https://4.chacra.ceph.com/r/ceph/wip-batrick-testing-20240327.230800/e0e48ba6ee8c8b4c8f38d3b879d6eafbbb1efc15/ubuntu/jammy/flavors/default jammy/main amd64 libradosstriper1 amd64 19.0.0-2499-ge0e48ba6-1jammy [466 kB] 2024-03-28T14:04:14.645 INFO:teuthology.orchestra.run.smithi104.stdout:Get:59 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-werkzeug all 2.0.2+dfsg1-1ubuntu0.22.04.1 [181 kB] 2024-03-28T14:04:14.688 INFO:teuthology.orchestra.run.smithi171.stdout:Get:112 https://4.chacra.ceph.com/r/ceph/wip-batrick-testing-20240327.230800/e0e48ba6ee8c8b4c8f38d3b879d6eafbbb1efc15/ubuntu/jammy/flavors/default jammy/main amd64 cephadm amd64 19.0.0-2499-ge0e48ba6-1jammy [790 kB] 2024-03-28T14:04:14.728 INFO:teuthology.orchestra.run.smithi104.stdout:Get:60 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-bcrypt amd64 3.2.0-1build1 [32.7 kB] 2024-03-28T14:04:14.740 INFO:teuthology.orchestra.run.smithi104.stdout:Get:61 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-openssl all 21.0.0-1 [45.2 kB] 2024-03-28T14:04:14.752 INFO:teuthology.orchestra.run.smithi104.stdout:Get:62 https://4.chacra.ceph.com/r/ceph/wip-batrick-testing-20240327.230800/e0e48ba6ee8c8b4c8f38d3b879d6eafbbb1efc15/ubuntu/jammy/flavors/default jammy/main amd64 ceph-common amd64 19.0.0-2499-ge0e48ba6-1jammy [25.3 MB] 2024-03-28T14:04:14.753 INFO:teuthology.orchestra.run.smithi104.stdout:Get:63 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-asyncssh all 2.5.0-1 [185 kB] 2024-03-28T14:04:14.793 INFO:teuthology.orchestra.run.smithi104.stdout:Get:64 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-babel-localedata all 2.8.0+dfsg.1-7 [4,982 kB] 2024-03-28T14:04:14.802 INFO:teuthology.orchestra.run.smithi171.stdout:Get:113 https://4.chacra.ceph.com/r/ceph/wip-batrick-testing-20240327.230800/e0e48ba6ee8c8b4c8f38d3b879d6eafbbb1efc15/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-cephadm all 19.0.0-2499-ge0e48ba6-1jammy [147 kB] 2024-03-28T14:04:14.805 INFO:teuthology.orchestra.run.smithi171.stdout:Get:114 https://4.chacra.ceph.com/r/ceph/wip-batrick-testing-20240327.230800/e0e48ba6ee8c8b4c8f38d3b879d6eafbbb1efc15/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-dashboard all 19.0.0-2499-ge0e48ba6-1jammy [2,242 kB] 2024-03-28T14:04:14.847 INFO:teuthology.orchestra.run.smithi171.stdout:Get:115 https://4.chacra.ceph.com/r/ceph/wip-batrick-testing-20240327.230800/e0e48ba6ee8c8b4c8f38d3b879d6eafbbb1efc15/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-diskprediction-local all 19.0.0-2499-ge0e48ba6-1jammy [8,625 kB] 2024-03-28T14:04:15.056 INFO:teuthology.orchestra.run.smithi104.stdout:Get:65 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-babel all 2.8.0+dfsg.1-7 [85.1 kB] 2024-03-28T14:04:15.058 INFO:teuthology.orchestra.run.smithi104.stdout:Get:66 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-jinja2 all 3.0.3-1ubuntu0.1 [108 kB] 2024-03-28T14:04:15.061 INFO:teuthology.orchestra.run.smithi104.stdout:Get:67 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-repoze.lru all 0.7-2 [12.1 kB] 2024-03-28T14:04:15.220 INFO:teuthology.orchestra.run.smithi171.stdout:Get:116 https://4.chacra.ceph.com/r/ceph/wip-batrick-testing-20240327.230800/e0e48ba6ee8c8b4c8f38d3b879d6eafbbb1efc15/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-k8sevents all 19.0.0-2499-ge0e48ba6-1jammy [14.2 kB] 2024-03-28T14:04:15.220 INFO:teuthology.orchestra.run.smithi171.stdout:Get:117 https://4.chacra.ceph.com/r/ceph/wip-batrick-testing-20240327.230800/e0e48ba6ee8c8b4c8f38d3b879d6eafbbb1efc15/ubuntu/jammy/flavors/default jammy/main amd64 ceph-test amd64 19.0.0-2499-ge0e48ba6-1jammy [50.3 MB] 2024-03-28T14:04:15.225 INFO:teuthology.orchestra.run.smithi104.stdout:Get:68 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-routes all 2.5.1-1ubuntu1 [89.0 kB] 2024-03-28T14:04:15.552 INFO:teuthology.orchestra.run.smithi104.stdout:Get:69 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn-lib amd64 0.23.2-5ubuntu6 [2,058 kB] 2024-03-28T14:04:15.959 INFO:teuthology.orchestra.run.smithi104.stdout:Get:70 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-joblib all 0.17.0-4ubuntu1 [204 kB] 2024-03-28T14:04:15.968 INFO:teuthology.orchestra.run.smithi104.stdout:Get:71 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-threadpoolctl all 3.1.0-1 [21.3 kB] 2024-03-28T14:04:15.970 INFO:teuthology.orchestra.run.smithi104.stdout:Get:72 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn all 0.23.2-5ubuntu6 [1,829 kB] 2024-03-28T14:04:16.044 INFO:teuthology.orchestra.run.smithi104.stdout:Get:73 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cachetools all 5.0.0-1 [9,722 B] 2024-03-28T14:04:16.045 INFO:teuthology.orchestra.run.smithi104.stdout:Get:74 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1 all 0.4.8-1 [50.9 kB] 2024-03-28T14:04:16.047 INFO:teuthology.orchestra.run.smithi104.stdout:Get:75 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1-modules all 0.2.1-1 [33.0 kB] 2024-03-28T14:04:16.048 INFO:teuthology.orchestra.run.smithi104.stdout:Get:76 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-rsa all 4.8-1 [28.4 kB] 2024-03-28T14:04:16.049 INFO:teuthology.orchestra.run.smithi104.stdout:Get:77 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-google-auth all 1.5.1-3 [35.7 kB] 2024-03-28T14:04:16.215 INFO:teuthology.orchestra.run.smithi104.stdout:Get:78 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-requests-oauthlib all 1.3.0+ds-0.1 [18.7 kB] 2024-03-28T14:04:16.309 INFO:teuthology.orchestra.run.smithi104.stdout:Get:79 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-websocket all 1.2.3-1 [34.7 kB] 2024-03-28T14:04:16.406 INFO:teuthology.orchestra.run.smithi104.stdout:Get:80 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-kubernetes all 12.0.1-1ubuntu1 [353 kB] 2024-03-28T14:04:16.653 INFO:teuthology.orchestra.run.smithi104.stdout:Get:81 http://archive.ubuntu.com/ubuntu jammy/main amd64 libonig5 amd64 6.9.7.1-2build1 [172 kB] 2024-03-28T14:04:16.696 INFO:teuthology.orchestra.run.smithi104.stdout:Get:82 http://archive.ubuntu.com/ubuntu jammy/main amd64 libjq1 amd64 1.6-2.1ubuntu3 [133 kB] 2024-03-28T14:04:16.720 INFO:teuthology.orchestra.run.smithi104.stdout:Get:83 http://archive.ubuntu.com/ubuntu jammy/main amd64 jq amd64 1.6-2.1ubuntu3 [52.5 kB] 2024-03-28T14:04:16.729 INFO:teuthology.orchestra.run.smithi104.stdout:Get:84 http://archive.ubuntu.com/ubuntu jammy/main amd64 socat amd64 1.7.4.1-3ubuntu4 [349 kB] 2024-03-28T14:04:16.773 INFO:teuthology.orchestra.run.smithi104.stdout:Get:85 http://archive.ubuntu.com/ubuntu jammy/universe amd64 xmlstarlet amd64 1.6.1-2.1 [265 kB] 2024-03-28T14:04:16.798 INFO:teuthology.orchestra.run.smithi104.stdout:Get:86 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-colorama all 0.4.4-1 [24.5 kB] 2024-03-28T14:04:16.800 INFO:teuthology.orchestra.run.smithi104.stdout:Get:87 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-socket amd64 3.0~rc1+git+ac3201d-6 [78.9 kB] 2024-03-28T14:04:16.807 INFO:teuthology.orchestra.run.smithi104.stdout:Get:88 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-sec amd64 1.0.2-1 [37.6 kB] 2024-03-28T14:04:16.974 INFO:teuthology.orchestra.run.smithi104.stdout:Get:89 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 nvme-cli amd64 1.16-3ubuntu0.1 [474 kB] 2024-03-28T14:04:17.505 INFO:teuthology.orchestra.run.smithi104.stdout:Get:90 http://archive.ubuntu.com/ubuntu jammy/main amd64 pkg-config amd64 0.29.2-1ubuntu3 [48.2 kB] 2024-03-28T14:04:17.510 INFO:teuthology.orchestra.run.smithi104.stdout:Get:91 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python-asyncssh-doc all 2.5.0-1 [318 kB] 2024-03-28T14:04:17.550 INFO:teuthology.orchestra.run.smithi104.stdout:Get:92 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-iniconfig all 1.1.1-2 [6,024 B] 2024-03-28T14:04:17.551 INFO:teuthology.orchestra.run.smithi104.stdout:Get:93 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastescript all 2.0.2-4 [54.6 kB] 2024-03-28T14:04:17.558 INFO:teuthology.orchestra.run.smithi104.stdout:Get:94 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pluggy all 0.13.0-7.1 [19.0 kB] 2024-03-28T14:04:17.560 INFO:teuthology.orchestra.run.smithi104.stdout:Get:95 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-psutil amd64 5.9.0-1build1 [158 kB] 2024-03-28T14:04:17.601 INFO:teuthology.orchestra.run.smithi104.stdout:Get:96 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-py all 1.10.0-1 [71.9 kB] 2024-03-28T14:04:17.609 INFO:teuthology.orchestra.run.smithi104.stdout:Get:97 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pygments all 2.11.2+dfsg-2 [750 kB] 2024-03-28T14:04:17.682 INFO:teuthology.orchestra.run.smithi104.stdout:Get:98 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyinotify all 0.9.6-1.3 [24.8 kB] 2024-03-28T14:04:17.683 INFO:teuthology.orchestra.run.smithi104.stdout:Get:99 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-toml all 0.10.2-1 [16.5 kB] 2024-03-28T14:04:17.772 INFO:teuthology.orchestra.run.smithi171.stdout:Get:118 https://4.chacra.ceph.com/r/ceph/wip-batrick-testing-20240327.230800/e0e48ba6ee8c8b4c8f38d3b879d6eafbbb1efc15/ubuntu/jammy/flavors/default jammy/main amd64 ceph-volume all 19.0.0-2499-ge0e48ba6-1jammy [121 kB] 2024-03-28T14:04:17.775 INFO:teuthology.orchestra.run.smithi171.stdout:Get:119 https://4.chacra.ceph.com/r/ceph/wip-batrick-testing-20240327.230800/e0e48ba6ee8c8b4c8f38d3b879d6eafbbb1efc15/ubuntu/jammy/flavors/default jammy/main amd64 cephfs-mirror amd64 19.0.0-2499-ge0e48ba6-1jammy [229 kB] 2024-03-28T14:04:17.780 INFO:teuthology.orchestra.run.smithi171.stdout:Get:120 https://4.chacra.ceph.com/r/ceph/wip-batrick-testing-20240327.230800/e0e48ba6ee8c8b4c8f38d3b879d6eafbbb1efc15/ubuntu/jammy/flavors/default jammy/main amd64 cephfs-shell all 19.0.0-2499-ge0e48ba6-1jammy [34.0 kB] 2024-03-28T14:04:17.781 INFO:teuthology.orchestra.run.smithi171.stdout:Get:121 https://4.chacra.ceph.com/r/ceph/wip-batrick-testing-20240327.230800/e0e48ba6ee8c8b4c8f38d3b879d6eafbbb1efc15/ubuntu/jammy/flavors/default jammy/main amd64 cephfs-top all 19.0.0-2499-ge0e48ba6-1jammy [23.4 kB] 2024-03-28T14:04:17.781 INFO:teuthology.orchestra.run.smithi171.stdout:Get:122 https://4.chacra.ceph.com/r/ceph/wip-batrick-testing-20240327.230800/e0e48ba6ee8c8b4c8f38d3b879d6eafbbb1efc15/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs-dev amd64 19.0.0-2499-ge0e48ba6-1jammy [38.9 kB] 2024-03-28T14:04:17.782 INFO:teuthology.orchestra.run.smithi171.stdout:Get:123 https://4.chacra.ceph.com/r/ceph/wip-batrick-testing-20240327.230800/e0e48ba6ee8c8b4c8f38d3b879d6eafbbb1efc15/ubuntu/jammy/flavors/default jammy/main amd64 radosgw amd64 19.0.0-2499-ge0e48ba6-1jammy [13.2 MB] 2024-03-28T14:04:17.849 INFO:teuthology.orchestra.run.smithi104.stdout:Get:100 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pytest all 6.2.5-1ubuntu2 [214 kB] 2024-03-28T14:04:18.210 INFO:teuthology.orchestra.run.smithi171.stdout:Get:124 https://4.chacra.ceph.com/r/ceph/wip-batrick-testing-20240327.230800/e0e48ba6ee8c8b4c8f38d3b879d6eafbbb1efc15/ubuntu/jammy/flavors/default jammy/main amd64 rbd-fuse amd64 19.0.0-2499-ge0e48ba6-1jammy [91.4 kB] 2024-03-28T14:04:18.282 INFO:teuthology.orchestra.run.smithi104.stdout:Get:101 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplejson amd64 3.17.6-1build1 [54.7 kB] 2024-03-28T14:04:18.297 INFO:teuthology.orchestra.run.smithi104.stdout:Get:102 http://archive.ubuntu.com/ubuntu jammy/universe amd64 qttranslations5-l10n all 5.15.3-1 [1,983 kB] 2024-03-28T14:04:18.828 INFO:teuthology.orchestra.run.smithi171.stdout:Fetched 179 MB in 7s (25.0 MB/s) 2024-03-28T14:04:18.965 INFO:teuthology.orchestra.run.smithi171.stdout:Selecting previously unselected package python3-pyperclip. 2024-03-28T14:04:19.053 INFO:teuthology.orchestra.run.smithi171.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 165997 files and directories currently installed.) 2024-03-28T14:04:19.059 INFO:teuthology.orchestra.run.smithi171.stdout:Preparing to unpack .../000-python3-pyperclip_1.8.2-2_all.deb ... 2024-03-28T14:04:19.097 INFO:teuthology.orchestra.run.smithi171.stdout:Unpacking python3-pyperclip (1.8.2-2) ... 2024-03-28T14:04:19.358 INFO:teuthology.orchestra.run.smithi171.stdout:Selecting previously unselected package python3-wcwidth. 2024-03-28T14:04:19.373 INFO:teuthology.orchestra.run.smithi171.stdout:Preparing to unpack .../001-python3-wcwidth_0.2.5+dfsg1-1_all.deb ... 2024-03-28T14:04:19.384 INFO:teuthology.orchestra.run.smithi104.stdout:Get:103 https://4.chacra.ceph.com/r/ceph/wip-batrick-testing-20240327.230800/e0e48ba6ee8c8b4c8f38d3b879d6eafbbb1efc15/ubuntu/jammy/flavors/default jammy/main amd64 ceph-base amd64 19.0.0-2499-ge0e48ba6-1jammy [5,086 kB] 2024-03-28T14:04:19.408 INFO:teuthology.orchestra.run.smithi171.stdout:Unpacking python3-wcwidth (0.2.5+dfsg1-1) ... 2024-03-28T14:04:19.703 INFO:teuthology.orchestra.run.smithi171.stdout:Selecting previously unselected package python3-cmd2. 2024-03-28T14:04:19.729 INFO:teuthology.orchestra.run.smithi171.stdout:Preparing to unpack .../002-python3-cmd2_2.3.3-1_all.deb ... 2024-03-28T14:04:19.769 INFO:teuthology.orchestra.run.smithi171.stdout:Unpacking python3-cmd2 (2.3.3-1) ... 2024-03-28T14:04:20.036 INFO:teuthology.orchestra.run.smithi104.stdout:Get:104 https://4.chacra.ceph.com/r/ceph/wip-batrick-testing-20240327.230800/e0e48ba6ee8c8b4c8f38d3b879d6eafbbb1efc15/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-modules-core all 19.0.0-2499-ge0e48ba6-1jammy [244 kB] 2024-03-28T14:04:20.063 INFO:teuthology.orchestra.run.smithi104.stdout:Get:105 https://4.chacra.ceph.com/r/ceph/wip-batrick-testing-20240327.230800/e0e48ba6ee8c8b4c8f38d3b879d6eafbbb1efc15/ubuntu/jammy/flavors/default jammy/main amd64 libsqlite3-mod-ceph amd64 19.0.0-2499-ge0e48ba6-1jammy [124 kB] 2024-03-28T14:04:20.106 INFO:teuthology.orchestra.run.smithi104.stdout:Get:106 https://4.chacra.ceph.com/r/ceph/wip-batrick-testing-20240327.230800/e0e48ba6ee8c8b4c8f38d3b879d6eafbbb1efc15/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr amd64 19.0.0-2499-ge0e48ba6-1jammy [1,526 kB] 2024-03-28T14:04:20.114 INFO:teuthology.orchestra.run.smithi171.stdout:Selecting previously unselected package liblttng-ust1:amd64. 2024-03-28T14:04:20.142 INFO:teuthology.orchestra.run.smithi171.stdout:Preparing to unpack .../003-liblttng-ust1_2.13.1-1ubuntu1_amd64.deb ... 2024-03-28T14:04:20.222 INFO:teuthology.orchestra.run.smithi171.stdout:Unpacking liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2024-03-28T14:04:20.378 INFO:teuthology.orchestra.run.smithi104.stdout:Get:107 https://4.chacra.ceph.com/r/ceph/wip-batrick-testing-20240327.230800/e0e48ba6ee8c8b4c8f38d3b879d6eafbbb1efc15/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mon amd64 19.0.0-2499-ge0e48ba6-1jammy [6,259 kB] 2024-03-28T14:04:20.568 INFO:teuthology.orchestra.run.smithi171.stdout:Selecting previously unselected package libdouble-conversion3:amd64. 2024-03-28T14:04:20.594 INFO:teuthology.orchestra.run.smithi171.stdout:Preparing to unpack .../004-libdouble-conversion3_3.1.7-4_amd64.deb ... 2024-03-28T14:04:20.634 INFO:teuthology.orchestra.run.smithi171.stdout:Unpacking libdouble-conversion3:amd64 (3.1.7-4) ... 2024-03-28T14:04:20.937 INFO:teuthology.orchestra.run.smithi171.stdout:Selecting previously unselected package libpcre2-16-0:amd64. 2024-03-28T14:04:20.963 INFO:teuthology.orchestra.run.smithi171.stdout:Preparing to unpack .../005-libpcre2-16-0_10.39-3ubuntu0.1_amd64.deb ... 2024-03-28T14:04:21.004 INFO:teuthology.orchestra.run.smithi171.stdout:Unpacking libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2024-03-28T14:04:21.457 INFO:teuthology.orchestra.run.smithi171.stdout:Selecting previously unselected package libqt5core5a:amd64. 2024-03-28T14:04:21.484 INFO:teuthology.orchestra.run.smithi171.stdout:Preparing to unpack .../006-libqt5core5a_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-03-28T14:04:21.532 INFO:teuthology.orchestra.run.smithi171.stdout:Unpacking libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-03-28T14:04:21.577 INFO:teuthology.orchestra.run.smithi104.stdout:Get:108 https://4.chacra.ceph.com/r/ceph/wip-batrick-testing-20240327.230800/e0e48ba6ee8c8b4c8f38d3b879d6eafbbb1efc15/ubuntu/jammy/flavors/default jammy/main amd64 ceph-osd amd64 19.0.0-2499-ge0e48ba6-1jammy [22.8 MB] 2024-03-28T14:04:21.952 INFO:teuthology.orchestra.run.smithi171.stdout:Selecting previously unselected package libqt5dbus5:amd64. 2024-03-28T14:04:21.979 INFO:teuthology.orchestra.run.smithi171.stdout:Preparing to unpack .../007-libqt5dbus5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-03-28T14:04:22.019 INFO:teuthology.orchestra.run.smithi171.stdout:Unpacking libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-03-28T14:04:22.339 INFO:teuthology.orchestra.run.smithi171.stdout:Selecting previously unselected package libqt5network5:amd64. 2024-03-28T14:04:22.365 INFO:teuthology.orchestra.run.smithi171.stdout:Preparing to unpack .../008-libqt5network5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-03-28T14:04:22.398 INFO:teuthology.orchestra.run.smithi171.stdout:Unpacking libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-03-28T14:04:22.735 INFO:teuthology.orchestra.run.smithi171.stdout:Selecting previously unselected package libthrift-0.16.0:amd64. 2024-03-28T14:04:22.762 INFO:teuthology.orchestra.run.smithi171.stdout:Preparing to unpack .../009-libthrift-0.16.0_0.16.0-2_amd64.deb ... 2024-03-28T14:04:22.802 INFO:teuthology.orchestra.run.smithi171.stdout:Unpacking libthrift-0.16.0:amd64 (0.16.0-2) ... 2024-03-28T14:04:23.130 INFO:teuthology.orchestra.run.smithi171.stdout:Selecting previously unselected package librados2. 2024-03-28T14:04:23.156 INFO:teuthology.orchestra.run.smithi171.stdout:Preparing to unpack .../010-librados2_19.0.0-2499-ge0e48ba6-1jammy_amd64.deb ... 2024-03-28T14:04:23.196 INFO:teuthology.orchestra.run.smithi171.stdout:Unpacking librados2 (19.0.0-2499-ge0e48ba6-1jammy) ... 2024-03-28T14:04:23.599 INFO:teuthology.orchestra.run.smithi171.stdout:Selecting previously unselected package librbd1. 2024-03-28T14:04:23.614 INFO:teuthology.orchestra.run.smithi171.stdout:Preparing to unpack .../011-librbd1_19.0.0-2499-ge0e48ba6-1jammy_amd64.deb ... 2024-03-28T14:04:23.649 INFO:teuthology.orchestra.run.smithi171.stdout:Unpacking librbd1 (19.0.0-2499-ge0e48ba6-1jammy) ... 2024-03-28T14:04:24.110 INFO:teuthology.orchestra.run.smithi171.stdout:Selecting previously unselected package libcephfs2. 2024-03-28T14:04:24.136 INFO:teuthology.orchestra.run.smithi171.stdout:Preparing to unpack .../012-libcephfs2_19.0.0-2499-ge0e48ba6-1jammy_amd64.deb ... 2024-03-28T14:04:24.177 INFO:teuthology.orchestra.run.smithi171.stdout:Unpacking libcephfs2 (19.0.0-2499-ge0e48ba6-1jammy) ... 2024-03-28T14:04:24.688 INFO:teuthology.orchestra.run.smithi171.stdout:Selecting previously unselected package python3-rados. 2024-03-28T14:04:24.703 INFO:teuthology.orchestra.run.smithi171.stdout:Preparing to unpack .../013-python3-rados_19.0.0-2499-ge0e48ba6-1jammy_amd64.deb ... 2024-03-28T14:04:24.746 INFO:teuthology.orchestra.run.smithi171.stdout:Unpacking python3-rados (19.0.0-2499-ge0e48ba6-1jammy) ... 2024-03-28T14:04:25.028 INFO:teuthology.orchestra.run.smithi104.stdout:Get:109 https://4.chacra.ceph.com/r/ceph/wip-batrick-testing-20240327.230800/e0e48ba6ee8c8b4c8f38d3b879d6eafbbb1efc15/ubuntu/jammy/flavors/default jammy/main amd64 ceph amd64 19.0.0-2499-ge0e48ba6-1jammy [14.0 kB] 2024-03-28T14:04:25.029 INFO:teuthology.orchestra.run.smithi104.stdout:Get:110 https://4.chacra.ceph.com/r/ceph/wip-batrick-testing-20240327.230800/e0e48ba6ee8c8b4c8f38d3b879d6eafbbb1efc15/ubuntu/jammy/flavors/default jammy/main amd64 ceph-fuse amd64 19.0.0-2499-ge0e48ba6-1jammy [870 kB] 2024-03-28T14:04:25.048 INFO:teuthology.orchestra.run.smithi171.stdout:Selecting previously unselected package python3-ceph-argparse. 2024-03-28T14:04:25.075 INFO:teuthology.orchestra.run.smithi171.stdout:Preparing to unpack .../014-python3-ceph-argparse_19.0.0-2499-ge0e48ba6-1jammy_all.deb ... 2024-03-28T14:04:25.101 INFO:teuthology.orchestra.run.smithi104.stdout:Get:111 https://4.chacra.ceph.com/r/ceph/wip-batrick-testing-20240327.230800/e0e48ba6ee8c8b4c8f38d3b879d6eafbbb1efc15/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mds amd64 19.0.0-2499-ge0e48ba6-1jammy [2,434 kB] 2024-03-28T14:04:25.115 INFO:teuthology.orchestra.run.smithi171.stdout:Unpacking python3-ceph-argparse (19.0.0-2499-ge0e48ba6-1jammy) ... 2024-03-28T14:04:25.326 INFO:teuthology.orchestra.run.smithi171.stdout:Selecting previously unselected package python3-cephfs. 2024-03-28T14:04:25.340 INFO:teuthology.orchestra.run.smithi171.stdout:Preparing to unpack .../015-python3-cephfs_19.0.0-2499-ge0e48ba6-1jammy_amd64.deb ... 2024-03-28T14:04:25.376 INFO:teuthology.orchestra.run.smithi171.stdout:Unpacking python3-cephfs (19.0.0-2499-ge0e48ba6-1jammy) ... 2024-03-28T14:04:25.399 INFO:teuthology.orchestra.run.smithi104.stdout:Get:112 https://4.chacra.ceph.com/r/ceph/wip-batrick-testing-20240327.230800/e0e48ba6ee8c8b4c8f38d3b879d6eafbbb1efc15/ubuntu/jammy/flavors/default jammy/main amd64 cephadm amd64 19.0.0-2499-ge0e48ba6-1jammy [790 kB] 2024-03-28T14:04:25.507 INFO:teuthology.orchestra.run.smithi104.stdout:Get:113 https://4.chacra.ceph.com/r/ceph/wip-batrick-testing-20240327.230800/e0e48ba6ee8c8b4c8f38d3b879d6eafbbb1efc15/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-cephadm all 19.0.0-2499-ge0e48ba6-1jammy [147 kB] 2024-03-28T14:04:25.514 INFO:teuthology.orchestra.run.smithi104.stdout:Get:114 https://4.chacra.ceph.com/r/ceph/wip-batrick-testing-20240327.230800/e0e48ba6ee8c8b4c8f38d3b879d6eafbbb1efc15/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-dashboard all 19.0.0-2499-ge0e48ba6-1jammy [2,242 kB] 2024-03-28T14:04:25.712 INFO:teuthology.orchestra.run.smithi171.stdout:Selecting previously unselected package python3-ceph-common. 2024-03-28T14:04:25.738 INFO:teuthology.orchestra.run.smithi171.stdout:Preparing to unpack .../016-python3-ceph-common_19.0.0-2499-ge0e48ba6-1jammy_all.deb ... 2024-03-28T14:04:25.779 INFO:teuthology.orchestra.run.smithi171.stdout:Unpacking python3-ceph-common (19.0.0-2499-ge0e48ba6-1jammy) ... 2024-03-28T14:04:25.808 INFO:teuthology.orchestra.run.smithi104.stdout:Get:115 https://4.chacra.ceph.com/r/ceph/wip-batrick-testing-20240327.230800/e0e48ba6ee8c8b4c8f38d3b879d6eafbbb1efc15/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-diskprediction-local all 19.0.0-2499-ge0e48ba6-1jammy [8,625 kB] 2024-03-28T14:04:26.115 INFO:teuthology.orchestra.run.smithi171.stdout:Selecting previously unselected package python3-prettytable. 2024-03-28T14:04:26.141 INFO:teuthology.orchestra.run.smithi171.stdout:Preparing to unpack .../017-python3-prettytable_2.5.0-2_all.deb ... 2024-03-28T14:04:26.181 INFO:teuthology.orchestra.run.smithi171.stdout:Unpacking python3-prettytable (2.5.0-2) ... 2024-03-28T14:04:26.443 INFO:teuthology.orchestra.run.smithi171.stdout:Selecting previously unselected package python3-rbd. 2024-03-28T14:04:26.458 INFO:teuthology.orchestra.run.smithi171.stdout:Preparing to unpack .../018-python3-rbd_19.0.0-2499-ge0e48ba6-1jammy_amd64.deb ... 2024-03-28T14:04:26.501 INFO:teuthology.orchestra.run.smithi171.stdout:Unpacking python3-rbd (19.0.0-2499-ge0e48ba6-1jammy) ... 2024-03-28T14:04:26.804 INFO:teuthology.orchestra.run.smithi171.stdout:Selecting previously unselected package librdkafka1:amd64. 2024-03-28T14:04:26.830 INFO:teuthology.orchestra.run.smithi171.stdout:Preparing to unpack .../019-librdkafka1_1.8.0-1build1_amd64.deb ... 2024-03-28T14:04:26.862 INFO:teuthology.orchestra.run.smithi171.stdout:Unpacking librdkafka1:amd64 (1.8.0-1build1) ... 2024-03-28T14:04:27.060 INFO:teuthology.orchestra.run.smithi104.stdout:Get:116 https://4.chacra.ceph.com/r/ceph/wip-batrick-testing-20240327.230800/e0e48ba6ee8c8b4c8f38d3b879d6eafbbb1efc15/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-k8sevents all 19.0.0-2499-ge0e48ba6-1jammy [14.2 kB] 2024-03-28T14:04:27.060 INFO:teuthology.orchestra.run.smithi104.stdout:Get:117 https://4.chacra.ceph.com/r/ceph/wip-batrick-testing-20240327.230800/e0e48ba6ee8c8b4c8f38d3b879d6eafbbb1efc15/ubuntu/jammy/flavors/default jammy/main amd64 ceph-test amd64 19.0.0-2499-ge0e48ba6-1jammy [50.3 MB] 2024-03-28T14:04:27.131 INFO:teuthology.orchestra.run.smithi171.stdout:Selecting previously unselected package libreadline-dev:amd64. 2024-03-28T14:04:27.146 INFO:teuthology.orchestra.run.smithi171.stdout:Preparing to unpack .../020-libreadline-dev_8.1.2-1_amd64.deb ... 2024-03-28T14:04:27.190 INFO:teuthology.orchestra.run.smithi171.stdout:Unpacking libreadline-dev:amd64 (8.1.2-1) ... 2024-03-28T14:04:27.501 INFO:teuthology.orchestra.run.smithi171.stdout:Selecting previously unselected package liblua5.3-dev:amd64. 2024-03-28T14:04:27.527 INFO:teuthology.orchestra.run.smithi171.stdout:Preparing to unpack .../021-liblua5.3-dev_5.3.6-1build1_amd64.deb ... 2024-03-28T14:04:27.567 INFO:teuthology.orchestra.run.smithi171.stdout:Unpacking liblua5.3-dev:amd64 (5.3.6-1build1) ... 2024-03-28T14:04:27.888 INFO:teuthology.orchestra.run.smithi171.stdout:Selecting previously unselected package lua5.1. 2024-03-28T14:04:27.914 INFO:teuthology.orchestra.run.smithi171.stdout:Preparing to unpack .../022-lua5.1_5.1.5-8.1build4_amd64.deb ... 2024-03-28T14:04:27.954 INFO:teuthology.orchestra.run.smithi171.stdout:Unpacking lua5.1 (5.1.5-8.1build4) ... 2024-03-28T14:04:28.249 INFO:teuthology.orchestra.run.smithi171.stdout:Selecting previously unselected package lua-any. 2024-03-28T14:04:28.275 INFO:teuthology.orchestra.run.smithi171.stdout:Preparing to unpack .../023-lua-any_27ubuntu1_all.deb ... 2024-03-28T14:04:28.315 INFO:teuthology.orchestra.run.smithi171.stdout:Unpacking lua-any (27ubuntu1) ... 2024-03-28T14:04:28.559 INFO:teuthology.orchestra.run.smithi171.stdout:Selecting previously unselected package zip. 2024-03-28T14:04:28.575 INFO:teuthology.orchestra.run.smithi171.stdout:Preparing to unpack .../024-zip_3.0-12build2_amd64.deb ... 2024-03-28T14:04:28.618 INFO:teuthology.orchestra.run.smithi171.stdout:Unpacking zip (3.0-12build2) ... 2024-03-28T14:04:28.938 INFO:teuthology.orchestra.run.smithi171.stdout:Selecting previously unselected package unzip. 2024-03-28T14:04:28.964 INFO:teuthology.orchestra.run.smithi171.stdout:Preparing to unpack .../025-unzip_6.0-26ubuntu3.2_amd64.deb ... 2024-03-28T14:04:29.004 INFO:teuthology.orchestra.run.smithi171.stdout:Unpacking unzip (6.0-26ubuntu3.2) ... 2024-03-28T14:04:29.358 INFO:teuthology.orchestra.run.smithi171.stdout:Selecting previously unselected package luarocks. 2024-03-28T14:04:29.373 INFO:teuthology.orchestra.run.smithi171.stdout:Preparing to unpack .../026-luarocks_3.8.0+dfsg1-1_all.deb ... 2024-03-28T14:04:29.712 INFO:teuthology.orchestra.run.smithi171.stdout:Unpacking luarocks (3.8.0+dfsg1-1) ... 2024-03-28T14:04:30.191 INFO:teuthology.orchestra.run.smithi171.stdout:Selecting previously unselected package librgw2. 2024-03-28T14:04:30.218 INFO:teuthology.orchestra.run.smithi171.stdout:Preparing to unpack .../027-librgw2_19.0.0-2499-ge0e48ba6-1jammy_amd64.deb ... 2024-03-28T14:04:30.257 INFO:teuthology.orchestra.run.smithi171.stdout:Unpacking librgw2 (19.0.0-2499-ge0e48ba6-1jammy) ... 2024-03-28T14:04:30.793 INFO:teuthology.orchestra.run.smithi171.stdout:Selecting previously unselected package python3-rgw. 2024-03-28T14:04:30.809 INFO:teuthology.orchestra.run.smithi171.stdout:Preparing to unpack .../028-python3-rgw_19.0.0-2499-ge0e48ba6-1jammy_amd64.deb ... 2024-03-28T14:04:30.851 INFO:teuthology.orchestra.run.smithi171.stdout:Unpacking python3-rgw (19.0.0-2499-ge0e48ba6-1jammy) ... 2024-03-28T14:04:31.138 INFO:teuthology.orchestra.run.smithi171.stdout:Selecting previously unselected package liboath0:amd64. 2024-03-28T14:04:31.153 INFO:teuthology.orchestra.run.smithi171.stdout:Preparing to unpack .../029-liboath0_2.6.7-3build1_amd64.deb ... 2024-03-28T14:04:31.188 INFO:teuthology.orchestra.run.smithi171.stdout:Unpacking liboath0:amd64 (2.6.7-3build1) ... 2024-03-28T14:04:31.491 INFO:teuthology.orchestra.run.smithi171.stdout:Selecting previously unselected package libradosstriper1. 2024-03-28T14:04:31.517 INFO:teuthology.orchestra.run.smithi171.stdout:Preparing to unpack .../030-libradosstriper1_19.0.0-2499-ge0e48ba6-1jammy_amd64.deb ... 2024-03-28T14:04:31.557 INFO:teuthology.orchestra.run.smithi171.stdout:Unpacking libradosstriper1 (19.0.0-2499-ge0e48ba6-1jammy) ... 2024-03-28T14:04:31.986 INFO:teuthology.orchestra.run.smithi171.stdout:Selecting previously unselected package ceph-common. 2024-03-28T14:04:32.013 INFO:teuthology.orchestra.run.smithi171.stdout:Preparing to unpack .../031-ceph-common_19.0.0-2499-ge0e48ba6-1jammy_amd64.deb ... 2024-03-28T14:04:32.052 INFO:teuthology.orchestra.run.smithi171.stdout:Unpacking ceph-common (19.0.0-2499-ge0e48ba6-1jammy) ... 2024-03-28T14:04:32.512 INFO:teuthology.orchestra.run.smithi104.stdout:Get:118 https://4.chacra.ceph.com/r/ceph/wip-batrick-testing-20240327.230800/e0e48ba6ee8c8b4c8f38d3b879d6eafbbb1efc15/ubuntu/jammy/flavors/default jammy/main amd64 ceph-volume all 19.0.0-2499-ge0e48ba6-1jammy [121 kB] 2024-03-28T14:04:32.514 INFO:teuthology.orchestra.run.smithi104.stdout:Get:119 https://4.chacra.ceph.com/r/ceph/wip-batrick-testing-20240327.230800/e0e48ba6ee8c8b4c8f38d3b879d6eafbbb1efc15/ubuntu/jammy/flavors/default jammy/main amd64 cephfs-mirror amd64 19.0.0-2499-ge0e48ba6-1jammy [229 kB] 2024-03-28T14:04:32.519 INFO:teuthology.orchestra.run.smithi104.stdout:Get:120 https://4.chacra.ceph.com/r/ceph/wip-batrick-testing-20240327.230800/e0e48ba6ee8c8b4c8f38d3b879d6eafbbb1efc15/ubuntu/jammy/flavors/default jammy/main amd64 cephfs-shell all 19.0.0-2499-ge0e48ba6-1jammy [34.0 kB] 2024-03-28T14:04:32.520 INFO:teuthology.orchestra.run.smithi104.stdout:Get:121 https://4.chacra.ceph.com/r/ceph/wip-batrick-testing-20240327.230800/e0e48ba6ee8c8b4c8f38d3b879d6eafbbb1efc15/ubuntu/jammy/flavors/default jammy/main amd64 cephfs-top all 19.0.0-2499-ge0e48ba6-1jammy [23.4 kB] 2024-03-28T14:04:32.520 INFO:teuthology.orchestra.run.smithi104.stdout:Get:122 https://4.chacra.ceph.com/r/ceph/wip-batrick-testing-20240327.230800/e0e48ba6ee8c8b4c8f38d3b879d6eafbbb1efc15/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs-dev amd64 19.0.0-2499-ge0e48ba6-1jammy [38.9 kB] 2024-03-28T14:04:32.522 INFO:teuthology.orchestra.run.smithi104.stdout:Get:123 https://4.chacra.ceph.com/r/ceph/wip-batrick-testing-20240327.230800/e0e48ba6ee8c8b4c8f38d3b879d6eafbbb1efc15/ubuntu/jammy/flavors/default jammy/main amd64 radosgw amd64 19.0.0-2499-ge0e48ba6-1jammy [13.2 MB] 2024-03-28T14:04:33.358 INFO:teuthology.orchestra.run.smithi171.stdout:Selecting previously unselected package ceph-base. 2024-03-28T14:04:33.384 INFO:teuthology.orchestra.run.smithi171.stdout:Preparing to unpack .../032-ceph-base_19.0.0-2499-ge0e48ba6-1jammy_amd64.deb ... 2024-03-28T14:04:33.433 INFO:teuthology.orchestra.run.smithi171.stdout:Unpacking ceph-base (19.0.0-2499-ge0e48ba6-1jammy) ... 2024-03-28T14:04:33.478 INFO:teuthology.orchestra.run.smithi104.stdout:Get:124 https://4.chacra.ceph.com/r/ceph/wip-batrick-testing-20240327.230800/e0e48ba6ee8c8b4c8f38d3b879d6eafbbb1efc15/ubuntu/jammy/flavors/default jammy/main amd64 rbd-fuse amd64 19.0.0-2499-ge0e48ba6-1jammy [91.4 kB] 2024-03-28T14:04:34.037 INFO:teuthology.orchestra.run.smithi171.stdout:Selecting previously unselected package python3-jaraco.functools. 2024-03-28T14:04:34.063 INFO:teuthology.orchestra.run.smithi171.stdout:Preparing to unpack .../033-python3-jaraco.functools_3.4.0-2_all.deb ... 2024-03-28T14:04:34.091 INFO:teuthology.orchestra.run.smithi104.stdout:Fetched 179 MB in 23s (7,945 kB/s) 2024-03-28T14:04:34.103 INFO:teuthology.orchestra.run.smithi171.stdout:Unpacking python3-jaraco.functools (3.4.0-2) ... 2024-03-28T14:04:34.231 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package python3-pyperclip. 2024-03-28T14:04:34.330 INFO:teuthology.orchestra.run.smithi104.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 165997 files and directories currently installed.) 2024-03-28T14:04:34.335 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../000-python3-pyperclip_1.8.2-2_all.deb ... 2024-03-28T14:04:34.379 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking python3-pyperclip (1.8.2-2) ... 2024-03-28T14:04:34.406 INFO:teuthology.orchestra.run.smithi171.stdout:Selecting previously unselected package python3-cheroot. 2024-03-28T14:04:34.433 INFO:teuthology.orchestra.run.smithi171.stdout:Preparing to unpack .../034-python3-cheroot_8.5.2+ds1-1ubuntu3.1_all.deb ... 2024-03-28T14:04:34.473 INFO:teuthology.orchestra.run.smithi171.stdout:Unpacking python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2024-03-28T14:04:34.649 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package python3-wcwidth. 2024-03-28T14:04:34.675 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../001-python3-wcwidth_0.2.5+dfsg1-1_all.deb ... 2024-03-28T14:04:34.715 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking python3-wcwidth (0.2.5+dfsg1-1) ... 2024-03-28T14:04:34.792 INFO:teuthology.orchestra.run.smithi171.stdout:Selecting previously unselected package python3-jaraco.classes. 2024-03-28T14:04:34.819 INFO:teuthology.orchestra.run.smithi171.stdout:Preparing to unpack .../035-python3-jaraco.classes_3.2.1-3_all.deb ... 2024-03-28T14:04:34.859 INFO:teuthology.orchestra.run.smithi171.stdout:Unpacking python3-jaraco.classes (3.2.1-3) ... 2024-03-28T14:04:35.010 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package python3-cmd2. 2024-03-28T14:04:35.037 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../002-python3-cmd2_2.3.3-1_all.deb ... 2024-03-28T14:04:35.076 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking python3-cmd2 (2.3.3-1) ... 2024-03-28T14:04:35.170 INFO:teuthology.orchestra.run.smithi171.stdout:Selecting previously unselected package python3-jaraco.text. 2024-03-28T14:04:35.197 INFO:teuthology.orchestra.run.smithi171.stdout:Preparing to unpack .../036-python3-jaraco.text_3.6.0-2_all.deb ... 2024-03-28T14:04:35.237 INFO:teuthology.orchestra.run.smithi171.stdout:Unpacking python3-jaraco.text (3.6.0-2) ... 2024-03-28T14:04:35.413 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package liblttng-ust1:amd64. 2024-03-28T14:04:35.439 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../003-liblttng-ust1_2.13.1-1ubuntu1_amd64.deb ... 2024-03-28T14:04:35.515 INFO:teuthology.orchestra.run.smithi171.stdout:Selecting previously unselected package python3-jaraco.collections. 2024-03-28T14:04:35.521 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2024-03-28T14:04:35.541 INFO:teuthology.orchestra.run.smithi171.stdout:Preparing to unpack .../037-python3-jaraco.collections_3.4.0-2_all.deb ... 2024-03-28T14:04:35.581 INFO:teuthology.orchestra.run.smithi171.stdout:Unpacking python3-jaraco.collections (3.4.0-2) ... 2024-03-28T14:04:35.841 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package libdouble-conversion3:amd64. 2024-03-28T14:04:35.851 INFO:teuthology.orchestra.run.smithi171.stdout:Selecting previously unselected package python3-tempora. 2024-03-28T14:04:35.868 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../004-libdouble-conversion3_3.1.7-4_amd64.deb ... 2024-03-28T14:04:35.877 INFO:teuthology.orchestra.run.smithi171.stdout:Preparing to unpack .../038-python3-tempora_4.1.2-1_all.deb ... 2024-03-28T14:04:35.907 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking libdouble-conversion3:amd64 (3.1.7-4) ... 2024-03-28T14:04:35.917 INFO:teuthology.orchestra.run.smithi171.stdout:Unpacking python3-tempora (4.1.2-1) ... 2024-03-28T14:04:36.229 INFO:teuthology.orchestra.run.smithi171.stdout:Selecting previously unselected package python3-portend. 2024-03-28T14:04:36.252 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package libpcre2-16-0:amd64. 2024-03-28T14:04:36.255 INFO:teuthology.orchestra.run.smithi171.stdout:Preparing to unpack .../039-python3-portend_3.0.0-1_all.deb ... 2024-03-28T14:04:36.279 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../005-libpcre2-16-0_10.39-3ubuntu0.1_amd64.deb ... 2024-03-28T14:04:36.295 INFO:teuthology.orchestra.run.smithi171.stdout:Unpacking python3-portend (3.0.0-1) ... 2024-03-28T14:04:36.319 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2024-03-28T14:04:36.581 INFO:teuthology.orchestra.run.smithi171.stdout:Selecting previously unselected package python3-zc.lockfile. 2024-03-28T14:04:36.608 INFO:teuthology.orchestra.run.smithi171.stdout:Preparing to unpack .../040-python3-zc.lockfile_2.0-1_all.deb ... 2024-03-28T14:04:36.648 INFO:teuthology.orchestra.run.smithi171.stdout:Unpacking python3-zc.lockfile (2.0-1) ... 2024-03-28T14:04:36.755 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package libqt5core5a:amd64. 2024-03-28T14:04:36.782 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../006-libqt5core5a_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-03-28T14:04:36.831 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-03-28T14:04:36.909 INFO:teuthology.orchestra.run.smithi171.stdout:Selecting previously unselected package python3-cherrypy3. 2024-03-28T14:04:36.935 INFO:teuthology.orchestra.run.smithi171.stdout:Preparing to unpack .../041-python3-cherrypy3_18.6.1-4_all.deb ... 2024-03-28T14:04:36.967 INFO:teuthology.orchestra.run.smithi171.stdout:Unpacking python3-cherrypy3 (18.6.1-4) ... 2024-03-28T14:04:37.225 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package libqt5dbus5:amd64. 2024-03-28T14:04:37.251 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../007-libqt5dbus5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-03-28T14:04:37.288 INFO:teuthology.orchestra.run.smithi171.stdout:Selecting previously unselected package python3-natsort. 2024-03-28T14:04:37.292 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-03-28T14:04:37.314 INFO:teuthology.orchestra.run.smithi171.stdout:Preparing to unpack .../042-python3-natsort_8.0.2-1_all.deb ... 2024-03-28T14:04:37.354 INFO:teuthology.orchestra.run.smithi171.stdout:Unpacking python3-natsort (8.0.2-1) ... 2024-03-28T14:04:37.636 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package libqt5network5:amd64. 2024-03-28T14:04:37.640 INFO:teuthology.orchestra.run.smithi171.stdout:Selecting previously unselected package python3-logutils. 2024-03-28T14:04:37.663 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../008-libqt5network5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-03-28T14:04:37.667 INFO:teuthology.orchestra.run.smithi171.stdout:Preparing to unpack .../043-python3-logutils_0.3.3-8_all.deb ... 2024-03-28T14:04:37.703 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-03-28T14:04:37.707 INFO:teuthology.orchestra.run.smithi171.stdout:Unpacking python3-logutils (0.3.3-8) ... 2024-03-28T14:04:37.935 INFO:teuthology.orchestra.run.smithi171.stdout:Selecting previously unselected package python3-markupsafe. 2024-03-28T14:04:37.961 INFO:teuthology.orchestra.run.smithi171.stdout:Preparing to unpack .../044-python3-markupsafe_2.0.1-2build1_amd64.deb ... 2024-03-28T14:04:38.001 INFO:teuthology.orchestra.run.smithi171.stdout:Unpacking python3-markupsafe (2.0.1-2build1) ... 2024-03-28T14:04:38.048 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package libthrift-0.16.0:amd64. 2024-03-28T14:04:38.074 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../009-libthrift-0.16.0_0.16.0-2_amd64.deb ... 2024-03-28T14:04:38.114 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking libthrift-0.16.0:amd64 (0.16.0-2) ... 2024-03-28T14:04:38.304 INFO:teuthology.orchestra.run.smithi171.stdout:Selecting previously unselected package python3-mako. 2024-03-28T14:04:38.331 INFO:teuthology.orchestra.run.smithi171.stdout:Preparing to unpack .../045-python3-mako_1.1.3+ds1-2ubuntu0.1_all.deb ... 2024-03-28T14:04:38.371 INFO:teuthology.orchestra.run.smithi171.stdout:Unpacking python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2024-03-28T14:04:38.425 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package librados2. 2024-03-28T14:04:38.452 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../010-librados2_19.0.0-2499-ge0e48ba6-1jammy_amd64.deb ... 2024-03-28T14:04:38.492 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking librados2 (19.0.0-2499-ge0e48ba6-1jammy) ... 2024-03-28T14:04:38.716 INFO:teuthology.orchestra.run.smithi171.stdout:Selecting previously unselected package python3-simplegeneric. 2024-03-28T14:04:38.742 INFO:teuthology.orchestra.run.smithi171.stdout:Preparing to unpack .../046-python3-simplegeneric_0.8.1-3_all.deb ... 2024-03-28T14:04:38.782 INFO:teuthology.orchestra.run.smithi171.stdout:Unpacking python3-simplegeneric (0.8.1-3) ... 2024-03-28T14:04:39.085 INFO:teuthology.orchestra.run.smithi171.stdout:Selecting previously unselected package python3-singledispatch. 2024-03-28T14:04:39.112 INFO:teuthology.orchestra.run.smithi171.stdout:Preparing to unpack .../047-python3-singledispatch_3.4.0.3-3_all.deb ... 2024-03-28T14:04:39.152 INFO:teuthology.orchestra.run.smithi171.stdout:Unpacking python3-singledispatch (3.4.0.3-3) ... 2024-03-28T14:04:39.195 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package librbd1. 2024-03-28T14:04:39.221 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../011-librbd1_19.0.0-2499-ge0e48ba6-1jammy_amd64.deb ... 2024-03-28T14:04:39.261 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking librbd1 (19.0.0-2499-ge0e48ba6-1jammy) ... 2024-03-28T14:04:39.396 INFO:teuthology.orchestra.run.smithi171.stdout:Selecting previously unselected package python3-webob. 2024-03-28T14:04:39.423 INFO:teuthology.orchestra.run.smithi171.stdout:Preparing to unpack .../048-python3-webob_1%3a1.8.6-1.1_all.deb ... 2024-03-28T14:04:39.463 INFO:teuthology.orchestra.run.smithi171.stdout:Unpacking python3-webob (1:1.8.6-1.1) ... 2024-03-28T14:04:39.714 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package libcephfs2. 2024-03-28T14:04:39.741 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../012-libcephfs2_19.0.0-2499-ge0e48ba6-1jammy_amd64.deb ... 2024-03-28T14:04:39.780 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking libcephfs2 (19.0.0-2499-ge0e48ba6-1jammy) ... 2024-03-28T14:04:39.799 INFO:teuthology.orchestra.run.smithi171.stdout:Selecting previously unselected package python3-waitress. 2024-03-28T14:04:39.826 INFO:teuthology.orchestra.run.smithi171.stdout:Preparing to unpack .../049-python3-waitress_1.4.4-1.1ubuntu1_all.deb ... 2024-03-28T14:04:39.874 INFO:teuthology.orchestra.run.smithi171.stdout:Unpacking python3-waitress (1.4.4-1.1ubuntu1) ... 2024-03-28T14:04:40.058 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package python3-rados. 2024-03-28T14:04:40.074 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../013-python3-rados_19.0.0-2499-ge0e48ba6-1jammy_amd64.deb ... 2024-03-28T14:04:40.108 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking python3-rados (19.0.0-2499-ge0e48ba6-1jammy) ... 2024-03-28T14:04:40.360 INFO:teuthology.orchestra.run.smithi171.stdout:Selecting previously unselected package python3-tempita. 2024-03-28T14:04:40.390 INFO:teuthology.orchestra.run.smithi171.stdout:Preparing to unpack .../050-python3-tempita_0.5.2-6ubuntu1_all.deb ... 2024-03-28T14:04:40.402 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package python3-ceph-argparse. 2024-03-28T14:04:40.427 INFO:teuthology.orchestra.run.smithi171.stdout:Unpacking python3-tempita (0.5.2-6ubuntu1) ... 2024-03-28T14:04:40.429 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../014-python3-ceph-argparse_19.0.0-2499-ge0e48ba6-1jammy_all.deb ... 2024-03-28T14:04:40.461 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking python3-ceph-argparse (19.0.0-2499-ge0e48ba6-1jammy) ... 2024-03-28T14:04:40.680 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package python3-cephfs. 2024-03-28T14:04:40.695 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../015-python3-cephfs_19.0.0-2499-ge0e48ba6-1jammy_amd64.deb ... 2024-03-28T14:04:40.705 INFO:teuthology.orchestra.run.smithi171.stdout:Selecting previously unselected package python3-paste. 2024-03-28T14:04:40.731 INFO:teuthology.orchestra.run.smithi171.stdout:Preparing to unpack .../051-python3-paste_3.5.0+dfsg1-1_all.deb ... 2024-03-28T14:04:40.738 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking python3-cephfs (19.0.0-2499-ge0e48ba6-1jammy) ... 2024-03-28T14:04:40.763 INFO:teuthology.orchestra.run.smithi171.stdout:Unpacking python3-paste (3.5.0+dfsg1-1) ... 2024-03-28T14:04:41.057 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package python3-ceph-common. 2024-03-28T14:04:41.084 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../016-python3-ceph-common_19.0.0-2499-ge0e48ba6-1jammy_all.deb ... 2024-03-28T14:04:41.107 INFO:teuthology.orchestra.run.smithi171.stdout:Selecting previously unselected package python-pastedeploy-tpl. 2024-03-28T14:04:41.123 INFO:teuthology.orchestra.run.smithi171.stdout:Preparing to unpack .../052-python-pastedeploy-tpl_2.1.1-1_all.deb ... 2024-03-28T14:04:41.124 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking python3-ceph-common (19.0.0-2499-ge0e48ba6-1jammy) ... 2024-03-28T14:04:41.158 INFO:teuthology.orchestra.run.smithi171.stdout:Unpacking python-pastedeploy-tpl (2.1.1-1) ... 2024-03-28T14:04:41.436 INFO:teuthology.orchestra.run.smithi171.stdout:Selecting previously unselected package python3-pastedeploy. 2024-03-28T14:04:41.444 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package python3-prettytable. 2024-03-28T14:04:41.462 INFO:teuthology.orchestra.run.smithi171.stdout:Preparing to unpack .../053-python3-pastedeploy_2.1.1-1_all.deb ... 2024-03-28T14:04:41.470 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../017-python3-prettytable_2.5.0-2_all.deb ... 2024-03-28T14:04:41.502 INFO:teuthology.orchestra.run.smithi171.stdout:Unpacking python3-pastedeploy (2.1.1-1) ... 2024-03-28T14:04:41.510 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking python3-prettytable (2.5.0-2) ... 2024-03-28T14:04:41.771 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package python3-rbd. 2024-03-28T14:04:41.797 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../018-python3-rbd_19.0.0-2499-ge0e48ba6-1jammy_amd64.deb ... 2024-03-28T14:04:41.838 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking python3-rbd (19.0.0-2499-ge0e48ba6-1jammy) ... 2024-03-28T14:04:41.847 INFO:teuthology.orchestra.run.smithi171.stdout:Selecting previously unselected package python3-webtest. 2024-03-28T14:04:41.873 INFO:teuthology.orchestra.run.smithi171.stdout:Preparing to unpack .../054-python3-webtest_2.0.35-1_all.deb ... 2024-03-28T14:04:41.913 INFO:teuthology.orchestra.run.smithi171.stdout:Unpacking python3-webtest (2.0.35-1) ... 2024-03-28T14:04:42.124 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package librdkafka1:amd64. 2024-03-28T14:04:42.150 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../019-librdkafka1_1.8.0-1build1_amd64.deb ... 2024-03-28T14:04:42.174 INFO:teuthology.orchestra.run.smithi171.stdout:Selecting previously unselected package python3-pecan. 2024-03-28T14:04:42.190 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking librdkafka1:amd64 (1.8.0-1build1) ... 2024-03-28T14:04:42.201 INFO:teuthology.orchestra.run.smithi171.stdout:Preparing to unpack .../055-python3-pecan_1.3.3-4ubuntu2_all.deb ... 2024-03-28T14:04:42.233 INFO:teuthology.orchestra.run.smithi171.stdout:Unpacking python3-pecan (1.3.3-4ubuntu2) ... 2024-03-28T14:04:42.468 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package libreadline-dev:amd64. 2024-03-28T14:04:42.483 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../020-libreadline-dev_8.1.2-1_amd64.deb ... 2024-03-28T14:04:42.526 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking libreadline-dev:amd64 (8.1.2-1) ... 2024-03-28T14:04:42.563 INFO:teuthology.orchestra.run.smithi171.stdout:Selecting previously unselected package python3-certifi. 2024-03-28T14:04:42.589 INFO:teuthology.orchestra.run.smithi171.stdout:Preparing to unpack .../056-python3-certifi_2020.6.20-1_all.deb ... 2024-03-28T14:04:42.629 INFO:teuthology.orchestra.run.smithi171.stdout:Unpacking python3-certifi (2020.6.20-1) ... 2024-03-28T14:04:42.855 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package liblua5.3-dev:amd64. 2024-03-28T14:04:42.870 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../021-liblua5.3-dev_5.3.6-1build1_amd64.deb ... 2024-03-28T14:04:42.915 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking liblua5.3-dev:amd64 (5.3.6-1build1) ... 2024-03-28T14:04:42.982 INFO:teuthology.orchestra.run.smithi171.stdout:Selecting previously unselected package python3-idna. 2024-03-28T14:04:43.012 INFO:teuthology.orchestra.run.smithi171.stdout:Preparing to unpack .../057-python3-idna_3.3-1_all.deb ... 2024-03-28T14:04:43.049 INFO:teuthology.orchestra.run.smithi171.stdout:Unpacking python3-idna (3.3-1) ... 2024-03-28T14:04:43.199 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package lua5.1. 2024-03-28T14:04:43.226 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../022-lua5.1_5.1.5-8.1build4_amd64.deb ... 2024-03-28T14:04:43.265 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking lua5.1 (5.1.5-8.1build4) ... 2024-03-28T14:04:43.368 INFO:teuthology.orchestra.run.smithi171.stdout:Selecting previously unselected package python3-urllib3. 2024-03-28T14:04:43.395 INFO:teuthology.orchestra.run.smithi171.stdout:Preparing to unpack .../058-python3-urllib3_1.26.5-1~exp1ubuntu0.1_all.deb ... 2024-03-28T14:04:43.434 INFO:teuthology.orchestra.run.smithi171.stdout:Unpacking python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ... 2024-03-28T14:04:43.560 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package lua-any. 2024-03-28T14:04:43.575 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../023-lua-any_27ubuntu1_all.deb ... 2024-03-28T14:04:43.618 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking lua-any (27ubuntu1) ... 2024-03-28T14:04:43.771 INFO:teuthology.orchestra.run.smithi171.stdout:Selecting previously unselected package python3-requests. 2024-03-28T14:04:43.798 INFO:teuthology.orchestra.run.smithi171.stdout:Preparing to unpack .../059-python3-requests_2.25.1+dfsg-2ubuntu0.1_all.deb ... 2024-03-28T14:04:43.838 INFO:teuthology.orchestra.run.smithi171.stdout:Unpacking python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2024-03-28T14:04:43.871 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package zip. 2024-03-28T14:04:43.897 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../024-zip_3.0-12build2_amd64.deb ... 2024-03-28T14:04:43.938 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking zip (3.0-12build2) ... 2024-03-28T14:04:44.124 INFO:teuthology.orchestra.run.smithi171.stdout:Selecting previously unselected package python3-werkzeug. 2024-03-28T14:04:44.151 INFO:teuthology.orchestra.run.smithi171.stdout:Preparing to unpack .../060-python3-werkzeug_2.0.2+dfsg1-1ubuntu0.22.04.1_all.deb ... 2024-03-28T14:04:44.190 INFO:teuthology.orchestra.run.smithi171.stdout:Unpacking python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.1) ... 2024-03-28T14:04:44.382 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package unzip. 2024-03-28T14:04:44.409 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../025-unzip_6.0-26ubuntu3.2_amd64.deb ... 2024-03-28T14:04:44.616 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking unzip (6.0-26ubuntu3.2) ... 2024-03-28T14:04:45.377 INFO:teuthology.orchestra.run.smithi171.stdout:Selecting previously unselected package ceph-mgr-modules-core. 2024-03-28T14:04:45.404 INFO:teuthology.orchestra.run.smithi171.stdout:Preparing to unpack .../061-ceph-mgr-modules-core_19.0.0-2499-ge0e48ba6-1jammy_all.deb ... 2024-03-28T14:04:45.443 INFO:teuthology.orchestra.run.smithi171.stdout:Unpacking ceph-mgr-modules-core (19.0.0-2499-ge0e48ba6-1jammy) ... 2024-03-28T14:04:45.528 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package luarocks. 2024-03-28T14:04:45.555 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../026-luarocks_3.8.0+dfsg1-1_all.deb ... 2024-03-28T14:04:45.595 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking luarocks (3.8.0+dfsg1-1) ... 2024-03-28T14:04:45.782 INFO:teuthology.orchestra.run.smithi171.stdout:Selecting previously unselected package libsqlite3-mod-ceph. 2024-03-28T14:04:45.809 INFO:teuthology.orchestra.run.smithi171.stdout:Preparing to unpack .../062-libsqlite3-mod-ceph_19.0.0-2499-ge0e48ba6-1jammy_amd64.deb ... 2024-03-28T14:04:45.848 INFO:teuthology.orchestra.run.smithi171.stdout:Unpacking libsqlite3-mod-ceph (19.0.0-2499-ge0e48ba6-1jammy) ... 2024-03-28T14:04:45.969 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package librgw2. 2024-03-28T14:04:45.996 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../027-librgw2_19.0.0-2499-ge0e48ba6-1jammy_amd64.deb ... 2024-03-28T14:04:46.035 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking librgw2 (19.0.0-2499-ge0e48ba6-1jammy) ... 2024-03-28T14:04:46.151 INFO:teuthology.orchestra.run.smithi171.stdout:Selecting previously unselected package python3-bcrypt. 2024-03-28T14:04:46.178 INFO:teuthology.orchestra.run.smithi171.stdout:Preparing to unpack .../063-python3-bcrypt_3.2.0-1build1_amd64.deb ... 2024-03-28T14:04:46.218 INFO:teuthology.orchestra.run.smithi171.stdout:Unpacking python3-bcrypt (3.2.0-1build1) ... 2024-03-28T14:04:46.529 INFO:teuthology.orchestra.run.smithi171.stdout:Selecting previously unselected package python3-openssl. 2024-03-28T14:04:46.555 INFO:teuthology.orchestra.run.smithi171.stdout:Preparing to unpack .../064-python3-openssl_21.0.0-1_all.deb ... 2024-03-28T14:04:46.580 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package python3-rgw. 2024-03-28T14:04:46.595 INFO:teuthology.orchestra.run.smithi171.stdout:Unpacking python3-openssl (21.0.0-1) ... 2024-03-28T14:04:46.607 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../028-python3-rgw_19.0.0-2499-ge0e48ba6-1jammy_amd64.deb ... 2024-03-28T14:04:46.638 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking python3-rgw (19.0.0-2499-ge0e48ba6-1jammy) ... 2024-03-28T14:04:46.974 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package liboath0:amd64. 2024-03-28T14:04:46.999 INFO:teuthology.orchestra.run.smithi171.stdout:Selecting previously unselected package ceph-mgr. 2024-03-28T14:04:47.001 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../029-liboath0_2.6.7-3build1_amd64.deb ... 2024-03-28T14:04:47.025 INFO:teuthology.orchestra.run.smithi171.stdout:Preparing to unpack .../065-ceph-mgr_19.0.0-2499-ge0e48ba6-1jammy_amd64.deb ... 2024-03-28T14:04:47.040 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking liboath0:amd64 (2.6.7-3build1) ... 2024-03-28T14:04:47.073 INFO:teuthology.orchestra.run.smithi171.stdout:Unpacking ceph-mgr (19.0.0-2499-ge0e48ba6-1jammy) ... 2024-03-28T14:04:47.352 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package libradosstriper1. 2024-03-28T14:04:47.379 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../030-libradosstriper1_19.0.0-2499-ge0e48ba6-1jammy_amd64.deb ... 2024-03-28T14:04:47.418 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking libradosstriper1 (19.0.0-2499-ge0e48ba6-1jammy) ... 2024-03-28T14:04:47.552 INFO:teuthology.orchestra.run.smithi171.stdout:Selecting previously unselected package ceph-mon. 2024-03-28T14:04:47.578 INFO:teuthology.orchestra.run.smithi171.stdout:Preparing to unpack .../066-ceph-mon_19.0.0-2499-ge0e48ba6-1jammy_amd64.deb ... 2024-03-28T14:04:47.626 INFO:teuthology.orchestra.run.smithi171.stdout:Unpacking ceph-mon (19.0.0-2499-ge0e48ba6-1jammy) ... 2024-03-28T14:04:47.822 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package ceph-common. 2024-03-28T14:04:47.849 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../031-ceph-common_19.0.0-2499-ge0e48ba6-1jammy_amd64.deb ... 2024-03-28T14:04:47.888 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking ceph-common (19.0.0-2499-ge0e48ba6-1jammy) ... 2024-03-28T14:04:48.247 INFO:teuthology.orchestra.run.smithi171.stdout:Selecting previously unselected package ceph-osd. 2024-03-28T14:04:48.274 INFO:teuthology.orchestra.run.smithi171.stdout:Preparing to unpack .../067-ceph-osd_19.0.0-2499-ge0e48ba6-1jammy_amd64.deb ... 2024-03-28T14:04:48.313 INFO:teuthology.orchestra.run.smithi171.stdout:Unpacking ceph-osd (19.0.0-2499-ge0e48ba6-1jammy) ... 2024-03-28T14:04:49.193 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package ceph-base. 2024-03-28T14:04:49.220 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../032-ceph-base_19.0.0-2499-ge0e48ba6-1jammy_amd64.deb ... 2024-03-28T14:04:49.269 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking ceph-base (19.0.0-2499-ge0e48ba6-1jammy) ... 2024-03-28T14:04:49.375 INFO:teuthology.orchestra.run.smithi171.stdout:Selecting previously unselected package ceph. 2024-03-28T14:04:49.401 INFO:teuthology.orchestra.run.smithi171.stdout:Preparing to unpack .../068-ceph_19.0.0-2499-ge0e48ba6-1jammy_amd64.deb ... 2024-03-28T14:04:49.449 INFO:teuthology.orchestra.run.smithi171.stdout:Unpacking ceph (19.0.0-2499-ge0e48ba6-1jammy) ... 2024-03-28T14:04:49.844 INFO:teuthology.orchestra.run.smithi171.stdout:Selecting previously unselected package ceph-fuse. 2024-03-28T14:04:49.871 INFO:teuthology.orchestra.run.smithi171.stdout:Preparing to unpack .../069-ceph-fuse_19.0.0-2499-ge0e48ba6-1jammy_amd64.deb ... 2024-03-28T14:04:49.919 INFO:teuthology.orchestra.run.smithi171.stdout:Unpacking ceph-fuse (19.0.0-2499-ge0e48ba6-1jammy) ... 2024-03-28T14:04:49.997 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package python3-jaraco.functools. 2024-03-28T14:04:50.013 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../033-python3-jaraco.functools_3.4.0-2_all.deb ... 2024-03-28T14:04:50.056 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking python3-jaraco.functools (3.4.0-2) ... 2024-03-28T14:04:50.339 INFO:teuthology.orchestra.run.smithi171.stdout:Selecting previously unselected package ceph-mds. 2024-03-28T14:04:50.363 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package python3-cheroot. 2024-03-28T14:04:50.365 INFO:teuthology.orchestra.run.smithi171.stdout:Preparing to unpack .../070-ceph-mds_19.0.0-2499-ge0e48ba6-1jammy_amd64.deb ... 2024-03-28T14:04:50.390 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../034-python3-cheroot_8.5.2+ds1-1ubuntu3.1_all.deb ... 2024-03-28T14:04:50.405 INFO:teuthology.orchestra.run.smithi171.stdout:Unpacking ceph-mds (19.0.0-2499-ge0e48ba6-1jammy) ... 2024-03-28T14:04:50.429 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2024-03-28T14:04:50.749 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package python3-jaraco.classes. 2024-03-28T14:04:50.776 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../035-python3-jaraco.classes_3.2.1-3_all.deb ... 2024-03-28T14:04:50.815 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking python3-jaraco.classes (3.2.1-3) ... 2024-03-28T14:04:50.816 INFO:teuthology.orchestra.run.smithi171.stdout:Selecting previously unselected package cephadm. 2024-03-28T14:04:50.844 INFO:teuthology.orchestra.run.smithi171.stdout:Preparing to unpack .../071-cephadm_19.0.0-2499-ge0e48ba6-1jammy_amd64.deb ... 2024-03-28T14:04:50.891 INFO:teuthology.orchestra.run.smithi171.stdout:Unpacking cephadm (19.0.0-2499-ge0e48ba6-1jammy) ... 2024-03-28T14:04:51.118 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package python3-jaraco.text. 2024-03-28T14:04:51.145 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../036-python3-jaraco.text_3.6.0-2_all.deb ... 2024-03-28T14:04:51.184 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking python3-jaraco.text (3.6.0-2) ... 2024-03-28T14:04:51.269 INFO:teuthology.orchestra.run.smithi171.stdout:Selecting previously unselected package python3-asyncssh. 2024-03-28T14:04:51.296 INFO:teuthology.orchestra.run.smithi171.stdout:Preparing to unpack .../072-python3-asyncssh_2.5.0-1_all.deb ... 2024-03-28T14:04:51.344 INFO:teuthology.orchestra.run.smithi171.stdout:Unpacking python3-asyncssh (2.5.0-1) ... 2024-03-28T14:04:51.471 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package python3-jaraco.collections. 2024-03-28T14:04:51.498 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../037-python3-jaraco.collections_3.4.0-2_all.deb ... 2024-03-28T14:04:51.529 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking python3-jaraco.collections (3.4.0-2) ... 2024-03-28T14:04:51.714 INFO:teuthology.orchestra.run.smithi171.stdout:Selecting previously unselected package python-babel-localedata. 2024-03-28T14:04:51.740 INFO:teuthology.orchestra.run.smithi171.stdout:Preparing to unpack .../073-python-babel-localedata_2.8.0+dfsg.1-7_all.deb ... 2024-03-28T14:04:51.788 INFO:teuthology.orchestra.run.smithi171.stdout:Unpacking python-babel-localedata (2.8.0+dfsg.1-7) ... 2024-03-28T14:04:51.807 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package python3-tempora. 2024-03-28T14:04:51.833 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../038-python3-tempora_4.1.2-1_all.deb ... 2024-03-28T14:04:51.874 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking python3-tempora (4.1.2-1) ... 2024-03-28T14:04:52.185 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package python3-portend. 2024-03-28T14:04:52.211 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../039-python3-portend_3.0.0-1_all.deb ... 2024-03-28T14:04:52.251 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking python3-portend (3.0.0-1) ... 2024-03-28T14:04:52.529 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package python3-zc.lockfile. 2024-03-28T14:04:52.556 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../040-python3-zc.lockfile_2.0-1_all.deb ... 2024-03-28T14:04:52.596 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking python3-zc.lockfile (2.0-1) ... 2024-03-28T14:04:52.827 INFO:teuthology.orchestra.run.smithi171.stdout:Selecting previously unselected package python3-babel. 2024-03-28T14:04:52.849 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package python3-cherrypy3. 2024-03-28T14:04:52.854 INFO:teuthology.orchestra.run.smithi171.stdout:Preparing to unpack .../074-python3-babel_2.8.0+dfsg.1-7_all.deb ... 2024-03-28T14:04:52.865 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../041-python3-cherrypy3_18.6.1-4_all.deb ... 2024-03-28T14:04:52.893 INFO:teuthology.orchestra.run.smithi171.stdout:Unpacking python3-babel (2.8.0+dfsg.1-7) ... 2024-03-28T14:04:52.906 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking python3-cherrypy3 (18.6.1-4) ... 2024-03-28T14:04:53.196 INFO:teuthology.orchestra.run.smithi171.stdout:Selecting previously unselected package python3-jinja2. 2024-03-28T14:04:53.212 INFO:teuthology.orchestra.run.smithi171.stdout:Preparing to unpack .../075-python3-jinja2_3.0.3-1ubuntu0.1_all.deb ... 2024-03-28T14:04:53.252 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package python3-natsort. 2024-03-28T14:04:53.254 INFO:teuthology.orchestra.run.smithi171.stdout:Unpacking python3-jinja2 (3.0.3-1ubuntu0.1) ... 2024-03-28T14:04:53.278 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../042-python3-natsort_8.0.2-1_all.deb ... 2024-03-28T14:04:53.318 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking python3-natsort (8.0.2-1) ... 2024-03-28T14:04:53.583 INFO:teuthology.orchestra.run.smithi171.stdout:Selecting previously unselected package ceph-mgr-cephadm. 2024-03-28T14:04:53.610 INFO:teuthology.orchestra.run.smithi171.stdout:Preparing to unpack .../076-ceph-mgr-cephadm_19.0.0-2499-ge0e48ba6-1jammy_all.deb ... 2024-03-28T14:04:53.621 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package python3-logutils. 2024-03-28T14:04:53.637 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../043-python3-logutils_0.3.3-8_all.deb ... 2024-03-28T14:04:53.649 INFO:teuthology.orchestra.run.smithi171.stdout:Unpacking ceph-mgr-cephadm (19.0.0-2499-ge0e48ba6-1jammy) ... 2024-03-28T14:04:53.671 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking python3-logutils (0.3.3-8) ... 2024-03-28T14:04:53.944 INFO:teuthology.orchestra.run.smithi171.stdout:Selecting previously unselected package python3-repoze.lru. 2024-03-28T14:04:53.949 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package python3-markupsafe. 2024-03-28T14:04:53.962 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../044-python3-markupsafe_2.0.1-2build1_amd64.deb ... 2024-03-28T14:04:53.971 INFO:teuthology.orchestra.run.smithi171.stdout:Preparing to unpack .../077-python3-repoze.lru_0.7-2_all.deb ... 2024-03-28T14:04:53.999 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking python3-markupsafe (2.0.1-2build1) ... 2024-03-28T14:04:54.010 INFO:teuthology.orchestra.run.smithi171.stdout:Unpacking python3-repoze.lru (0.7-2) ... 2024-03-28T14:04:54.243 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package python3-mako. 2024-03-28T14:04:54.256 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../045-python3-mako_1.1.3+ds1-2ubuntu0.1_all.deb ... 2024-03-28T14:04:54.293 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2024-03-28T14:04:54.313 INFO:teuthology.orchestra.run.smithi171.stdout:Selecting previously unselected package python3-routes. 2024-03-28T14:04:54.342 INFO:teuthology.orchestra.run.smithi171.stdout:Preparing to unpack .../078-python3-routes_2.5.1-1ubuntu1_all.deb ... 2024-03-28T14:04:54.380 INFO:teuthology.orchestra.run.smithi171.stdout:Unpacking python3-routes (2.5.1-1ubuntu1) ... 2024-03-28T14:04:54.605 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package python3-simplegeneric. 2024-03-28T14:04:54.632 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../046-python3-simplegeneric_0.8.1-3_all.deb ... 2024-03-28T14:04:54.672 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking python3-simplegeneric (0.8.1-3) ... 2024-03-28T14:04:54.725 INFO:teuthology.orchestra.run.smithi171.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2024-03-28T14:04:54.752 INFO:teuthology.orchestra.run.smithi171.stdout:Preparing to unpack .../079-ceph-mgr-dashboard_19.0.0-2499-ge0e48ba6-1jammy_all.deb ... 2024-03-28T14:04:54.791 INFO:teuthology.orchestra.run.smithi171.stdout:Unpacking ceph-mgr-dashboard (19.0.0-2499-ge0e48ba6-1jammy) ... 2024-03-28T14:04:54.974 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package python3-singledispatch. 2024-03-28T14:04:55.001 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../047-python3-singledispatch_3.4.0.3-3_all.deb ... 2024-03-28T14:04:55.041 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking python3-singledispatch (3.4.0.3-3) ... 2024-03-28T14:04:55.368 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package python3-webob. 2024-03-28T14:04:55.395 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../048-python3-webob_1%3a1.8.6-1.1_all.deb ... 2024-03-28T14:04:55.435 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking python3-webob (1:1.8.6-1.1) ... 2024-03-28T14:04:55.772 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package python3-waitress. 2024-03-28T14:04:55.798 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../049-python3-waitress_1.4.4-1.1ubuntu1_all.deb ... 2024-03-28T14:04:55.833 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking python3-waitress (1.4.4-1.1ubuntu1) ... 2024-03-28T14:04:56.099 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package python3-tempita. 2024-03-28T14:04:56.126 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../050-python3-tempita_0.5.2-6ubuntu1_all.deb ... 2024-03-28T14:04:56.166 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking python3-tempita (0.5.2-6ubuntu1) ... 2024-03-28T14:04:56.313 INFO:teuthology.orchestra.run.smithi171.stdout:Selecting previously unselected package python3-sklearn-lib:amd64. 2024-03-28T14:04:56.329 INFO:teuthology.orchestra.run.smithi171.stdout:Preparing to unpack .../080-python3-sklearn-lib_0.23.2-5ubuntu6_amd64.deb ... 2024-03-28T14:04:56.363 INFO:teuthology.orchestra.run.smithi171.stdout:Unpacking python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2024-03-28T14:04:56.469 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package python3-paste. 2024-03-28T14:04:56.495 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../051-python3-paste_3.5.0+dfsg1-1_all.deb ... 2024-03-28T14:04:56.535 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking python3-paste (3.5.0+dfsg1-1) ... 2024-03-28T14:04:56.825 INFO:teuthology.orchestra.run.smithi171.stdout:Selecting previously unselected package python3-joblib. 2024-03-28T14:04:56.840 INFO:teuthology.orchestra.run.smithi171.stdout:Preparing to unpack .../081-python3-joblib_0.17.0-4ubuntu1_all.deb ... 2024-03-28T14:04:56.847 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package python-pastedeploy-tpl. 2024-03-28T14:04:56.862 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../052-python-pastedeploy-tpl_2.1.1-1_all.deb ... 2024-03-28T14:04:56.874 INFO:teuthology.orchestra.run.smithi171.stdout:Unpacking python3-joblib (0.17.0-4ubuntu1) ... 2024-03-28T14:04:56.905 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking python-pastedeploy-tpl (2.1.1-1) ... 2024-03-28T14:04:57.233 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package python3-pastedeploy. 2024-03-28T14:04:57.237 INFO:teuthology.orchestra.run.smithi171.stdout:Selecting previously unselected package python3-threadpoolctl. 2024-03-28T14:04:57.259 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../053-python3-pastedeploy_2.1.1-1_all.deb ... 2024-03-28T14:04:57.264 INFO:teuthology.orchestra.run.smithi171.stdout:Preparing to unpack .../082-python3-threadpoolctl_3.1.0-1_all.deb ... 2024-03-28T14:04:57.299 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking python3-pastedeploy (2.1.1-1) ... 2024-03-28T14:04:57.397 INFO:teuthology.orchestra.run.smithi171.stdout:Unpacking python3-threadpoolctl (3.1.0-1) ... 2024-03-28T14:04:57.602 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package python3-webtest. 2024-03-28T14:04:57.629 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../054-python3-webtest_2.0.35-1_all.deb ... 2024-03-28T14:04:57.650 INFO:teuthology.orchestra.run.smithi171.stdout:Selecting previously unselected package python3-sklearn. 2024-03-28T14:04:57.668 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking python3-webtest (2.0.35-1) ... 2024-03-28T14:04:57.677 INFO:teuthology.orchestra.run.smithi171.stdout:Preparing to unpack .../083-python3-sklearn_0.23.2-5ubuntu6_all.deb ... 2024-03-28T14:04:57.716 INFO:teuthology.orchestra.run.smithi171.stdout:Unpacking python3-sklearn (0.23.2-5ubuntu6) ... 2024-03-28T14:04:57.963 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package python3-pecan. 2024-03-28T14:04:57.990 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../055-python3-pecan_1.3.3-4ubuntu2_all.deb ... 2024-03-28T14:04:58.030 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking python3-pecan (1.3.3-4ubuntu2) ... 2024-03-28T14:04:58.376 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package python3-certifi. 2024-03-28T14:04:58.403 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../056-python3-certifi_2020.6.20-1_all.deb ... 2024-03-28T14:04:58.425 INFO:teuthology.orchestra.run.smithi171.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2024-03-28T14:04:58.434 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking python3-certifi (2020.6.20-1) ... 2024-03-28T14:04:58.441 INFO:teuthology.orchestra.run.smithi171.stdout:Preparing to unpack .../084-ceph-mgr-diskprediction-local_19.0.0-2499-ge0e48ba6-1jammy_all.deb ... 2024-03-28T14:04:58.483 INFO:teuthology.orchestra.run.smithi171.stdout:Unpacking ceph-mgr-diskprediction-local (19.0.0-2499-ge0e48ba6-1jammy) ... 2024-03-28T14:04:58.737 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package python3-idna. 2024-03-28T14:04:58.764 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../057-python3-idna_3.3-1_all.deb ... 2024-03-28T14:04:58.812 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking python3-idna (3.3-1) ... 2024-03-28T14:04:59.107 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package python3-urllib3. 2024-03-28T14:04:59.133 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../058-python3-urllib3_1.26.5-1~exp1ubuntu0.1_all.deb ... 2024-03-28T14:04:59.173 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ... 2024-03-28T14:04:59.462 INFO:teuthology.orchestra.run.smithi171.stdout:Selecting previously unselected package python3-cachetools. 2024-03-28T14:04:59.489 INFO:teuthology.orchestra.run.smithi171.stdout:Preparing to unpack .../085-python3-cachetools_5.0.0-1_all.deb ... 2024-03-28T14:04:59.493 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package python3-requests. 2024-03-28T14:04:59.519 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../059-python3-requests_2.25.1+dfsg-2ubuntu0.1_all.deb ... 2024-03-28T14:04:59.528 INFO:teuthology.orchestra.run.smithi171.stdout:Unpacking python3-cachetools (5.0.0-1) ... 2024-03-28T14:04:59.559 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2024-03-28T14:04:59.831 INFO:teuthology.orchestra.run.smithi171.stdout:Selecting previously unselected package python3-pyasn1. 2024-03-28T14:04:59.858 INFO:teuthology.orchestra.run.smithi171.stdout:Preparing to unpack .../086-python3-pyasn1_0.4.8-1_all.deb ... 2024-03-28T14:04:59.888 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package python3-werkzeug. 2024-03-28T14:04:59.897 INFO:teuthology.orchestra.run.smithi171.stdout:Unpacking python3-pyasn1 (0.4.8-1) ... 2024-03-28T14:04:59.914 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../060-python3-werkzeug_2.0.2+dfsg1-1ubuntu0.22.04.1_all.deb ... 2024-03-28T14:04:59.954 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.1) ... 2024-03-28T14:05:00.234 INFO:teuthology.orchestra.run.smithi171.stdout:Selecting previously unselected package python3-pyasn1-modules. 2024-03-28T14:05:00.261 INFO:teuthology.orchestra.run.smithi171.stdout:Preparing to unpack .../087-python3-pyasn1-modules_0.2.1-1_all.deb ... 2024-03-28T14:05:00.301 INFO:teuthology.orchestra.run.smithi171.stdout:Unpacking python3-pyasn1-modules (0.2.1-1) ... 2024-03-28T14:05:00.440 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package ceph-mgr-modules-core. 2024-03-28T14:05:00.456 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../061-ceph-mgr-modules-core_19.0.0-2499-ge0e48ba6-1jammy_all.deb ... 2024-03-28T14:05:00.498 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking ceph-mgr-modules-core (19.0.0-2499-ge0e48ba6-1jammy) ... 2024-03-28T14:05:00.654 INFO:teuthology.orchestra.run.smithi171.stdout:Selecting previously unselected package python3-rsa. 2024-03-28T14:05:00.681 INFO:teuthology.orchestra.run.smithi171.stdout:Preparing to unpack .../088-python3-rsa_4.8-1_all.deb ... 2024-03-28T14:05:00.720 INFO:teuthology.orchestra.run.smithi171.stdout:Unpacking python3-rsa (4.8-1) ... 2024-03-28T14:05:00.912 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package libsqlite3-mod-ceph. 2024-03-28T14:05:00.939 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../062-libsqlite3-mod-ceph_19.0.0-2499-ge0e48ba6-1jammy_amd64.deb ... 2024-03-28T14:05:00.978 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking libsqlite3-mod-ceph (19.0.0-2499-ge0e48ba6-1jammy) ... 2024-03-28T14:05:01.041 INFO:teuthology.orchestra.run.smithi171.stdout:Selecting previously unselected package python3-google-auth. 2024-03-28T14:05:01.068 INFO:teuthology.orchestra.run.smithi171.stdout:Preparing to unpack .../089-python3-google-auth_1.5.1-3_all.deb ... 2024-03-28T14:05:01.107 INFO:teuthology.orchestra.run.smithi171.stdout:Unpacking python3-google-auth (1.5.1-3) ... 2024-03-28T14:05:01.247 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package python3-bcrypt. 2024-03-28T14:05:01.263 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../063-python3-bcrypt_3.2.0-1build1_amd64.deb ... 2024-03-28T14:05:01.297 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking python3-bcrypt (3.2.0-1build1) ... 2024-03-28T14:05:01.411 INFO:teuthology.orchestra.run.smithi171.stdout:Selecting previously unselected package python3-requests-oauthlib. 2024-03-28T14:05:01.438 INFO:teuthology.orchestra.run.smithi171.stdout:Preparing to unpack .../090-python3-requests-oauthlib_1.3.0+ds-0.1_all.deb ... 2024-03-28T14:05:01.502 INFO:teuthology.orchestra.run.smithi171.stdout:Unpacking python3-requests-oauthlib (1.3.0+ds-0.1) ... 2024-03-28T14:05:01.592 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package python3-openssl. 2024-03-28T14:05:01.619 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../064-python3-openssl_21.0.0-1_all.deb ... 2024-03-28T14:05:01.658 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking python3-openssl (21.0.0-1) ... 2024-03-28T14:05:01.789 INFO:teuthology.orchestra.run.smithi171.stdout:Selecting previously unselected package python3-websocket. 2024-03-28T14:05:01.817 INFO:teuthology.orchestra.run.smithi171.stdout:Preparing to unpack .../091-python3-websocket_1.2.3-1_all.deb ... 2024-03-28T14:05:01.855 INFO:teuthology.orchestra.run.smithi171.stdout:Unpacking python3-websocket (1.2.3-1) ... 2024-03-28T14:05:02.037 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package ceph-mgr. 2024-03-28T14:05:02.064 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../065-ceph-mgr_19.0.0-2499-ge0e48ba6-1jammy_amd64.deb ... 2024-03-28T14:05:02.103 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking ceph-mgr (19.0.0-2499-ge0e48ba6-1jammy) ... 2024-03-28T14:05:02.175 INFO:teuthology.orchestra.run.smithi171.stdout:Selecting previously unselected package python3-kubernetes. 2024-03-28T14:05:02.203 INFO:teuthology.orchestra.run.smithi171.stdout:Preparing to unpack .../092-python3-kubernetes_12.0.1-1ubuntu1_all.deb ... 2024-03-28T14:05:02.242 INFO:teuthology.orchestra.run.smithi171.stdout:Unpacking python3-kubernetes (12.0.1-1ubuntu1) ... 2024-03-28T14:05:02.456 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package ceph-mon. 2024-03-28T14:05:02.484 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../066-ceph-mon_19.0.0-2499-ge0e48ba6-1jammy_amd64.deb ... 2024-03-28T14:05:02.523 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking ceph-mon (19.0.0-2499-ge0e48ba6-1jammy) ... 2024-03-28T14:05:02.908 INFO:teuthology.orchestra.run.smithi171.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2024-03-28T14:05:02.924 INFO:teuthology.orchestra.run.smithi171.stdout:Preparing to unpack .../093-ceph-mgr-k8sevents_19.0.0-2499-ge0e48ba6-1jammy_all.deb ... 2024-03-28T14:05:02.966 INFO:teuthology.orchestra.run.smithi171.stdout:Unpacking ceph-mgr-k8sevents (19.0.0-2499-ge0e48ba6-1jammy) ... 2024-03-28T14:05:03.101 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package ceph-osd. 2024-03-28T14:05:03.128 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../067-ceph-osd_19.0.0-2499-ge0e48ba6-1jammy_amd64.deb ... 2024-03-28T14:05:03.167 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking ceph-osd (19.0.0-2499-ge0e48ba6-1jammy) ... 2024-03-28T14:05:03.243 INFO:teuthology.orchestra.run.smithi171.stdout:Selecting previously unselected package libonig5:amd64. 2024-03-28T14:05:03.259 INFO:teuthology.orchestra.run.smithi171.stdout:Preparing to unpack .../094-libonig5_6.9.7.1-2build1_amd64.deb ... 2024-03-28T14:05:03.293 INFO:teuthology.orchestra.run.smithi171.stdout:Unpacking libonig5:amd64 (6.9.7.1-2build1) ... 2024-03-28T14:05:03.630 INFO:teuthology.orchestra.run.smithi171.stdout:Selecting previously unselected package libjq1:amd64. 2024-03-28T14:05:03.657 INFO:teuthology.orchestra.run.smithi171.stdout:Preparing to unpack .../095-libjq1_1.6-2.1ubuntu3_amd64.deb ... 2024-03-28T14:05:03.696 INFO:teuthology.orchestra.run.smithi171.stdout:Unpacking libjq1:amd64 (1.6-2.1ubuntu3) ... 2024-03-28T14:05:03.974 INFO:teuthology.orchestra.run.smithi171.stdout:Selecting previously unselected package jq. 2024-03-28T14:05:04.001 INFO:teuthology.orchestra.run.smithi171.stdout:Preparing to unpack .../096-jq_1.6-2.1ubuntu3_amd64.deb ... 2024-03-28T14:05:04.040 INFO:teuthology.orchestra.run.smithi171.stdout:Unpacking jq (1.6-2.1ubuntu3) ... 2024-03-28T14:05:04.221 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package ceph. 2024-03-28T14:05:04.248 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../068-ceph_19.0.0-2499-ge0e48ba6-1jammy_amd64.deb ... 2024-03-28T14:05:04.284 INFO:teuthology.orchestra.run.smithi171.stdout:Selecting previously unselected package socat. 2024-03-28T14:05:04.287 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking ceph (19.0.0-2499-ge0e48ba6-1jammy) ... 2024-03-28T14:05:04.300 INFO:teuthology.orchestra.run.smithi171.stdout:Preparing to unpack .../097-socat_1.7.4.1-3ubuntu4_amd64.deb ... 2024-03-28T14:05:04.334 INFO:teuthology.orchestra.run.smithi171.stdout:Unpacking socat (1.7.4.1-3ubuntu4) ... 2024-03-28T14:05:04.548 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package ceph-fuse. 2024-03-28T14:05:04.575 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../069-ceph-fuse_19.0.0-2499-ge0e48ba6-1jammy_amd64.deb ... 2024-03-28T14:05:04.606 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking ceph-fuse (19.0.0-2499-ge0e48ba6-1jammy) ... 2024-03-28T14:05:04.637 INFO:teuthology.orchestra.run.smithi171.stdout:Selecting previously unselected package xmlstarlet. 2024-03-28T14:05:04.653 INFO:teuthology.orchestra.run.smithi171.stdout:Preparing to unpack .../098-xmlstarlet_1.6.1-2.1_amd64.deb ... 2024-03-28T14:05:04.687 INFO:teuthology.orchestra.run.smithi171.stdout:Unpacking xmlstarlet (1.6.1-2.1) ... 2024-03-28T14:05:04.968 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package ceph-mds. 2024-03-28T14:05:04.995 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../070-ceph-mds_19.0.0-2499-ge0e48ba6-1jammy_amd64.deb ... 2024-03-28T14:05:05.034 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking ceph-mds (19.0.0-2499-ge0e48ba6-1jammy) ... 2024-03-28T14:05:05.084 INFO:teuthology.orchestra.run.smithi171.stdout:Selecting previously unselected package ceph-test. 2024-03-28T14:05:05.111 INFO:teuthology.orchestra.run.smithi171.stdout:Preparing to unpack .../099-ceph-test_19.0.0-2499-ge0e48ba6-1jammy_amd64.deb ... 2024-03-28T14:05:05.150 INFO:teuthology.orchestra.run.smithi171.stdout:Unpacking ceph-test (19.0.0-2499-ge0e48ba6-1jammy) ... 2024-03-28T14:05:05.454 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package cephadm. 2024-03-28T14:05:05.481 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../071-cephadm_19.0.0-2499-ge0e48ba6-1jammy_amd64.deb ... 2024-03-28T14:05:05.521 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking cephadm (19.0.0-2499-ge0e48ba6-1jammy) ... 2024-03-28T14:05:05.999 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package python3-asyncssh. 2024-03-28T14:05:06.026 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../072-python3-asyncssh_2.5.0-1_all.deb ... 2024-03-28T14:05:06.065 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking python3-asyncssh (2.5.0-1) ... 2024-03-28T14:05:06.376 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package python-babel-localedata. 2024-03-28T14:05:06.392 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../073-python-babel-localedata_2.8.0+dfsg.1-7_all.deb ... 2024-03-28T14:05:06.434 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking python-babel-localedata (2.8.0+dfsg.1-7) ... 2024-03-28T14:05:07.387 INFO:teuthology.orchestra.run.smithi171.stdout:Selecting previously unselected package ceph-volume. 2024-03-28T14:05:07.415 INFO:teuthology.orchestra.run.smithi171.stdout:Preparing to unpack .../100-ceph-volume_19.0.0-2499-ge0e48ba6-1jammy_all.deb ... 2024-03-28T14:05:07.454 INFO:teuthology.orchestra.run.smithi171.stdout:Unpacking ceph-volume (19.0.0-2499-ge0e48ba6-1jammy) ... 2024-03-28T14:05:07.481 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package python3-babel. 2024-03-28T14:05:07.508 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../074-python3-babel_2.8.0+dfsg.1-7_all.deb ... 2024-03-28T14:05:07.547 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking python3-babel (2.8.0+dfsg.1-7) ... 2024-03-28T14:05:07.833 INFO:teuthology.orchestra.run.smithi171.stdout:Selecting previously unselected package cephfs-mirror. 2024-03-28T14:05:07.861 INFO:teuthology.orchestra.run.smithi171.stdout:Preparing to unpack .../101-cephfs-mirror_19.0.0-2499-ge0e48ba6-1jammy_amd64.deb ... 2024-03-28T14:05:07.899 INFO:teuthology.orchestra.run.smithi171.stdout:Unpacking cephfs-mirror (19.0.0-2499-ge0e48ba6-1jammy) ... 2024-03-28T14:05:07.900 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package python3-jinja2. 2024-03-28T14:05:07.927 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../075-python3-jinja2_3.0.3-1ubuntu0.1_all.deb ... 2024-03-28T14:05:07.967 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking python3-jinja2 (3.0.3-1ubuntu0.1) ... 2024-03-28T14:05:08.219 INFO:teuthology.orchestra.run.smithi171.stdout:Selecting previously unselected package python3-colorama. 2024-03-28T14:05:08.246 INFO:teuthology.orchestra.run.smithi171.stdout:Preparing to unpack .../102-python3-colorama_0.4.4-1_all.deb ... 2024-03-28T14:05:08.285 INFO:teuthology.orchestra.run.smithi171.stdout:Unpacking python3-colorama (0.4.4-1) ... 2024-03-28T14:05:08.320 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package ceph-mgr-cephadm. 2024-03-28T14:05:08.346 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../076-ceph-mgr-cephadm_19.0.0-2499-ge0e48ba6-1jammy_all.deb ... 2024-03-28T14:05:08.386 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking ceph-mgr-cephadm (19.0.0-2499-ge0e48ba6-1jammy) ... 2024-03-28T14:05:08.546 INFO:teuthology.orchestra.run.smithi171.stdout:Selecting previously unselected package cephfs-shell. 2024-03-28T14:05:08.562 INFO:teuthology.orchestra.run.smithi171.stdout:Preparing to unpack .../103-cephfs-shell_19.0.0-2499-ge0e48ba6-1jammy_all.deb ... 2024-03-28T14:05:08.604 INFO:teuthology.orchestra.run.smithi171.stdout:Unpacking cephfs-shell (19.0.0-2499-ge0e48ba6-1jammy) ... 2024-03-28T14:05:08.723 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package python3-repoze.lru. 2024-03-28T14:05:08.750 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../077-python3-repoze.lru_0.7-2_all.deb ... 2024-03-28T14:05:08.789 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking python3-repoze.lru (0.7-2) ... 2024-03-28T14:05:08.840 INFO:teuthology.orchestra.run.smithi171.stdout:Selecting previously unselected package cephfs-top. 2024-03-28T14:05:08.856 INFO:teuthology.orchestra.run.smithi171.stdout:Preparing to unpack .../104-cephfs-top_19.0.0-2499-ge0e48ba6-1jammy_all.deb ... 2024-03-28T14:05:08.907 INFO:teuthology.orchestra.run.smithi171.stdout:Unpacking cephfs-top (19.0.0-2499-ge0e48ba6-1jammy) ... 2024-03-28T14:05:09.093 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package python3-routes. 2024-03-28T14:05:09.119 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../078-python3-routes_2.5.1-1ubuntu1_all.deb ... 2024-03-28T14:05:09.159 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking python3-routes (2.5.1-1ubuntu1) ... 2024-03-28T14:05:09.176 INFO:teuthology.orchestra.run.smithi171.stdout:Selecting previously unselected package libcephfs-dev. 2024-03-28T14:05:09.203 INFO:teuthology.orchestra.run.smithi171.stdout:Preparing to unpack .../105-libcephfs-dev_19.0.0-2499-ge0e48ba6-1jammy_amd64.deb ... 2024-03-28T14:05:09.242 INFO:teuthology.orchestra.run.smithi171.stdout:Unpacking libcephfs-dev (19.0.0-2499-ge0e48ba6-1jammy) ... 2024-03-28T14:05:09.495 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2024-03-28T14:05:09.523 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../079-ceph-mgr-dashboard_19.0.0-2499-ge0e48ba6-1jammy_all.deb ... 2024-03-28T14:05:09.562 INFO:teuthology.orchestra.run.smithi171.stdout:Selecting previously unselected package lua-socket:amd64. 2024-03-28T14:05:09.570 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking ceph-mgr-dashboard (19.0.0-2499-ge0e48ba6-1jammy) ... 2024-03-28T14:05:09.589 INFO:teuthology.orchestra.run.smithi171.stdout:Preparing to unpack .../106-lua-socket_3.0~rc1+git+ac3201d-6_amd64.deb ... 2024-03-28T14:05:09.628 INFO:teuthology.orchestra.run.smithi171.stdout:Unpacking lua-socket:amd64 (3.0~rc1+git+ac3201d-6) ... 2024-03-28T14:05:09.974 INFO:teuthology.orchestra.run.smithi171.stdout:Selecting previously unselected package lua-sec:amd64. 2024-03-28T14:05:10.002 INFO:teuthology.orchestra.run.smithi171.stdout:Preparing to unpack .../107-lua-sec_1.0.2-1_amd64.deb ... 2024-03-28T14:05:10.041 INFO:teuthology.orchestra.run.smithi171.stdout:Unpacking lua-sec:amd64 (1.0.2-1) ... 2024-03-28T14:05:10.420 INFO:teuthology.orchestra.run.smithi171.stdout:Selecting previously unselected package nvme-cli. 2024-03-28T14:05:10.447 INFO:teuthology.orchestra.run.smithi171.stdout:Preparing to unpack .../108-nvme-cli_1.16-3ubuntu0.1_amd64.deb ... 2024-03-28T14:05:10.486 INFO:teuthology.orchestra.run.smithi171.stdout:Unpacking nvme-cli (1.16-3ubuntu0.1) ... 2024-03-28T14:05:10.866 INFO:teuthology.orchestra.run.smithi171.stdout:Selecting previously unselected package pkg-config. 2024-03-28T14:05:10.894 INFO:teuthology.orchestra.run.smithi171.stdout:Preparing to unpack .../109-pkg-config_0.29.2-1ubuntu3_amd64.deb ... 2024-03-28T14:05:10.933 INFO:teuthology.orchestra.run.smithi171.stdout:Unpacking pkg-config (0.29.2-1ubuntu3) ... 2024-03-28T14:05:11.066 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package python3-sklearn-lib:amd64. 2024-03-28T14:05:11.083 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../080-python3-sklearn-lib_0.23.2-5ubuntu6_amd64.deb ... 2024-03-28T14:05:11.125 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2024-03-28T14:05:11.152 INFO:teuthology.orchestra.run.smithi171.stdout:Selecting previously unselected package python-asyncssh-doc. 2024-03-28T14:05:11.169 INFO:teuthology.orchestra.run.smithi171.stdout:Preparing to unpack .../110-python-asyncssh-doc_2.5.0-1_all.deb ... 2024-03-28T14:05:11.202 INFO:teuthology.orchestra.run.smithi171.stdout:Unpacking python-asyncssh-doc (2.5.0-1) ... 2024-03-28T14:05:11.670 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package python3-joblib. 2024-03-28T14:05:11.697 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../081-python3-joblib_0.17.0-4ubuntu1_all.deb ... 2024-03-28T14:05:11.736 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking python3-joblib (0.17.0-4ubuntu1) ... 2024-03-28T14:05:11.988 INFO:teuthology.orchestra.run.smithi171.stdout:Selecting previously unselected package python3-iniconfig. 2024-03-28T14:05:12.016 INFO:teuthology.orchestra.run.smithi171.stdout:Preparing to unpack .../111-python3-iniconfig_1.1.1-2_all.deb ... 2024-03-28T14:05:12.031 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package python3-threadpoolctl. 2024-03-28T14:05:12.047 INFO:teuthology.orchestra.run.smithi171.stdout:Unpacking python3-iniconfig (1.1.1-2) ... 2024-03-28T14:05:12.047 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../082-python3-threadpoolctl_3.1.0-1_all.deb ... 2024-03-28T14:05:12.176 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking python3-threadpoolctl (3.1.0-1) ... 2024-03-28T14:05:12.316 INFO:teuthology.orchestra.run.smithi171.stdout:Selecting previously unselected package python3-pastescript. 2024-03-28T14:05:12.344 INFO:teuthology.orchestra.run.smithi171.stdout:Preparing to unpack .../112-python3-pastescript_2.0.2-4_all.deb ... 2024-03-28T14:05:12.382 INFO:teuthology.orchestra.run.smithi171.stdout:Unpacking python3-pastescript (2.0.2-4) ... 2024-03-28T14:05:12.453 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package python3-sklearn. 2024-03-28T14:05:12.480 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../083-python3-sklearn_0.23.2-5ubuntu6_all.deb ... 2024-03-28T14:05:12.519 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking python3-sklearn (0.23.2-5ubuntu6) ... 2024-03-28T14:05:12.702 INFO:teuthology.orchestra.run.smithi171.stdout:Selecting previously unselected package python3-pluggy. 2024-03-28T14:05:12.730 INFO:teuthology.orchestra.run.smithi171.stdout:Preparing to unpack .../113-python3-pluggy_0.13.0-7.1_all.deb ... 2024-03-28T14:05:12.769 INFO:teuthology.orchestra.run.smithi171.stdout:Unpacking python3-pluggy (0.13.0-7.1) ... 2024-03-28T14:05:13.080 INFO:teuthology.orchestra.run.smithi171.stdout:Selecting previously unselected package python3-psutil. 2024-03-28T14:05:13.108 INFO:teuthology.orchestra.run.smithi171.stdout:Preparing to unpack .../114-python3-psutil_5.9.0-1build1_amd64.deb ... 2024-03-28T14:05:13.146 INFO:teuthology.orchestra.run.smithi171.stdout:Unpacking python3-psutil (5.9.0-1build1) ... 2024-03-28T14:05:13.178 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2024-03-28T14:05:13.207 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../084-ceph-mgr-diskprediction-local_19.0.0-2499-ge0e48ba6-1jammy_all.deb ... 2024-03-28T14:05:13.253 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking ceph-mgr-diskprediction-local (19.0.0-2499-ge0e48ba6-1jammy) ... 2024-03-28T14:05:13.491 INFO:teuthology.orchestra.run.smithi171.stdout:Selecting previously unselected package python3-py. 2024-03-28T14:05:13.519 INFO:teuthology.orchestra.run.smithi171.stdout:Preparing to unpack .../115-python3-py_1.10.0-1_all.deb ... 2024-03-28T14:05:13.558 INFO:teuthology.orchestra.run.smithi171.stdout:Unpacking python3-py (1.10.0-1) ... 2024-03-28T14:05:13.886 INFO:teuthology.orchestra.run.smithi171.stdout:Selecting previously unselected package python3-pygments. 2024-03-28T14:05:13.914 INFO:teuthology.orchestra.run.smithi171.stdout:Preparing to unpack .../116-python3-pygments_2.11.2+dfsg-2_all.deb ... 2024-03-28T14:05:13.952 INFO:teuthology.orchestra.run.smithi171.stdout:Unpacking python3-pygments (2.11.2+dfsg-2) ... 2024-03-28T14:05:14.181 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package python3-cachetools. 2024-03-28T14:05:14.208 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../085-python3-cachetools_5.0.0-1_all.deb ... 2024-03-28T14:05:14.248 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking python3-cachetools (5.0.0-1) ... 2024-03-28T14:05:14.473 INFO:teuthology.orchestra.run.smithi171.stdout:Selecting previously unselected package python3-pyinotify. 2024-03-28T14:05:14.501 INFO:teuthology.orchestra.run.smithi171.stdout:Preparing to unpack .../117-python3-pyinotify_0.9.6-1.3_all.deb ... 2024-03-28T14:05:14.548 INFO:teuthology.orchestra.run.smithi171.stdout:Unpacking python3-pyinotify (0.9.6-1.3) ... 2024-03-28T14:05:14.550 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package python3-pyasn1. 2024-03-28T14:05:14.578 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../086-python3-pyasn1_0.4.8-1_all.deb ... 2024-03-28T14:05:14.617 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking python3-pyasn1 (0.4.8-1) ... 2024-03-28T14:05:14.926 INFO:teuthology.orchestra.run.smithi171.stdout:Selecting previously unselected package python3-toml. 2024-03-28T14:05:14.953 INFO:teuthology.orchestra.run.smithi171.stdout:Preparing to unpack .../118-python3-toml_0.10.2-1_all.deb ... 2024-03-28T14:05:14.987 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package python3-pyasn1-modules. 2024-03-28T14:05:15.000 INFO:teuthology.orchestra.run.smithi171.stdout:Unpacking python3-toml (0.10.2-1) ... 2024-03-28T14:05:15.014 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../087-python3-pyasn1-modules_0.2.1-1_all.deb ... 2024-03-28T14:05:15.054 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking python3-pyasn1-modules (0.2.1-1) ... 2024-03-28T14:05:15.312 INFO:teuthology.orchestra.run.smithi171.stdout:Selecting previously unselected package python3-pytest. 2024-03-28T14:05:15.339 INFO:teuthology.orchestra.run.smithi171.stdout:Preparing to unpack .../119-python3-pytest_6.2.5-1ubuntu2_all.deb ... 2024-03-28T14:05:15.340 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package python3-rsa. 2024-03-28T14:05:15.367 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../088-python3-rsa_4.8-1_all.deb ... 2024-03-28T14:05:15.378 INFO:teuthology.orchestra.run.smithi171.stdout:Unpacking python3-pytest (6.2.5-1ubuntu2) ... 2024-03-28T14:05:15.398 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking python3-rsa (4.8-1) ... 2024-03-28T14:05:15.702 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package python3-google-auth. 2024-03-28T14:05:15.729 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../089-python3-google-auth_1.5.1-3_all.deb ... 2024-03-28T14:05:15.768 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking python3-google-auth (1.5.1-3) ... 2024-03-28T14:05:15.782 INFO:teuthology.orchestra.run.smithi171.stdout:Selecting previously unselected package python3-simplejson. 2024-03-28T14:05:15.810 INFO:teuthology.orchestra.run.smithi171.stdout:Preparing to unpack .../120-python3-simplejson_3.17.6-1build1_amd64.deb ... 2024-03-28T14:05:15.857 INFO:teuthology.orchestra.run.smithi171.stdout:Unpacking python3-simplejson (3.17.6-1build1) ... 2024-03-28T14:05:16.088 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package python3-requests-oauthlib. 2024-03-28T14:05:16.115 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../090-python3-requests-oauthlib_1.3.0+ds-0.1_all.deb ... 2024-03-28T14:05:16.146 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking python3-requests-oauthlib (1.3.0+ds-0.1) ... 2024-03-28T14:05:16.184 INFO:teuthology.orchestra.run.smithi171.stdout:Selecting previously unselected package qttranslations5-l10n. 2024-03-28T14:05:16.212 INFO:teuthology.orchestra.run.smithi171.stdout:Preparing to unpack .../121-qttranslations5-l10n_5.15.3-1_all.deb ... 2024-03-28T14:05:16.259 INFO:teuthology.orchestra.run.smithi171.stdout:Unpacking qttranslations5-l10n (5.15.3-1) ... 2024-03-28T14:05:16.441 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package python3-websocket. 2024-03-28T14:05:16.470 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../091-python3-websocket_1.2.3-1_all.deb ... 2024-03-28T14:05:16.507 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking python3-websocket (1.2.3-1) ... 2024-03-28T14:05:16.819 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package python3-kubernetes. 2024-03-28T14:05:16.846 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../092-python3-kubernetes_12.0.1-1ubuntu1_all.deb ... 2024-03-28T14:05:16.885 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking python3-kubernetes (12.0.1-1ubuntu1) ... 2024-03-28T14:05:17.022 INFO:teuthology.orchestra.run.smithi171.stdout:Selecting previously unselected package radosgw. 2024-03-28T14:05:17.049 INFO:teuthology.orchestra.run.smithi171.stdout:Preparing to unpack .../122-radosgw_19.0.0-2499-ge0e48ba6-1jammy_amd64.deb ... 2024-03-28T14:05:17.088 INFO:teuthology.orchestra.run.smithi171.stdout:Unpacking radosgw (19.0.0-2499-ge0e48ba6-1jammy) ... 2024-03-28T14:05:17.549 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2024-03-28T14:05:17.578 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../093-ceph-mgr-k8sevents_19.0.0-2499-ge0e48ba6-1jammy_all.deb ... 2024-03-28T14:05:17.616 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking ceph-mgr-k8sevents (19.0.0-2499-ge0e48ba6-1jammy) ... 2024-03-28T14:05:17.908 INFO:teuthology.orchestra.run.smithi171.stdout:Selecting previously unselected package rbd-fuse. 2024-03-28T14:05:17.924 INFO:teuthology.orchestra.run.smithi171.stdout:Preparing to unpack .../123-rbd-fuse_19.0.0-2499-ge0e48ba6-1jammy_amd64.deb ... 2024-03-28T14:05:17.927 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package libonig5:amd64. 2024-03-28T14:05:17.955 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../094-libonig5_6.9.7.1-2build1_amd64.deb ... 2024-03-28T14:05:17.966 INFO:teuthology.orchestra.run.smithi171.stdout:Unpacking rbd-fuse (19.0.0-2499-ge0e48ba6-1jammy) ... 2024-03-28T14:05:17.993 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking libonig5:amd64 (6.9.7.1-2build1) ... 2024-03-28T14:05:18.287 INFO:teuthology.orchestra.run.smithi171.stdout:Setting up python3-iniconfig (1.1.1-2) ... 2024-03-28T14:05:18.313 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package libjq1:amd64. 2024-03-28T14:05:18.341 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../095-libjq1_1.6-2.1ubuntu3_amd64.deb ... 2024-03-28T14:05:18.380 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking libjq1:amd64 (1.6-2.1ubuntu3) ... 2024-03-28T14:05:18.546 INFO:teuthology.orchestra.run.smithi171.stdout:Setting up libdouble-conversion3:amd64 (3.1.7-4) ... 2024-03-28T14:05:18.665 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package jq. 2024-03-28T14:05:18.672 INFO:teuthology.orchestra.run.smithi171.stdout:Setting up nvme-cli (1.16-3ubuntu0.1) ... 2024-03-28T14:05:18.682 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../096-jq_1.6-2.1ubuntu3_amd64.deb ... 2024-03-28T14:05:18.724 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking jq (1.6-2.1ubuntu3) ... 2024-03-28T14:05:18.936 INFO:teuthology.orchestra.run.smithi171.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmefc-boot-connections.service -> /lib/systemd/system/nvmefc-boot-connections.service. 2024-03-28T14:05:18.951 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package socat. 2024-03-28T14:05:18.967 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../097-socat_1.7.4.1-3ubuntu4_amd64.deb ... 2024-03-28T14:05:19.001 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking socat (1.7.4.1-3ubuntu4) ... 2024-03-28T14:05:19.276 INFO:teuthology.orchestra.run.smithi171.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmf-autoconnect.service -> /lib/systemd/system/nvmf-autoconnect.service. 2024-03-28T14:05:19.280 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package xmlstarlet. 2024-03-28T14:05:19.307 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../098-xmlstarlet_1.6.1-2.1_amd64.deb ... 2024-03-28T14:05:19.346 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking xmlstarlet (1.6.1-2.1) ... 2024-03-28T14:05:19.692 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package ceph-test. 2024-03-28T14:05:19.719 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../099-ceph-test_19.0.0-2499-ge0e48ba6-1jammy_amd64.deb ... 2024-03-28T14:05:19.742 INFO:teuthology.orchestra.run.smithi171.stdout:nvmf-connect.target is a disabled or a static unit, not starting it. 2024-03-28T14:05:19.758 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking ceph-test (19.0.0-2499-ge0e48ba6-1jammy) ... 2024-03-28T14:05:19.991 INFO:teuthology.orchestra.run.smithi171.stdout:Setting up cephadm (19.0.0-2499-ge0e48ba6-1jammy) ... 2024-03-28T14:05:20.474 INFO:teuthology.orchestra.run.smithi171.stdout:Adding system user cephadm....done 2024-03-28T14:05:20.630 INFO:teuthology.orchestra.run.smithi171.stdout:Setting up python3-waitress (1.4.4-1.1ubuntu1) ... 2024-03-28T14:05:20.906 INFO:teuthology.orchestra.run.smithi171.stdout:Setting up python3-jaraco.classes (3.2.1-3) ... 2024-03-28T14:05:21.141 INFO:teuthology.orchestra.run.smithi171.stdout:Setting up python-asyncssh-doc (2.5.0-1) ... 2024-03-28T14:05:21.250 INFO:teuthology.orchestra.run.smithi171.stdout:Setting up python3-jaraco.functools (3.4.0-2) ... 2024-03-28T14:05:21.501 INFO:teuthology.orchestra.run.smithi171.stdout:Setting up python3-repoze.lru (0.7-2) ... 2024-03-28T14:05:21.761 INFO:teuthology.orchestra.run.smithi171.stdout:Setting up liboath0:amd64 (2.6.7-3build1) ... 2024-03-28T14:05:21.887 INFO:teuthology.orchestra.run.smithi171.stdout:Setting up python3-py (1.10.0-1) ... 2024-03-28T14:05:21.998 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package ceph-volume. 2024-03-28T14:05:22.023 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../100-ceph-volume_19.0.0-2499-ge0e48ba6-1jammy_all.deb ... 2024-03-28T14:05:22.082 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking ceph-volume (19.0.0-2499-ge0e48ba6-1jammy) ... 2024-03-28T14:05:22.197 INFO:teuthology.orchestra.run.smithi171.stdout:Setting up python3-joblib (0.17.0-4ubuntu1) ... 2024-03-28T14:05:22.458 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package cephfs-mirror. 2024-03-28T14:05:22.485 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../101-cephfs-mirror_19.0.0-2499-ge0e48ba6-1jammy_amd64.deb ... 2024-03-28T14:05:22.524 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking cephfs-mirror (19.0.0-2499-ge0e48ba6-1jammy) ... 2024-03-28T14:05:22.565 INFO:teuthology.orchestra.run.smithi171.stdout:Setting up python3-cachetools (5.0.0-1) ... 2024-03-28T14:05:22.802 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package python3-colorama. 2024-03-28T14:05:22.808 INFO:teuthology.orchestra.run.smithi171.stdout:Setting up python3-colorama (0.4.4-1) ... 2024-03-28T14:05:22.829 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../102-python3-colorama_0.4.4-1_all.deb ... 2024-03-28T14:05:22.860 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking python3-colorama (0.4.4-1) ... 2024-03-28T14:05:23.051 INFO:teuthology.orchestra.run.smithi171.stdout:Setting up python3-openssl (21.0.0-1) ... 2024-03-28T14:05:23.105 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package cephfs-shell. 2024-03-28T14:05:23.132 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../103-cephfs-shell_19.0.0-2499-ge0e48ba6-1jammy_all.deb ... 2024-03-28T14:05:23.171 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking cephfs-shell (19.0.0-2499-ge0e48ba6-1jammy) ... 2024-03-28T14:05:23.335 INFO:teuthology.orchestra.run.smithi171.stdout:Setting up unzip (6.0-26ubuntu3.2) ... 2024-03-28T14:05:23.432 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package cephfs-top. 2024-03-28T14:05:23.448 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../104-cephfs-top_19.0.0-2499-ge0e48ba6-1jammy_all.deb ... 2024-03-28T14:05:23.490 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking cephfs-top (19.0.0-2499-ge0e48ba6-1jammy) ... 2024-03-28T14:05:23.595 INFO:teuthology.orchestra.run.smithi171.stdout:Setting up python3-bcrypt (3.2.0-1build1) ... 2024-03-28T14:05:23.743 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package libcephfs-dev. 2024-03-28T14:05:23.770 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../105-libcephfs-dev_19.0.0-2499-ge0e48ba6-1jammy_amd64.deb ... 2024-03-28T14:05:23.809 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking libcephfs-dev (19.0.0-2499-ge0e48ba6-1jammy) ... 2024-03-28T14:05:23.863 INFO:teuthology.orchestra.run.smithi171.stdout:Setting up python3-pyinotify (0.9.6-1.3) ... 2024-03-28T14:05:24.112 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package lua-socket:amd64. 2024-03-28T14:05:24.139 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../106-lua-socket_3.0~rc1+git+ac3201d-6_amd64.deb ... 2024-03-28T14:05:24.173 INFO:teuthology.orchestra.run.smithi171.stdout:Setting up python3-threadpoolctl (3.1.0-1) ... 2024-03-28T14:05:24.178 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking lua-socket:amd64 (3.0~rc1+git+ac3201d-6) ... 2024-03-28T14:05:24.432 INFO:teuthology.orchestra.run.smithi171.stdout:Setting up python3-ceph-argparse (19.0.0-2499-ge0e48ba6-1jammy) ... 2024-03-28T14:05:24.466 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package lua-sec:amd64. 2024-03-28T14:05:24.493 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../107-lua-sec_1.0.2-1_amd64.deb ... 2024-03-28T14:05:24.524 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking lua-sec:amd64 (1.0.2-1) ... 2024-03-28T14:05:24.666 INFO:teuthology.orchestra.run.smithi171.stdout:Setting up python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2024-03-28T14:05:24.767 INFO:teuthology.orchestra.run.smithi171.stdout:Setting up lua-socket:amd64 (3.0~rc1+git+ac3201d-6) ... 2024-03-28T14:05:24.885 INFO:teuthology.orchestra.run.smithi171.stdout:Setting up python3-pyperclip (1.8.2-2) ... 2024-03-28T14:05:24.895 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package nvme-cli. 2024-03-28T14:05:24.922 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../108-nvme-cli_1.16-3ubuntu0.1_amd64.deb ... 2024-03-28T14:05:24.961 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking nvme-cli (1.16-3ubuntu0.1) ... 2024-03-28T14:05:25.136 INFO:teuthology.orchestra.run.smithi171.stdout:Setting up libreadline-dev:amd64 (8.1.2-1) ... 2024-03-28T14:05:25.262 INFO:teuthology.orchestra.run.smithi171.stdout:Setting up python3-markupsafe (2.0.1-2build1) ... 2024-03-28T14:05:25.375 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package pkg-config. 2024-03-28T14:05:25.402 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../109-pkg-config_0.29.2-1ubuntu3_amd64.deb ... 2024-03-28T14:05:25.441 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking pkg-config (0.29.2-1ubuntu3) ... 2024-03-28T14:05:25.521 INFO:teuthology.orchestra.run.smithi171.stdout:Setting up lua5.1 (5.1.5-8.1build4) ... 2024-03-28T14:05:25.719 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package python-asyncssh-doc. 2024-03-28T14:05:25.746 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../110-python-asyncssh-doc_2.5.0-1_all.deb ... 2024-03-28T14:05:25.785 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking python-asyncssh-doc (2.5.0-1) ... 2024-03-28T14:05:25.889 INFO:teuthology.orchestra.run.smithi171.stdout:update-alternatives: using /usr/bin/lua5.1 to provide /usr/bin/lua (lua-interpreter) in auto mode 2024-03-28T14:05:25.910 INFO:teuthology.orchestra.run.smithi171.stdout:update-alternatives: using /usr/bin/luac5.1 to provide /usr/bin/luac (lua-compiler) in auto mode 2024-03-28T14:05:25.975 INFO:teuthology.orchestra.run.smithi171.stdout:Setting up libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2024-03-28T14:05:26.092 INFO:teuthology.orchestra.run.smithi171.stdout:Setting up python3-psutil (5.9.0-1build1) ... 2024-03-28T14:05:26.230 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package python3-iniconfig. 2024-03-28T14:05:26.258 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../111-python3-iniconfig_1.1.1-2_all.deb ... 2024-03-28T14:05:26.297 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking python3-iniconfig (1.1.1-2) ... 2024-03-28T14:05:26.443 INFO:teuthology.orchestra.run.smithi171.stdout:Setting up python-babel-localedata (2.8.0+dfsg.1-7) ... 2024-03-28T14:05:26.558 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package python3-pastescript. 2024-03-28T14:05:26.574 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../112-python3-pastescript_2.0.2-4_all.deb ... 2024-03-28T14:05:26.603 INFO:teuthology.orchestra.run.smithi171.stdout:Setting up python3-natsort (8.0.2-1) ... 2024-03-28T14:05:26.607 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking python3-pastescript (2.0.2-4) ... 2024-03-28T14:05:26.879 INFO:teuthology.orchestra.run.smithi171.stdout:Setting up python3-routes (2.5.1-1ubuntu1) ... 2024-03-28T14:05:26.911 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package python3-pluggy. 2024-03-28T14:05:26.938 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../113-python3-pluggy_0.13.0-7.1_all.deb ... 2024-03-28T14:05:26.977 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking python3-pluggy (0.13.0-7.1) ... 2024-03-28T14:05:27.289 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package python3-psutil. 2024-03-28T14:05:27.316 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../114-python3-psutil_5.9.0-1build1_amd64.deb ... 2024-03-28T14:05:27.321 INFO:teuthology.orchestra.run.smithi171.stdout:Setting up python3-simplejson (3.17.6-1build1) ... 2024-03-28T14:05:27.355 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking python3-psutil (5.9.0-1build1) ... 2024-03-28T14:05:27.656 INFO:teuthology.orchestra.run.smithi171.stdout:Setting up zip (3.0-12build2) ... 2024-03-28T14:05:27.683 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package python3-py. 2024-03-28T14:05:27.713 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../115-python3-py_1.10.0-1_all.deb ... 2024-03-28T14:05:27.749 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking python3-py (1.10.0-1) ... 2024-03-28T14:05:27.774 INFO:teuthology.orchestra.run.smithi171.stdout:Setting up python3-pygments (2.11.2+dfsg-2) ... 2024-03-28T14:05:28.053 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package python3-pygments. 2024-03-28T14:05:28.080 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../116-python3-pygments_2.11.2+dfsg-2_all.deb ... 2024-03-28T14:05:28.119 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking python3-pygments (2.11.2+dfsg-2) ... 2024-03-28T14:05:28.459 INFO:teuthology.orchestra.run.smithi171.stdout:Setting up python3-tempita (0.5.2-6ubuntu1) ... 2024-03-28T14:05:28.542 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package python3-pyinotify. 2024-03-28T14:05:28.570 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../117-python3-pyinotify_0.9.6-1.3_all.deb ... 2024-03-28T14:05:28.600 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking python3-pyinotify (0.9.6-1.3) ... 2024-03-28T14:05:28.693 INFO:teuthology.orchestra.run.smithi171.stdout:Setting up python-pastedeploy-tpl (2.1.1-1) ... 2024-03-28T14:05:28.819 INFO:teuthology.orchestra.run.smithi171.stdout:Setting up qttranslations5-l10n (5.15.3-1) ... 2024-03-28T14:05:28.902 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package python3-toml. 2024-03-28T14:05:28.930 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../118-python3-toml_0.10.2-1_all.deb ... 2024-03-28T14:05:28.945 INFO:teuthology.orchestra.run.smithi171.stdout:Setting up python3-wcwidth (0.2.5+dfsg1-1) ... 2024-03-28T14:05:28.969 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking python3-toml (0.10.2-1) ... 2024-03-28T14:05:29.238 INFO:teuthology.orchestra.run.smithi171.stdout:Setting up python3-asyncssh (2.5.0-1) ... 2024-03-28T14:05:29.272 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package python3-pytest. 2024-03-28T14:05:29.300 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../119-python3-pytest_6.2.5-1ubuntu2_all.deb ... 2024-03-28T14:05:29.338 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking python3-pytest (6.2.5-1ubuntu2) ... 2024-03-28T14:05:29.631 INFO:teuthology.orchestra.run.smithi171.stdout:Setting up python3-certifi (2020.6.20-1) ... 2024-03-28T14:05:29.658 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package python3-simplejson. 2024-03-28T14:05:29.686 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../120-python3-simplejson_3.17.6-1build1_amd64.deb ... 2024-03-28T14:05:29.724 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking python3-simplejson (3.17.6-1build1) ... 2024-03-28T14:05:29.866 INFO:teuthology.orchestra.run.smithi171.stdout:Setting up python3-paste (3.5.0+dfsg1-1) ... 2024-03-28T14:05:29.993 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package qttranslations5-l10n. 2024-03-28T14:05:30.009 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../121-qttranslations5-l10n_5.15.3-1_all.deb ... 2024-03-28T14:05:30.052 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking qttranslations5-l10n (5.15.3-1) ... 2024-03-28T14:05:30.267 INFO:teuthology.orchestra.run.smithi171.stdout:Setting up python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2024-03-28T14:05:30.543 INFO:teuthology.orchestra.run.smithi171.stdout:Setting up python3-cmd2 (2.3.3-1) ... 2024-03-28T14:05:30.805 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package radosgw. 2024-03-28T14:05:30.833 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../122-radosgw_19.0.0-2499-ge0e48ba6-1jammy_amd64.deb ... 2024-03-28T14:05:30.869 INFO:teuthology.orchestra.run.smithi171.stdout:Setting up python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.1) ... 2024-03-28T14:05:30.880 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking radosgw (19.0.0-2499-ge0e48ba6-1jammy) ... 2024-03-28T14:05:31.204 INFO:teuthology.orchestra.run.smithi171.stdout:Setting up python3-jaraco.text (3.6.0-2) ... 2024-03-28T14:05:31.539 INFO:teuthology.orchestra.run.smithi171.stdout:Setting up socat (1.7.4.1-3ubuntu4) ... 2024-03-28T14:05:31.665 INFO:teuthology.orchestra.run.smithi171.stdout:Setting up python3-ceph-common (19.0.0-2499-ge0e48ba6-1jammy) ... 2024-03-28T14:05:31.850 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package rbd-fuse. 2024-03-28T14:05:31.866 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../123-rbd-fuse_19.0.0-2499-ge0e48ba6-1jammy_amd64.deb ... 2024-03-28T14:05:31.899 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking rbd-fuse (19.0.0-2499-ge0e48ba6-1jammy) ... 2024-03-28T14:05:31.958 INFO:teuthology.orchestra.run.smithi171.stdout:Setting up python3-idna (3.3-1) ... 2024-03-28T14:05:32.170 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up python3-iniconfig (1.1.1-2) ... 2024-03-28T14:05:32.251 INFO:teuthology.orchestra.run.smithi171.stdout:Setting up python3-sklearn (0.23.2-5ubuntu6) ... 2024-03-28T14:05:32.421 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up libdouble-conversion3:amd64 (3.1.7-4) ... 2024-03-28T14:05:32.555 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up nvme-cli (1.16-3ubuntu0.1) ... 2024-03-28T14:05:32.816 INFO:teuthology.orchestra.run.smithi104.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmefc-boot-connections.service -> /lib/systemd/system/nvmefc-boot-connections.service. 2024-03-28T14:05:33.130 INFO:teuthology.orchestra.run.smithi104.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmf-autoconnect.service -> /lib/systemd/system/nvmf-autoconnect.service. 2024-03-28T14:05:33.431 INFO:teuthology.orchestra.run.smithi171.stdout:Setting up pkg-config (0.29.2-1ubuntu3) ... 2024-03-28T14:05:33.575 INFO:teuthology.orchestra.run.smithi104.stdout:nvmf-connect.target is a disabled or a static unit, not starting it. 2024-03-28T14:05:33.657 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up cephadm (19.0.0-2499-ge0e48ba6-1jammy) ... 2024-03-28T14:05:33.700 INFO:teuthology.orchestra.run.smithi171.stdout:Setting up libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-03-28T14:05:33.834 INFO:teuthology.orchestra.run.smithi171.stdout:Setting up python3-toml (0.10.2-1) ... 2024-03-28T14:05:34.093 INFO:teuthology.orchestra.run.smithi171.stdout:Setting up librdkafka1:amd64 (1.8.0-1build1) ... 2024-03-28T14:05:34.219 INFO:teuthology.orchestra.run.smithi171.stdout:Setting up xmlstarlet (1.6.1-2.1) ... 2024-03-28T14:05:34.345 INFO:teuthology.orchestra.run.smithi171.stdout:Setting up python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ... 2024-03-28T14:05:34.466 INFO:teuthology.orchestra.run.smithi104.stdout:Adding system user cephadm....done 2024-03-28T14:05:34.630 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up python3-waitress (1.4.4-1.1ubuntu1) ... 2024-03-28T14:05:34.647 INFO:teuthology.orchestra.run.smithi171.stdout:Setting up python3-pluggy (0.13.0-7.1) ... 2024-03-28T14:05:34.898 INFO:teuthology.orchestra.run.smithi171.stdout:Setting up python3-zc.lockfile (2.0-1) ... 2024-03-28T14:05:34.906 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up python3-jaraco.classes (3.2.1-3) ... 2024-03-28T14:05:35.124 INFO:teuthology.orchestra.run.smithi171.stdout:Setting up libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-03-28T14:05:35.158 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up python-asyncssh-doc (2.5.0-1) ... 2024-03-28T14:05:35.250 INFO:teuthology.orchestra.run.smithi171.stdout:Setting up python3-pyasn1 (0.4.8-1) ... 2024-03-28T14:05:35.275 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up python3-jaraco.functools (3.4.0-2) ... 2024-03-28T14:05:35.534 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up python3-repoze.lru (0.7-2) ... 2024-03-28T14:05:35.535 INFO:teuthology.orchestra.run.smithi171.stdout:Setting up python3-singledispatch (3.4.0.3-3) ... 2024-03-28T14:05:35.769 INFO:teuthology.orchestra.run.smithi171.stdout:Setting up python3-logutils (0.3.3-8) ... 2024-03-28T14:05:35.785 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up liboath0:amd64 (2.6.7-3build1) ... 2024-03-28T14:05:35.912 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up python3-py (1.10.0-1) ... 2024-03-28T14:05:36.045 INFO:teuthology.orchestra.run.smithi171.stdout:Setting up python3-tempora (4.1.2-1) ... 2024-03-28T14:05:36.214 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up python3-joblib (0.17.0-4ubuntu1) ... 2024-03-28T14:05:36.306 INFO:teuthology.orchestra.run.smithi171.stdout:Setting up python3-simplegeneric (0.8.1-3) ... 2024-03-28T14:05:36.564 INFO:teuthology.orchestra.run.smithi171.stdout:Setting up python3-prettytable (2.5.0-2) ... 2024-03-28T14:05:36.599 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up python3-cachetools (5.0.0-1) ... 2024-03-28T14:05:36.815 INFO:teuthology.orchestra.run.smithi171.stdout:Setting up liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2024-03-28T14:05:36.841 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up python3-colorama (0.4.4-1) ... 2024-03-28T14:05:36.917 INFO:teuthology.orchestra.run.smithi171.stdout:Setting up python3-websocket (1.2.3-1) ... 2024-03-28T14:05:37.092 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up python3-openssl (21.0.0-1) ... 2024-03-28T14:05:37.193 INFO:teuthology.orchestra.run.smithi171.stdout:Setting up libonig5:amd64 (6.9.7.1-2build1) ... 2024-03-28T14:05:37.294 INFO:teuthology.orchestra.run.smithi171.stdout:Setting up python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2024-03-28T14:05:37.360 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up unzip (6.0-26ubuntu3.2) ... 2024-03-28T14:05:37.570 INFO:teuthology.orchestra.run.smithi171.stdout:Setting up python3-webob (1:1.8.6-1.1) ... 2024-03-28T14:05:37.628 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up python3-bcrypt (3.2.0-1build1) ... 2024-03-28T14:05:37.854 INFO:teuthology.orchestra.run.smithi171.stdout:Setting up python3-jaraco.collections (3.4.0-2) ... 2024-03-28T14:05:37.888 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up python3-pyinotify (0.9.6-1.3) ... 2024-03-28T14:05:38.114 INFO:teuthology.orchestra.run.smithi171.stdout:Setting up liblua5.3-dev:amd64 (5.3.6-1build1) ... 2024-03-28T14:05:38.240 INFO:teuthology.orchestra.run.smithi171.stdout:Setting up lua-sec:amd64 (1.0.2-1) ... 2024-03-28T14:05:38.264 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up python3-threadpoolctl (3.1.0-1) ... 2024-03-28T14:05:38.341 INFO:teuthology.orchestra.run.smithi171.stdout:Setting up python3-babel (2.8.0+dfsg.1-7) ... 2024-03-28T14:05:38.415 INFO:teuthology.orchestra.run.smithi171.stdout:update-alternatives: using /usr/bin/pybabel-python3 to provide /usr/bin/pybabel (pybabel) in auto mode 2024-03-28T14:05:38.515 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up python3-ceph-argparse (19.0.0-2499-ge0e48ba6-1jammy) ... 2024-03-28T14:05:38.668 INFO:teuthology.orchestra.run.smithi171.stdout:Setting up libjq1:amd64 (1.6-2.1ubuntu3) ... 2024-03-28T14:05:38.766 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2024-03-28T14:05:38.768 INFO:teuthology.orchestra.run.smithi171.stdout:Setting up python3-pytest (6.2.5-1ubuntu2) ... 2024-03-28T14:05:38.883 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up lua-socket:amd64 (3.0~rc1+git+ac3201d-6) ... 2024-03-28T14:05:39.009 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up python3-pyperclip (1.8.2-2) ... 2024-03-28T14:05:39.137 INFO:teuthology.orchestra.run.smithi171.stdout:Setting up python3-pastedeploy (2.1.1-1) ... 2024-03-28T14:05:39.277 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up libreadline-dev:amd64 (8.1.2-1) ... 2024-03-28T14:05:39.378 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up python3-markupsafe (2.0.1-2build1) ... 2024-03-28T14:05:39.398 INFO:teuthology.orchestra.run.smithi171.stdout:Setting up lua-any (27ubuntu1) ... 2024-03-28T14:05:39.556 INFO:teuthology.orchestra.run.smithi171.stdout:Setting up python3-portend (3.0.0-1) ... 2024-03-28T14:05:39.629 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up lua5.1 (5.1.5-8.1build4) ... 2024-03-28T14:05:39.815 INFO:teuthology.orchestra.run.smithi171.stdout:Setting up libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-03-28T14:05:39.941 INFO:teuthology.orchestra.run.smithi171.stdout:Setting up python3-jinja2 (3.0.3-1ubuntu0.1) ... 2024-03-28T14:05:40.078 INFO:teuthology.orchestra.run.smithi104.stdout:update-alternatives: using /usr/bin/lua5.1 to provide /usr/bin/lua (lua-interpreter) in auto mode 2024-03-28T14:05:40.127 INFO:teuthology.orchestra.run.smithi104.stdout:update-alternatives: using /usr/bin/luac5.1 to provide /usr/bin/luac (lua-compiler) in auto mode 2024-03-28T14:05:40.190 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2024-03-28T14:05:40.259 INFO:teuthology.orchestra.run.smithi171.stdout:Setting up python3-pyasn1-modules (0.2.1-1) ... 2024-03-28T14:05:40.316 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up python3-psutil (5.9.0-1build1) ... 2024-03-28T14:05:40.552 INFO:teuthology.orchestra.run.smithi171.stdout:Setting up python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2024-03-28T14:05:40.684 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up python-babel-localedata (2.8.0+dfsg.1-7) ... 2024-03-28T14:05:40.810 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up python3-natsort (8.0.2-1) ... 2024-03-28T14:05:40.811 INFO:teuthology.orchestra.run.smithi171.stdout:Setting up jq (1.6-2.1ubuntu3) ... 2024-03-28T14:05:41.023 INFO:teuthology.orchestra.run.smithi171.stdout:Setting up python3-rsa (4.8-1) ... 2024-03-28T14:05:41.077 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up python3-routes (2.5.1-1ubuntu1) ... 2024-03-28T14:05:41.274 INFO:teuthology.orchestra.run.smithi171.stdout:Setting up python3-webtest (2.0.35-1) ... 2024-03-28T14:05:41.328 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up python3-simplejson (3.17.6-1build1) ... 2024-03-28T14:05:41.558 INFO:teuthology.orchestra.run.smithi171.stdout:Setting up python3-requests-oauthlib (1.3.0+ds-0.1) ... 2024-03-28T14:05:41.638 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up zip (3.0-12build2) ... 2024-03-28T14:05:41.764 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up python3-pygments (2.11.2+dfsg-2) ... 2024-03-28T14:05:41.810 INFO:teuthology.orchestra.run.smithi171.stdout:Setting up python3-cherrypy3 (18.6.1-4) ... 2024-03-28T14:05:42.195 INFO:teuthology.orchestra.run.smithi171.stdout:Setting up python3-pastescript (2.0.2-4) ... 2024-03-28T14:05:42.424 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up python3-tempita (0.5.2-6ubuntu1) ... 2024-03-28T14:05:42.479 INFO:teuthology.orchestra.run.smithi171.stdout:Setting up python3-pecan (1.3.3-4ubuntu2) ... 2024-03-28T14:05:42.683 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up python-pastedeploy-tpl (2.1.1-1) ... 2024-03-28T14:05:42.792 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up qttranslations5-l10n (5.15.3-1) ... 2024-03-28T14:05:42.832 INFO:teuthology.orchestra.run.smithi171.stdout:Setting up libthrift-0.16.0:amd64 (0.16.0-2) ... 2024-03-28T14:05:42.902 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up python3-wcwidth (0.2.5+dfsg1-1) ... 2024-03-28T14:05:42.958 INFO:teuthology.orchestra.run.smithi171.stdout:Setting up librados2 (19.0.0-2499-ge0e48ba6-1jammy) ... 2024-03-28T14:05:43.086 INFO:teuthology.orchestra.run.smithi171.stdout:Setting up libsqlite3-mod-ceph (19.0.0-2499-ge0e48ba6-1jammy) ... 2024-03-28T14:05:43.186 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up python3-asyncssh (2.5.0-1) ... 2024-03-28T14:05:43.193 INFO:teuthology.orchestra.run.smithi171.stdout:Setting up luarocks (3.8.0+dfsg1-1) ... 2024-03-28T14:05:43.462 INFO:teuthology.orchestra.run.smithi171.stdout:Setting up libcephfs2 (19.0.0-2499-ge0e48ba6-1jammy) ... 2024-03-28T14:05:43.589 INFO:teuthology.orchestra.run.smithi171.stdout:Setting up libradosstriper1 (19.0.0-2499-ge0e48ba6-1jammy) ... 2024-03-28T14:05:43.595 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up python3-certifi (2020.6.20-1) ... 2024-03-28T14:05:43.714 INFO:teuthology.orchestra.run.smithi171.stdout:Setting up python3-google-auth (1.5.1-3) ... 2024-03-28T14:05:43.846 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up python3-paste (3.5.0+dfsg1-1) ... 2024-03-28T14:05:43.999 INFO:teuthology.orchestra.run.smithi171.stdout:Setting up librbd1 (19.0.0-2499-ge0e48ba6-1jammy) ... 2024-03-28T14:05:44.125 INFO:teuthology.orchestra.run.smithi171.stdout:Setting up ceph-mgr-modules-core (19.0.0-2499-ge0e48ba6-1jammy) ... 2024-03-28T14:05:44.240 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2024-03-28T14:05:44.251 INFO:teuthology.orchestra.run.smithi171.stdout:Setting up ceph-fuse (19.0.0-2499-ge0e48ba6-1jammy) ... 2024-03-28T14:05:44.450 INFO:teuthology.orchestra.run.smithi171.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-03-28T14:05:44.450 INFO:teuthology.orchestra.run.smithi171.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-03-28T14:05:44.524 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up python3-cmd2 (2.3.3-1) ... 2024-03-28T14:05:44.825 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.1) ... 2024-03-28T14:05:44.977 INFO:teuthology.orchestra.run.smithi171.stdout:Setting up libcephfs-dev (19.0.0-2499-ge0e48ba6-1jammy) ... 2024-03-28T14:05:45.103 INFO:teuthology.orchestra.run.smithi171.stdout:Setting up python3-rados (19.0.0-2499-ge0e48ba6-1jammy) ... 2024-03-28T14:05:45.143 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up python3-jaraco.text (3.6.0-2) ... 2024-03-28T14:05:45.229 INFO:teuthology.orchestra.run.smithi171.stdout:Setting up librgw2 (19.0.0-2499-ge0e48ba6-1jammy) ... 2024-03-28T14:05:45.372 INFO:teuthology.orchestra.run.smithi171.stdout:Setting up cephfs-top (19.0.0-2499-ge0e48ba6-1jammy) ... 2024-03-28T14:05:45.386 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up socat (1.7.4.1-3ubuntu4) ... 2024-03-28T14:05:45.481 INFO:teuthology.orchestra.run.smithi171.stdout:Setting up python3-kubernetes (12.0.1-1ubuntu1) ... 2024-03-28T14:05:45.511 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up python3-ceph-common (19.0.0-2499-ge0e48ba6-1jammy) ... 2024-03-28T14:05:45.822 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up python3-idna (3.3-1) ... 2024-03-28T14:05:46.130 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up python3-sklearn (0.23.2-5ubuntu6) ... 2024-03-28T14:05:46.677 INFO:teuthology.orchestra.run.smithi171.stdout:Setting up python3-rbd (19.0.0-2499-ge0e48ba6-1jammy) ... 2024-03-28T14:05:46.778 INFO:teuthology.orchestra.run.smithi171.stdout:Setting up rbd-fuse (19.0.0-2499-ge0e48ba6-1jammy) ... 2024-03-28T14:05:46.879 INFO:teuthology.orchestra.run.smithi171.stdout:Setting up python3-rgw (19.0.0-2499-ge0e48ba6-1jammy) ... 2024-03-28T14:05:47.005 INFO:teuthology.orchestra.run.smithi171.stdout:Setting up python3-cephfs (19.0.0-2499-ge0e48ba6-1jammy) ... 2024-03-28T14:05:47.131 INFO:teuthology.orchestra.run.smithi171.stdout:Setting up cephfs-shell (19.0.0-2499-ge0e48ba6-1jammy) ... 2024-03-28T14:05:47.257 INFO:teuthology.orchestra.run.smithi171.stdout:Setting up ceph-common (19.0.0-2499-ge0e48ba6-1jammy) ... 2024-03-28T14:05:47.511 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up pkg-config (0.29.2-1ubuntu3) ... 2024-03-28T14:05:47.783 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-03-28T14:05:47.792 INFO:teuthology.orchestra.run.smithi171.stdout:Adding group ceph....done 2024-03-28T14:05:47.914 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up python3-toml (0.10.2-1) ... 2024-03-28T14:05:48.169 INFO:teuthology.orchestra.run.smithi171.stdout:Adding system user ceph....done 2024-03-28T14:05:48.173 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up librdkafka1:amd64 (1.8.0-1build1) ... 2024-03-28T14:05:48.299 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up xmlstarlet (1.6.1-2.1) ... 2024-03-28T14:05:48.303 INFO:teuthology.orchestra.run.smithi171.stdout:Setting system user ceph properties....done 2024-03-28T14:05:48.317 INFO:teuthology.orchestra.run.smithi171.stdout:chown: cannot access '/var/log/ceph/*.log*': No such file or directory 2024-03-28T14:05:48.425 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ... 2024-03-28T14:05:48.452 INFO:teuthology.orchestra.run.smithi171.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /lib/systemd/system/ceph.target. 2024-03-28T14:05:48.709 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up python3-pluggy (0.13.0-7.1) ... 2024-03-28T14:05:48.758 INFO:teuthology.orchestra.run.smithi171.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service -> /lib/systemd/system/rbdmap.service. 2024-03-28T14:05:48.961 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up python3-zc.lockfile (2.0-1) ... 2024-03-28T14:05:49.211 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-03-28T14:05:49.266 INFO:teuthology.orchestra.run.smithi171.stdout:Setting up cephfs-mirror (19.0.0-2499-ge0e48ba6-1jammy) ... 2024-03-28T14:05:49.346 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up python3-pyasn1 (0.4.8-1) ... 2024-03-28T14:05:49.479 INFO:teuthology.orchestra.run.smithi171.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target -> /lib/systemd/system/cephfs-mirror.target. 2024-03-28T14:05:49.479 INFO:teuthology.orchestra.run.smithi171.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target -> /lib/systemd/system/cephfs-mirror.target. 2024-03-28T14:05:49.647 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up python3-singledispatch (3.4.0.3-3) ... 2024-03-28T14:05:49.890 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up python3-logutils (0.3.3-8) ... 2024-03-28T14:05:49.985 INFO:teuthology.orchestra.run.smithi171.stdout:Setting up ceph-test (19.0.0-2499-ge0e48ba6-1jammy) ... 2024-03-28T14:05:50.102 INFO:teuthology.orchestra.run.smithi171.stdout:Setting up radosgw (19.0.0-2499-ge0e48ba6-1jammy) ... 2024-03-28T14:05:50.149 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up python3-tempora (4.1.2-1) ... 2024-03-28T14:05:50.400 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up python3-simplegeneric (0.8.1-3) ... 2024-03-28T14:05:50.578 INFO:teuthology.orchestra.run.smithi171.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-03-28T14:05:50.578 INFO:teuthology.orchestra.run.smithi171.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-03-28T14:05:50.642 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up python3-prettytable (2.5.0-2) ... 2024-03-28T14:05:50.885 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2024-03-28T14:05:51.011 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up python3-websocket (1.2.3-1) ... 2024-03-28T14:05:51.095 INFO:teuthology.orchestra.run.smithi171.stdout:Setting up ceph-base (19.0.0-2499-ge0e48ba6-1jammy) ... 2024-03-28T14:05:51.321 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up libonig5:amd64 (6.9.7.1-2build1) ... 2024-03-28T14:05:51.447 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2024-03-28T14:05:51.459 INFO:teuthology.orchestra.run.smithi171.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service -> /lib/systemd/system/ceph-crash.service. 2024-03-28T14:05:51.739 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up python3-webob (1:1.8.6-1.1) ... 2024-03-28T14:05:51.956 INFO:teuthology.orchestra.run.smithi171.stdout:Setting up ceph-mds (19.0.0-2499-ge0e48ba6-1jammy) ... 2024-03-28T14:05:52.040 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up python3-jaraco.collections (3.4.0-2) ... 2024-03-28T14:05:52.178 INFO:teuthology.orchestra.run.smithi171.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-03-28T14:05:52.179 INFO:teuthology.orchestra.run.smithi171.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-03-28T14:05:52.291 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up liblua5.3-dev:amd64 (5.3.6-1build1) ... 2024-03-28T14:05:52.400 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up lua-sec:amd64 (1.0.2-1) ... 2024-03-28T14:05:52.510 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up python3-babel (2.8.0+dfsg.1-7) ... 2024-03-28T14:05:52.583 INFO:teuthology.orchestra.run.smithi104.stdout:update-alternatives: using /usr/bin/pybabel-python3 to provide /usr/bin/pybabel (pybabel) in auto mode 2024-03-28T14:05:52.666 INFO:teuthology.orchestra.run.smithi171.stdout:Setting up ceph-mgr (19.0.0-2499-ge0e48ba6-1jammy) ... 2024-03-28T14:05:52.853 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up libjq1:amd64 (1.6-2.1ubuntu3) ... 2024-03-28T14:05:52.886 INFO:teuthology.orchestra.run.smithi171.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-03-28T14:05:52.886 INFO:teuthology.orchestra.run.smithi171.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-03-28T14:05:52.987 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up python3-pytest (6.2.5-1ubuntu2) ... 2024-03-28T14:05:53.389 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up python3-pastedeploy (2.1.1-1) ... 2024-03-28T14:05:53.571 INFO:teuthology.orchestra.run.smithi171.stdout:Setting up ceph-osd (19.0.0-2499-ge0e48ba6-1jammy) ... 2024-03-28T14:05:53.640 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up lua-any (27ubuntu1) ... 2024-03-28T14:05:53.765 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up python3-portend (3.0.0-1) ... 2024-03-28T14:05:53.883 INFO:teuthology.orchestra.run.smithi171.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-03-28T14:05:53.883 INFO:teuthology.orchestra.run.smithi171.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-03-28T14:05:54.025 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-03-28T14:05:54.151 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up python3-jinja2 (3.0.3-1ubuntu0.1) ... 2024-03-28T14:05:54.357 INFO:teuthology.orchestra.run.smithi171.stdout:Setting up ceph-mgr-k8sevents (19.0.0-2499-ge0e48ba6-1jammy) ... 2024-03-28T14:05:54.460 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up python3-pyasn1-modules (0.2.1-1) ... 2024-03-28T14:05:54.483 INFO:teuthology.orchestra.run.smithi171.stdout:Setting up ceph-mgr-diskprediction-local (19.0.0-2499-ge0e48ba6-1jammy) ... 2024-03-28T14:05:54.617 INFO:teuthology.orchestra.run.smithi171.stdout:Setting up ceph-mon (19.0.0-2499-ge0e48ba6-1jammy) ... 2024-03-28T14:05:54.761 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2024-03-28T14:05:54.827 INFO:teuthology.orchestra.run.smithi171.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-03-28T14:05:54.827 INFO:teuthology.orchestra.run.smithi171.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-03-28T14:05:55.037 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up jq (1.6-2.1ubuntu3) ... 2024-03-28T14:05:55.248 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up python3-rsa (4.8-1) ... 2024-03-28T14:05:55.343 INFO:teuthology.orchestra.run.smithi171.stdout:Setting up ceph-mgr-cephadm (19.0.0-2499-ge0e48ba6-1jammy) ... 2024-03-28T14:05:55.516 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up python3-webtest (2.0.35-1) ... 2024-03-28T14:05:55.519 INFO:teuthology.orchestra.run.smithi171.stdout:Setting up ceph (19.0.0-2499-ge0e48ba6-1jammy) ... 2024-03-28T14:05:55.670 INFO:teuthology.orchestra.run.smithi171.stdout:Setting up ceph-mgr-dashboard (19.0.0-2499-ge0e48ba6-1jammy) ... 2024-03-28T14:05:55.776 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up python3-requests-oauthlib (1.3.0+ds-0.1) ... 2024-03-28T14:05:55.840 INFO:teuthology.orchestra.run.smithi171.stdout:Setting up ceph-volume (19.0.0-2499-ge0e48ba6-1jammy) ... 2024-03-28T14:05:56.018 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up python3-cherrypy3 (18.6.1-4) ... 2024-03-28T14:05:56.177 INFO:teuthology.orchestra.run.smithi171.stdout:Processing triggers for mailcap (3.70+nmu1ubuntu1) ... 2024-03-28T14:05:56.294 INFO:teuthology.orchestra.run.smithi171.stdout:Processing triggers for libc-bin (2.35-0ubuntu3.1) ... 2024-03-28T14:05:56.412 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up python3-pastescript (2.0.2-4) ... 2024-03-28T14:05:56.464 INFO:teuthology.orchestra.run.smithi171.stdout:Processing triggers for man-db (2.10.2-1) ... 2024-03-28T14:05:56.697 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up python3-pecan (1.3.3-4ubuntu2) ... 2024-03-28T14:05:57.032 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up libthrift-0.16.0:amd64 (0.16.0-2) ... 2024-03-28T14:05:57.150 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up librados2 (19.0.0-2499-ge0e48ba6-1jammy) ... 2024-03-28T14:05:57.251 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up libsqlite3-mod-ceph (19.0.0-2499-ge0e48ba6-1jammy) ... 2024-03-28T14:05:57.360 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up luarocks (3.8.0+dfsg1-1) ... 2024-03-28T14:05:57.634 INFO:teuthology.orchestra.run.smithi171.stdout:Processing triggers for install-info (6.8-4build1) ... 2024-03-28T14:05:57.713 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up libcephfs2 (19.0.0-2499-ge0e48ba6-1jammy) ... 2024-03-28T14:05:57.839 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up libradosstriper1 (19.0.0-2499-ge0e48ba6-1jammy) ... 2024-03-28T14:05:57.982 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up python3-google-auth (1.5.1-3) ... 2024-03-28T14:05:58.258 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up librbd1 (19.0.0-2499-ge0e48ba6-1jammy) ... 2024-03-28T14:05:58.268 INFO:teuthology.orchestra.run.smithi171.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2024-03-28T14:05:58.271 DEBUG:teuthology.orchestra.run.smithi171:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install bison flex libelf-dev libssl-dev network-manager iproute2 util-linux dump indent libaio-dev libtool-bin uuid-dev xfslibs-dev postgresql postgresql-client postgresql-common postgresql-contrib 2024-03-28T14:05:58.329 INFO:teuthology.orchestra.run.smithi171.stdout:Reading package lists... 2024-03-28T14:05:58.375 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up ceph-mgr-modules-core (19.0.0-2499-ge0e48ba6-1jammy) ... 2024-03-28T14:05:58.476 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up ceph-fuse (19.0.0-2499-ge0e48ba6-1jammy) ... 2024-03-28T14:05:58.506 INFO:teuthology.orchestra.run.smithi171.stdout:Building dependency tree... 2024-03-28T14:05:58.507 INFO:teuthology.orchestra.run.smithi171.stdout:Reading state information... 2024-03-28T14:05:58.666 INFO:teuthology.orchestra.run.smithi171.stdout:iproute2 is already the newest version (5.15.0-1ubuntu2). 2024-03-28T14:05:58.666 INFO:teuthology.orchestra.run.smithi171.stdout:libtool-bin is already the newest version (2.4.6-15build2). 2024-03-28T14:05:58.666 INFO:teuthology.orchestra.run.smithi171.stdout:xfslibs-dev is already the newest version (5.13.0-1ubuntu2). 2024-03-28T14:05:58.666 INFO:teuthology.orchestra.run.smithi171.stdout:The following packages were automatically installed and are no longer required: 2024-03-28T14:05:58.666 INFO:teuthology.orchestra.run.smithi171.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-03-28T14:05:58.666 INFO:teuthology.orchestra.run.smithi171.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-03-28T14:05:58.666 INFO:teuthology.orchestra.run.smithi171.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-03-28T14:05:58.666 INFO:teuthology.orchestra.run.smithi171.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2024-03-28T14:05:58.667 INFO:teuthology.orchestra.run.smithi171.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2024-03-28T14:05:58.667 INFO:teuthology.orchestra.run.smithi171.stdout:Use 'sudo apt autoremove' to remove them. 2024-03-28T14:05:58.667 INFO:teuthology.orchestra.run.smithi171.stdout:The following additional packages will be installed: 2024-03-28T14:05:58.667 INFO:teuthology.orchestra.run.smithi171.stdout: libbluetooth3 libfl-dev libfl2 libjson-perl libllvm14 liblzo2-2 2024-03-28T14:05:58.667 INFO:teuthology.orchestra.run.smithi171.stdout: libmbim-glib4 libmbim-proxy libmm-glib0 libndp0 libnm0 libpq5 libqmi-glib5 2024-03-28T14:05:58.668 INFO:teuthology.orchestra.run.smithi171.stdout: libqmi-proxy libssl3 libteamdctl0 libuuid1 modemmanager network-manager-pptp 2024-03-28T14:05:58.668 INFO:teuthology.orchestra.run.smithi171.stdout: postgresql-14 postgresql-client-14 postgresql-client-common ppp pptp-linux 2024-03-28T14:05:58.668 INFO:teuthology.orchestra.run.smithi171.stdout: tcl tcl8.6 usb-modeswitch usb-modeswitch-data wpasupplicant 2024-03-28T14:05:58.669 INFO:teuthology.orchestra.run.smithi171.stdout:Suggested packages: 2024-03-28T14:05:58.669 INFO:teuthology.orchestra.run.smithi171.stdout: bison-doc flex-doc indent-doc libssl-doc avahi-autoipd libteam-utils 2024-03-28T14:05:58.669 INFO:teuthology.orchestra.run.smithi171.stdout: postgresql-doc postgresql-doc-14 tcl-tclreadline comgt wvdial 2024-03-28T14:05:58.669 INFO:teuthology.orchestra.run.smithi171.stdout: util-linux-locales wpagui libengine-pkcs11-openssl 2024-03-28T14:05:58.682 INFO:teuthology.orchestra.run.smithi104.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-03-28T14:05:58.682 INFO:teuthology.orchestra.run.smithi104.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-03-28T14:05:58.707 INFO:teuthology.orchestra.run.smithi171.stdout:The following NEW packages will be installed: 2024-03-28T14:05:58.708 INFO:teuthology.orchestra.run.smithi171.stdout: bison dump flex indent libaio-dev libbluetooth3 libelf-dev libfl-dev libfl2 2024-03-28T14:05:58.708 INFO:teuthology.orchestra.run.smithi171.stdout: libjson-perl libllvm14 liblzo2-2 libmbim-glib4 libmbim-proxy libmm-glib0 2024-03-28T14:05:58.708 INFO:teuthology.orchestra.run.smithi171.stdout: libndp0 libnm0 libpq5 libqmi-glib5 libqmi-proxy libssl-dev libteamdctl0 2024-03-28T14:05:58.708 INFO:teuthology.orchestra.run.smithi171.stdout: modemmanager network-manager network-manager-pptp postgresql postgresql-14 2024-03-28T14:05:58.709 INFO:teuthology.orchestra.run.smithi171.stdout: postgresql-client postgresql-client-14 postgresql-client-common 2024-03-28T14:05:58.709 INFO:teuthology.orchestra.run.smithi171.stdout: postgresql-common postgresql-contrib ppp pptp-linux tcl tcl8.6 2024-03-28T14:05:58.709 INFO:teuthology.orchestra.run.smithi171.stdout: usb-modeswitch usb-modeswitch-data wpasupplicant 2024-03-28T14:05:58.709 INFO:teuthology.orchestra.run.smithi171.stdout:The following packages will be upgraded: 2024-03-28T14:05:58.710 INFO:teuthology.orchestra.run.smithi171.stdout: libssl3 libuuid1 util-linux uuid-dev 2024-03-28T14:05:58.889 INFO:teuthology.orchestra.run.smithi171.stdout:4 upgraded, 39 newly installed, 0 to remove and 294 not upgraded. 2024-03-28T14:05:58.889 INFO:teuthology.orchestra.run.smithi171.stdout:Need to get 55.7 MB of archives. 2024-03-28T14:05:58.889 INFO:teuthology.orchestra.run.smithi171.stdout:After this operation, 202 MB of additional disk space will be used. 2024-03-28T14:05:58.890 INFO:teuthology.orchestra.run.smithi171.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 util-linux amd64 2.37.2-4ubuntu3.3 [1,063 kB] 2024-03-28T14:05:59.152 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up libcephfs-dev (19.0.0-2499-ge0e48ba6-1jammy) ... 2024-03-28T14:05:59.287 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up python3-rados (19.0.0-2499-ge0e48ba6-1jammy) ... 2024-03-28T14:05:59.413 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up librgw2 (19.0.0-2499-ge0e48ba6-1jammy) ... 2024-03-28T14:05:59.516 INFO:teuthology.orchestra.run.smithi171.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy/main amd64 flex amd64 2.6.4-8build2 [307 kB] 2024-03-28T14:05:59.533 INFO:teuthology.orchestra.run.smithi171.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libssl3 amd64 3.0.2-0ubuntu1.15 [1,905 kB] 2024-03-28T14:05:59.538 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up cephfs-top (19.0.0-2499-ge0e48ba6-1jammy) ... 2024-03-28T14:05:59.657 INFO:teuthology.orchestra.run.smithi171.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 uuid-dev amd64 2.37.2-4ubuntu3.3 [33.1 kB] 2024-03-28T14:05:59.664 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up python3-kubernetes (12.0.1-1ubuntu1) ... 2024-03-28T14:05:59.679 INFO:teuthology.orchestra.run.smithi171.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libuuid1 amd64 2.37.2-4ubuntu3.3 [24.4 kB] 2024-03-28T14:05:59.681 INFO:teuthology.orchestra.run.smithi171.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy/main amd64 bison amd64 2:3.8.2+dfsg-1build1 [748 kB] 2024-03-28T14:05:59.722 INFO:teuthology.orchestra.run.smithi171.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblzo2-2 amd64 2.10-2build3 [53.7 kB] 2024-03-28T14:05:59.724 INFO:teuthology.orchestra.run.smithi171.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy/universe amd64 dump amd64 0.4b47-3 [133 kB] 2024-03-28T14:05:59.731 INFO:teuthology.orchestra.run.smithi171.stdout:Get:9 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 indent amd64 2.2.12-1ubuntu0.22.04.1 [90.8 kB] 2024-03-28T14:05:59.735 INFO:teuthology.orchestra.run.smithi171.stdout:Get:10 http://archive.ubuntu.com/ubuntu jammy/main amd64 libaio-dev amd64 0.3.112-13build1 [21.2 kB] 2024-03-28T14:05:59.736 INFO:teuthology.orchestra.run.smithi171.stdout:Get:11 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libbluetooth3 amd64 5.64-0ubuntu1.1 [87.0 kB] 2024-03-28T14:05:59.903 INFO:teuthology.orchestra.run.smithi171.stdout:Get:12 http://archive.ubuntu.com/ubuntu jammy/main amd64 libelf-dev amd64 0.186-1build1 [64.4 kB] 2024-03-28T14:06:00.196 INFO:teuthology.orchestra.run.smithi171.stdout:Get:13 http://archive.ubuntu.com/ubuntu jammy/main amd64 libfl2 amd64 2.6.4-8build2 [10.7 kB] 2024-03-28T14:06:00.205 INFO:teuthology.orchestra.run.smithi171.stdout:Get:14 http://archive.ubuntu.com/ubuntu jammy/main amd64 libfl-dev amd64 2.6.4-8build2 [6,236 B] 2024-03-28T14:06:00.210 INFO:teuthology.orchestra.run.smithi171.stdout:Get:15 http://archive.ubuntu.com/ubuntu jammy/main amd64 libjson-perl all 4.04000-1 [81.8 kB] 2024-03-28T14:06:00.295 INFO:teuthology.orchestra.run.smithi171.stdout:Get:16 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libllvm14 amd64 1:14.0.0-1ubuntu1.1 [24.0 MB] 2024-03-28T14:06:00.818 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up python3-rbd (19.0.0-2499-ge0e48ba6-1jammy) ... 2024-03-28T14:06:00.944 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up rbd-fuse (19.0.0-2499-ge0e48ba6-1jammy) ... 2024-03-28T14:06:01.070 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up python3-rgw (19.0.0-2499-ge0e48ba6-1jammy) ... 2024-03-28T14:06:01.179 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up python3-cephfs (19.0.0-2499-ge0e48ba6-1jammy) ... 2024-03-28T14:06:01.288 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up cephfs-shell (19.0.0-2499-ge0e48ba6-1jammy) ... 2024-03-28T14:06:01.373 INFO:teuthology.orchestra.run.smithi171.stdout:Get:17 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libmbim-glib4 amd64 1.28.0-1~ubuntu20.04.1 [191 kB] 2024-03-28T14:06:01.377 INFO:teuthology.orchestra.run.smithi171.stdout:Get:18 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libmbim-proxy amd64 1.28.0-1~ubuntu20.04.1 [6,130 B] 2024-03-28T14:06:01.378 INFO:teuthology.orchestra.run.smithi171.stdout:Get:19 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libmm-glib0 amd64 1.20.0-1~ubuntu22.04.3 [263 kB] 2024-03-28T14:06:01.384 INFO:teuthology.orchestra.run.smithi171.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy/main amd64 libndp0 amd64 1.8-0ubuntu3 [12.3 kB] 2024-03-28T14:06:01.385 INFO:teuthology.orchestra.run.smithi171.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libnm0 amd64 1.36.6-0ubuntu2 [456 kB] 2024-03-28T14:06:01.396 INFO:teuthology.orchestra.run.smithi171.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libpq5 amd64 14.11-0ubuntu0.22.04.1 [144 kB] 2024-03-28T14:06:01.414 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up ceph-common (19.0.0-2499-ge0e48ba6-1jammy) ... 2024-03-28T14:06:01.563 INFO:teuthology.orchestra.run.smithi171.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libqmi-glib5 amd64 1.32.0-1ubuntu0.22.04.1 [772 kB] 2024-03-28T14:06:01.939 INFO:teuthology.orchestra.run.smithi104.stdout:Adding group ceph....done 2024-03-28T14:06:02.161 INFO:teuthology.orchestra.run.smithi171.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libqmi-proxy amd64 1.32.0-1ubuntu0.22.04.1 [6,072 B] 2024-03-28T14:06:02.161 INFO:teuthology.orchestra.run.smithi171.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libssl-dev amd64 3.0.2-0ubuntu1.15 [2,376 kB] 2024-03-28T14:06:02.308 INFO:teuthology.orchestra.run.smithi104.stdout:Adding system user ceph....done 2024-03-28T14:06:02.340 INFO:teuthology.orchestra.run.smithi171.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy/main amd64 libteamdctl0 amd64 1.31-1build2 [12.6 kB] 2024-03-28T14:06:02.341 INFO:teuthology.orchestra.run.smithi171.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 modemmanager amd64 1.20.0-1~ubuntu22.04.3 [1,094 kB] 2024-03-28T14:06:02.379 INFO:teuthology.orchestra.run.smithi171.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 network-manager amd64 1.36.6-0ubuntu2 [2,203 kB] 2024-03-28T14:06:02.441 INFO:teuthology.orchestra.run.smithi171.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy/main amd64 ppp amd64 2.4.9-1+1ubuntu3 [363 kB] 2024-03-28T14:06:02.447 INFO:teuthology.orchestra.run.smithi171.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy/main amd64 pptp-linux amd64 1.10.0-1build3 [42.5 kB] 2024-03-28T14:06:02.448 INFO:teuthology.orchestra.run.smithi171.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy/main amd64 network-manager-pptp amd64 1.2.10-1 [30.9 kB] 2024-03-28T14:06:02.449 INFO:teuthology.orchestra.run.smithi171.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy/main amd64 postgresql-client-common all 238 [29.6 kB] 2024-03-28T14:06:02.450 INFO:teuthology.orchestra.run.smithi171.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 postgresql-client-14 amd64 14.11-0ubuntu0.22.04.1 [1,222 kB] 2024-03-28T14:06:02.477 INFO:teuthology.orchestra.run.smithi104.stdout:Setting system user ceph properties....done 2024-03-28T14:06:02.494 INFO:teuthology.orchestra.run.smithi104.stdout:chown: cannot access '/var/log/ceph/*.log*': No such file or directory 2024-03-28T14:06:02.632 INFO:teuthology.orchestra.run.smithi104.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /lib/systemd/system/ceph.target. 2024-03-28T14:06:02.657 INFO:teuthology.orchestra.run.smithi171.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy/main amd64 postgresql-common all 238 [169 kB] 2024-03-28T14:06:02.973 INFO:teuthology.orchestra.run.smithi104.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service -> /lib/systemd/system/rbdmap.service. 2024-03-28T14:06:03.070 INFO:teuthology.orchestra.run.smithi171.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 postgresql-14 amd64 14.11-0ubuntu0.22.04.1 [16.2 MB] 2024-03-28T14:06:03.448 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up cephfs-mirror (19.0.0-2499-ge0e48ba6-1jammy) ... 2024-03-28T14:06:03.675 INFO:teuthology.orchestra.run.smithi104.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target -> /lib/systemd/system/cephfs-mirror.target. 2024-03-28T14:06:03.676 INFO:teuthology.orchestra.run.smithi104.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target -> /lib/systemd/system/cephfs-mirror.target. 2024-03-28T14:06:03.870 INFO:teuthology.orchestra.run.smithi171.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy/main amd64 postgresql all 14+238 [3,288 B] 2024-03-28T14:06:03.871 INFO:teuthology.orchestra.run.smithi171.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy/main amd64 postgresql-client all 14+238 [3,292 B] 2024-03-28T14:06:03.871 INFO:teuthology.orchestra.run.smithi171.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy/main amd64 postgresql-contrib all 14+238 [3,292 B] 2024-03-28T14:06:03.871 INFO:teuthology.orchestra.run.smithi171.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy/main amd64 tcl8.6 amd64 8.6.12+dfsg-1build1 [15.0 kB] 2024-03-28T14:06:03.905 INFO:teuthology.orchestra.run.smithi171.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy/main amd64 tcl amd64 8.6.11+1build2 [4,678 B] 2024-03-28T14:06:03.905 INFO:teuthology.orchestra.run.smithi171.stdout:Get:41 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 wpasupplicant amd64 2:2.10-6ubuntu2 [1,481 kB] 2024-03-28T14:06:03.935 INFO:teuthology.orchestra.run.smithi171.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy/main amd64 usb-modeswitch-data all 20191128-4 [33.2 kB] 2024-03-28T14:06:03.936 INFO:teuthology.orchestra.run.smithi171.stdout:Get:43 http://archive.ubuntu.com/ubuntu jammy/main amd64 usb-modeswitch amd64 2.6.1-3ubuntu2 [46.0 kB] 2024-03-28T14:06:04.141 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up ceph-test (19.0.0-2499-ge0e48ba6-1jammy) ... 2024-03-28T14:06:04.242 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up radosgw (19.0.0-2499-ge0e48ba6-1jammy) ... 2024-03-28T14:06:04.342 INFO:teuthology.orchestra.run.smithi171.stdout:Preconfiguring packages ... 2024-03-28T14:06:04.498 INFO:teuthology.orchestra.run.smithi171.stdout:Fetched 55.7 MB in 5s (10.7 MB/s) 2024-03-28T14:06:04.656 INFO:teuthology.orchestra.run.smithi104.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-03-28T14:06:04.656 INFO:teuthology.orchestra.run.smithi104.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-03-28T14:06:04.876 INFO:teuthology.orchestra.run.smithi171.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 173988 files and directories currently installed.) 2024-03-28T14:06:04.882 INFO:teuthology.orchestra.run.smithi171.stdout:Preparing to unpack .../util-linux_2.37.2-4ubuntu3.3_amd64.deb ... 2024-03-28T14:06:05.053 INFO:teuthology.orchestra.run.smithi171.stdout:Unpacking util-linux (2.37.2-4ubuntu3.3) over (2.37.2-4ubuntu3) ... 2024-03-28T14:06:05.144 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up ceph-base (19.0.0-2499-ge0e48ba6-1jammy) ... 2024-03-28T14:06:05.496 INFO:teuthology.orchestra.run.smithi104.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service -> /lib/systemd/system/ceph-crash.service. 2024-03-28T14:06:05.921 INFO:teuthology.orchestra.run.smithi171.stdout:Setting up util-linux (2.37.2-4ubuntu3.3) ... 2024-03-28T14:06:05.979 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up ceph-mds (19.0.0-2499-ge0e48ba6-1jammy) ... 2024-03-28T14:06:06.180 INFO:teuthology.orchestra.run.smithi104.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-03-28T14:06:06.181 INFO:teuthology.orchestra.run.smithi104.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-03-28T14:06:06.639 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up ceph-mgr (19.0.0-2499-ge0e48ba6-1jammy) ... 2024-03-28T14:06:06.855 INFO:teuthology.orchestra.run.smithi104.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-03-28T14:06:06.856 INFO:teuthology.orchestra.run.smithi104.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-03-28T14:06:06.893 INFO:teuthology.orchestra.run.smithi171.stdout:fstrim.service is a disabled or a static unit not running, not starting it. 2024-03-28T14:06:07.172 INFO:teuthology.orchestra.run.smithi171.stdout:Selecting previously unselected package flex. 2024-03-28T14:06:07.275 INFO:teuthology.orchestra.run.smithi171.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 173988 files and directories currently installed.) 2024-03-28T14:06:07.281 INFO:teuthology.orchestra.run.smithi171.stdout:Preparing to unpack .../flex_2.6.4-8build2_amd64.deb ... 2024-03-28T14:06:07.306 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up ceph-osd (19.0.0-2499-ge0e48ba6-1jammy) ... 2024-03-28T14:06:07.320 INFO:teuthology.orchestra.run.smithi171.stdout:Unpacking flex (2.6.4-8build2) ... 2024-03-28T14:06:07.588 INFO:teuthology.orchestra.run.smithi104.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-03-28T14:06:07.588 INFO:teuthology.orchestra.run.smithi104.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-03-28T14:06:07.854 INFO:teuthology.orchestra.run.smithi171.stdout:Preparing to unpack .../libssl3_3.0.2-0ubuntu1.15_amd64.deb ... 2024-03-28T14:06:08.019 INFO:teuthology.orchestra.run.smithi171.stdout:Unpacking libssl3:amd64 (3.0.2-0ubuntu1.15) over (3.0.2-0ubuntu1.7) ... 2024-03-28T14:06:08.092 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up ceph-mgr-k8sevents (19.0.0-2499-ge0e48ba6-1jammy) ... 2024-03-28T14:06:08.218 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up ceph-mgr-diskprediction-local (19.0.0-2499-ge0e48ba6-1jammy) ... 2024-03-28T14:06:08.361 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up ceph-mon (19.0.0-2499-ge0e48ba6-1jammy) ... 2024-03-28T14:06:08.413 INFO:teuthology.orchestra.run.smithi171.stdout:Setting up libssl3:amd64 (3.0.2-0ubuntu1.15) ... 2024-03-28T14:06:08.583 INFO:teuthology.orchestra.run.smithi104.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-03-28T14:06:08.583 INFO:teuthology.orchestra.run.smithi104.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-03-28T14:06:08.973 INFO:teuthology.orchestra.run.smithi171.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 174061 files and directories currently installed.) 2024-03-28T14:06:08.978 INFO:teuthology.orchestra.run.smithi171.stdout:Preparing to unpack .../uuid-dev_2.37.2-4ubuntu3.3_amd64.deb ... 2024-03-28T14:06:09.045 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up ceph-mgr-cephadm (19.0.0-2499-ge0e48ba6-1jammy) ... 2024-03-28T14:06:09.095 INFO:teuthology.orchestra.run.smithi171.stdout:Unpacking uuid-dev:amd64 (2.37.2-4ubuntu3.3) over (2.37.2-4ubuntu3) ... 2024-03-28T14:06:09.171 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up ceph (19.0.0-2499-ge0e48ba6-1jammy) ... 2024-03-28T14:06:09.272 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up ceph-mgr-dashboard (19.0.0-2499-ge0e48ba6-1jammy) ... 2024-03-28T14:06:09.415 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up ceph-volume (19.0.0-2499-ge0e48ba6-1jammy) ... 2024-03-28T14:06:09.497 INFO:teuthology.orchestra.run.smithi171.stdout:Preparing to unpack .../libuuid1_2.37.2-4ubuntu3.3_amd64.deb ... 2024-03-28T14:06:09.628 INFO:teuthology.orchestra.run.smithi171.stdout:Unpacking libuuid1:amd64 (2.37.2-4ubuntu3.3) over (2.37.2-4ubuntu3) ... 2024-03-28T14:06:09.767 INFO:teuthology.orchestra.run.smithi104.stdout:Processing triggers for mailcap (3.70+nmu1ubuntu1) ... 2024-03-28T14:06:09.876 INFO:teuthology.orchestra.run.smithi104.stdout:Processing triggers for libc-bin (2.35-0ubuntu3.1) ... 2024-03-28T14:06:09.932 INFO:teuthology.orchestra.run.smithi171.stdout:Setting up libuuid1:amd64 (2.37.2-4ubuntu3.3) ... 2024-03-28T14:06:10.012 INFO:teuthology.orchestra.run.smithi104.stdout:Processing triggers for man-db (2.10.2-1) ... 2024-03-28T14:06:10.347 INFO:teuthology.orchestra.run.smithi171.stdout:Selecting previously unselected package bison. 2024-03-28T14:06:10.452 INFO:teuthology.orchestra.run.smithi171.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 174061 files and directories currently installed.) 2024-03-28T14:06:10.458 INFO:teuthology.orchestra.run.smithi171.stdout:Preparing to unpack .../00-bison_2%3a3.8.2+dfsg-1build1_amd64.deb ... 2024-03-28T14:06:10.508 INFO:teuthology.orchestra.run.smithi171.stdout:Unpacking bison (2:3.8.2+dfsg-1build1) ... 2024-03-28T14:06:10.910 INFO:teuthology.orchestra.run.smithi171.stdout:Selecting previously unselected package liblzo2-2:amd64. 2024-03-28T14:06:10.938 INFO:teuthology.orchestra.run.smithi171.stdout:Preparing to unpack .../01-liblzo2-2_2.10-2build3_amd64.deb ... 2024-03-28T14:06:10.976 INFO:teuthology.orchestra.run.smithi171.stdout:Unpacking liblzo2-2:amd64 (2.10-2build3) ... 2024-03-28T14:06:11.121 INFO:teuthology.orchestra.run.smithi104.stdout:Processing triggers for install-info (6.8-4build1) ... 2024-03-28T14:06:11.322 INFO:teuthology.orchestra.run.smithi171.stdout:Selecting previously unselected package dump. 2024-03-28T14:06:11.350 INFO:teuthology.orchestra.run.smithi171.stdout:Preparing to unpack .../02-dump_0.4b47-3_amd64.deb ... 2024-03-28T14:06:11.388 INFO:teuthology.orchestra.run.smithi171.stdout:Unpacking dump (0.4b47-3) ... 2024-03-28T14:06:11.758 INFO:teuthology.orchestra.run.smithi104.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2024-03-28T14:06:11.759 INFO:teuthology.orchestra.run.smithi171.stdout:Selecting previously unselected package indent. 2024-03-28T14:06:11.761 DEBUG:teuthology.orchestra.run.smithi104:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install bison flex libelf-dev libssl-dev network-manager iproute2 util-linux dump indent libaio-dev libtool-bin uuid-dev xfslibs-dev postgresql postgresql-client postgresql-common postgresql-contrib 2024-03-28T14:06:11.788 INFO:teuthology.orchestra.run.smithi171.stdout:Preparing to unpack .../03-indent_2.2.12-1ubuntu0.22.04.1_amd64.deb ... 2024-03-28T14:06:11.817 INFO:teuthology.orchestra.run.smithi104.stdout:Reading package lists... 2024-03-28T14:06:11.826 INFO:teuthology.orchestra.run.smithi171.stdout:Unpacking indent (2.2.12-1ubuntu0.22.04.1) ... 2024-03-28T14:06:12.014 INFO:teuthology.orchestra.run.smithi104.stdout:Building dependency tree... 2024-03-28T14:06:12.015 INFO:teuthology.orchestra.run.smithi104.stdout:Reading state information... 2024-03-28T14:06:12.096 INFO:teuthology.orchestra.run.smithi171.stdout:Selecting previously unselected package libaio-dev:amd64. 2024-03-28T14:06:12.124 INFO:teuthology.orchestra.run.smithi171.stdout:Preparing to unpack .../04-libaio-dev_0.3.112-13build1_amd64.deb ... 2024-03-28T14:06:12.163 INFO:teuthology.orchestra.run.smithi171.stdout:Unpacking libaio-dev:amd64 (0.3.112-13build1) ... 2024-03-28T14:06:12.193 INFO:teuthology.orchestra.run.smithi104.stdout:iproute2 is already the newest version (5.15.0-1ubuntu2). 2024-03-28T14:06:12.193 INFO:teuthology.orchestra.run.smithi104.stdout:libtool-bin is already the newest version (2.4.6-15build2). 2024-03-28T14:06:12.194 INFO:teuthology.orchestra.run.smithi104.stdout:xfslibs-dev is already the newest version (5.13.0-1ubuntu2). 2024-03-28T14:06:12.194 INFO:teuthology.orchestra.run.smithi104.stdout:The following packages were automatically installed and are no longer required: 2024-03-28T14:06:12.194 INFO:teuthology.orchestra.run.smithi104.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-03-28T14:06:12.194 INFO:teuthology.orchestra.run.smithi104.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-03-28T14:06:12.194 INFO:teuthology.orchestra.run.smithi104.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-03-28T14:06:12.194 INFO:teuthology.orchestra.run.smithi104.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2024-03-28T14:06:12.195 INFO:teuthology.orchestra.run.smithi104.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2024-03-28T14:06:12.195 INFO:teuthology.orchestra.run.smithi104.stdout:Use 'sudo apt autoremove' to remove them. 2024-03-28T14:06:12.195 INFO:teuthology.orchestra.run.smithi104.stdout:The following additional packages will be installed: 2024-03-28T14:06:12.195 INFO:teuthology.orchestra.run.smithi104.stdout: libbluetooth3 libfl-dev libfl2 libjson-perl libllvm14 liblzo2-2 2024-03-28T14:06:12.195 INFO:teuthology.orchestra.run.smithi104.stdout: libmbim-glib4 libmbim-proxy libmm-glib0 libndp0 libnm0 libpq5 libqmi-glib5 2024-03-28T14:06:12.196 INFO:teuthology.orchestra.run.smithi104.stdout: libqmi-proxy libssl3 libteamdctl0 libuuid1 modemmanager network-manager-pptp 2024-03-28T14:06:12.196 INFO:teuthology.orchestra.run.smithi104.stdout: postgresql-14 postgresql-client-14 postgresql-client-common ppp pptp-linux 2024-03-28T14:06:12.196 INFO:teuthology.orchestra.run.smithi104.stdout: tcl tcl8.6 usb-modeswitch usb-modeswitch-data wpasupplicant 2024-03-28T14:06:12.197 INFO:teuthology.orchestra.run.smithi104.stdout:Suggested packages: 2024-03-28T14:06:12.197 INFO:teuthology.orchestra.run.smithi104.stdout: bison-doc flex-doc indent-doc libssl-doc avahi-autoipd libteam-utils 2024-03-28T14:06:12.197 INFO:teuthology.orchestra.run.smithi104.stdout: postgresql-doc postgresql-doc-14 tcl-tclreadline comgt wvdial 2024-03-28T14:06:12.197 INFO:teuthology.orchestra.run.smithi104.stdout: util-linux-locales wpagui libengine-pkcs11-openssl 2024-03-28T14:06:12.239 INFO:teuthology.orchestra.run.smithi104.stdout:The following NEW packages will be installed: 2024-03-28T14:06:12.240 INFO:teuthology.orchestra.run.smithi104.stdout: bison dump flex indent libaio-dev libbluetooth3 libelf-dev libfl-dev libfl2 2024-03-28T14:06:12.240 INFO:teuthology.orchestra.run.smithi104.stdout: libjson-perl libllvm14 liblzo2-2 libmbim-glib4 libmbim-proxy libmm-glib0 2024-03-28T14:06:12.240 INFO:teuthology.orchestra.run.smithi104.stdout: libndp0 libnm0 libpq5 libqmi-glib5 libqmi-proxy libssl-dev libteamdctl0 2024-03-28T14:06:12.240 INFO:teuthology.orchestra.run.smithi104.stdout: modemmanager network-manager network-manager-pptp postgresql postgresql-14 2024-03-28T14:06:12.240 INFO:teuthology.orchestra.run.smithi104.stdout: postgresql-client postgresql-client-14 postgresql-client-common 2024-03-28T14:06:12.240 INFO:teuthology.orchestra.run.smithi104.stdout: postgresql-common postgresql-contrib ppp pptp-linux tcl tcl8.6 2024-03-28T14:06:12.240 INFO:teuthology.orchestra.run.smithi104.stdout: usb-modeswitch usb-modeswitch-data wpasupplicant 2024-03-28T14:06:12.241 INFO:teuthology.orchestra.run.smithi104.stdout:The following packages will be upgraded: 2024-03-28T14:06:12.241 INFO:teuthology.orchestra.run.smithi104.stdout: libssl3 libuuid1 util-linux uuid-dev 2024-03-28T14:06:12.299 INFO:teuthology.orchestra.run.smithi104.stdout:4 upgraded, 39 newly installed, 0 to remove and 294 not upgraded. 2024-03-28T14:06:12.299 INFO:teuthology.orchestra.run.smithi104.stdout:Need to get 55.7 MB of archives. 2024-03-28T14:06:12.299 INFO:teuthology.orchestra.run.smithi104.stdout:After this operation, 202 MB of additional disk space will be used. 2024-03-28T14:06:12.299 INFO:teuthology.orchestra.run.smithi104.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 util-linux amd64 2.37.2-4ubuntu3.3 [1,063 kB] 2024-03-28T14:06:12.459 INFO:teuthology.orchestra.run.smithi104.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy/main amd64 flex amd64 2.6.4-8build2 [307 kB] 2024-03-28T14:06:12.466 INFO:teuthology.orchestra.run.smithi104.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libssl3 amd64 3.0.2-0ubuntu1.15 [1,905 kB] 2024-03-28T14:06:12.493 INFO:teuthology.orchestra.run.smithi171.stdout:Selecting previously unselected package libbluetooth3:amd64. 2024-03-28T14:06:12.506 INFO:teuthology.orchestra.run.smithi104.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 uuid-dev amd64 2.37.2-4ubuntu3.3 [33.1 kB] 2024-03-28T14:06:12.507 INFO:teuthology.orchestra.run.smithi104.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libuuid1 amd64 2.37.2-4ubuntu3.3 [24.4 kB] 2024-03-28T14:06:12.507 INFO:teuthology.orchestra.run.smithi104.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy/main amd64 bison amd64 2:3.8.2+dfsg-1build1 [748 kB] 2024-03-28T14:06:12.519 INFO:teuthology.orchestra.run.smithi104.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblzo2-2 amd64 2.10-2build3 [53.7 kB] 2024-03-28T14:06:12.522 INFO:teuthology.orchestra.run.smithi104.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy/universe amd64 dump amd64 0.4b47-3 [133 kB] 2024-03-28T14:06:12.522 INFO:teuthology.orchestra.run.smithi104.stdout:Get:9 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 indent amd64 2.2.12-1ubuntu0.22.04.1 [90.8 kB] 2024-03-28T14:06:12.523 INFO:teuthology.orchestra.run.smithi171.stdout:Preparing to unpack .../05-libbluetooth3_5.64-0ubuntu1.1_amd64.deb ... 2024-03-28T14:06:12.523 INFO:teuthology.orchestra.run.smithi104.stdout:Get:10 http://archive.ubuntu.com/ubuntu jammy/main amd64 libaio-dev amd64 0.3.112-13build1 [21.2 kB] 2024-03-28T14:06:12.524 INFO:teuthology.orchestra.run.smithi104.stdout:Get:11 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libbluetooth3 amd64 5.64-0ubuntu1.1 [87.0 kB] 2024-03-28T14:06:12.525 INFO:teuthology.orchestra.run.smithi104.stdout:Get:12 http://archive.ubuntu.com/ubuntu jammy/main amd64 libelf-dev amd64 0.186-1build1 [64.4 kB] 2024-03-28T14:06:12.528 INFO:teuthology.orchestra.run.smithi104.stdout:Get:13 http://archive.ubuntu.com/ubuntu jammy/main amd64 libfl2 amd64 2.6.4-8build2 [10.7 kB] 2024-03-28T14:06:12.547 INFO:teuthology.orchestra.run.smithi104.stdout:Get:14 http://archive.ubuntu.com/ubuntu jammy/main amd64 libfl-dev amd64 2.6.4-8build2 [6,236 B] 2024-03-28T14:06:12.547 INFO:teuthology.orchestra.run.smithi104.stdout:Get:15 http://archive.ubuntu.com/ubuntu jammy/main amd64 libjson-perl all 4.04000-1 [81.8 kB] 2024-03-28T14:06:12.548 INFO:teuthology.orchestra.run.smithi104.stdout:Get:16 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libllvm14 amd64 1:14.0.0-1ubuntu1.1 [24.0 MB] 2024-03-28T14:06:12.551 INFO:teuthology.orchestra.run.smithi171.stdout:Unpacking libbluetooth3:amd64 (5.64-0ubuntu1.1) ... 2024-03-28T14:06:12.812 INFO:teuthology.orchestra.run.smithi171.stdout:Selecting previously unselected package libelf-dev:amd64. 2024-03-28T14:06:12.829 INFO:teuthology.orchestra.run.smithi171.stdout:Preparing to unpack .../06-libelf-dev_0.186-1build1_amd64.deb ... 2024-03-28T14:06:12.871 INFO:teuthology.orchestra.run.smithi171.stdout:Unpacking libelf-dev:amd64 (0.186-1build1) ... 2024-03-28T14:06:12.944 INFO:teuthology.orchestra.run.smithi104.stdout:Get:17 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libmbim-glib4 amd64 1.28.0-1~ubuntu20.04.1 [191 kB] 2024-03-28T14:06:12.946 INFO:teuthology.orchestra.run.smithi104.stdout:Get:18 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libmbim-proxy amd64 1.28.0-1~ubuntu20.04.1 [6,130 B] 2024-03-28T14:06:12.946 INFO:teuthology.orchestra.run.smithi104.stdout:Get:19 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libmm-glib0 amd64 1.20.0-1~ubuntu22.04.3 [263 kB] 2024-03-28T14:06:12.950 INFO:teuthology.orchestra.run.smithi104.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy/main amd64 libndp0 amd64 1.8-0ubuntu3 [12.3 kB] 2024-03-28T14:06:12.950 INFO:teuthology.orchestra.run.smithi104.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libnm0 amd64 1.36.6-0ubuntu2 [456 kB] 2024-03-28T14:06:12.955 INFO:teuthology.orchestra.run.smithi104.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libpq5 amd64 14.11-0ubuntu0.22.04.1 [144 kB] 2024-03-28T14:06:12.956 INFO:teuthology.orchestra.run.smithi104.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libqmi-glib5 amd64 1.32.0-1ubuntu0.22.04.1 [772 kB] 2024-03-28T14:06:12.964 INFO:teuthology.orchestra.run.smithi104.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libqmi-proxy amd64 1.32.0-1ubuntu0.22.04.1 [6,072 B] 2024-03-28T14:06:12.965 INFO:teuthology.orchestra.run.smithi104.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libssl-dev amd64 3.0.2-0ubuntu1.15 [2,376 kB] 2024-03-28T14:06:12.988 INFO:teuthology.orchestra.run.smithi104.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy/main amd64 libteamdctl0 amd64 1.31-1build2 [12.6 kB] 2024-03-28T14:06:12.999 INFO:teuthology.orchestra.run.smithi104.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 modemmanager amd64 1.20.0-1~ubuntu22.04.3 [1,094 kB] 2024-03-28T14:06:13.013 INFO:teuthology.orchestra.run.smithi104.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 network-manager amd64 1.36.6-0ubuntu2 [2,203 kB] 2024-03-28T14:06:13.064 INFO:teuthology.orchestra.run.smithi104.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy/main amd64 ppp amd64 2.4.9-1+1ubuntu3 [363 kB] 2024-03-28T14:06:13.069 INFO:teuthology.orchestra.run.smithi104.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy/main amd64 pptp-linux amd64 1.10.0-1build3 [42.5 kB] 2024-03-28T14:06:13.069 INFO:teuthology.orchestra.run.smithi104.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy/main amd64 network-manager-pptp amd64 1.2.10-1 [30.9 kB] 2024-03-28T14:06:13.070 INFO:teuthology.orchestra.run.smithi104.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy/main amd64 postgresql-client-common all 238 [29.6 kB] 2024-03-28T14:06:13.071 INFO:teuthology.orchestra.run.smithi104.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 postgresql-client-14 amd64 14.11-0ubuntu0.22.04.1 [1,222 kB] 2024-03-28T14:06:13.086 INFO:teuthology.orchestra.run.smithi104.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy/main amd64 postgresql-common all 238 [169 kB] 2024-03-28T14:06:13.088 INFO:teuthology.orchestra.run.smithi104.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 postgresql-14 amd64 14.11-0ubuntu0.22.04.1 [16.2 MB] 2024-03-28T14:06:13.217 INFO:teuthology.orchestra.run.smithi171.stdout:Selecting previously unselected package libfl2:amd64. 2024-03-28T14:06:13.246 INFO:teuthology.orchestra.run.smithi171.stdout:Preparing to unpack .../07-libfl2_2.6.4-8build2_amd64.deb ... 2024-03-28T14:06:13.284 INFO:teuthology.orchestra.run.smithi171.stdout:Unpacking libfl2:amd64 (2.6.4-8build2) ... 2024-03-28T14:06:13.376 INFO:teuthology.orchestra.run.smithi104.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy/main amd64 postgresql all 14+238 [3,288 B] 2024-03-28T14:06:13.377 INFO:teuthology.orchestra.run.smithi104.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy/main amd64 postgresql-client all 14+238 [3,292 B] 2024-03-28T14:06:13.377 INFO:teuthology.orchestra.run.smithi104.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy/main amd64 postgresql-contrib all 14+238 [3,292 B] 2024-03-28T14:06:13.377 INFO:teuthology.orchestra.run.smithi104.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy/main amd64 tcl8.6 amd64 8.6.12+dfsg-1build1 [15.0 kB] 2024-03-28T14:06:13.377 INFO:teuthology.orchestra.run.smithi104.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy/main amd64 tcl amd64 8.6.11+1build2 [4,678 B] 2024-03-28T14:06:13.377 INFO:teuthology.orchestra.run.smithi104.stdout:Get:41 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 wpasupplicant amd64 2:2.10-6ubuntu2 [1,481 kB] 2024-03-28T14:06:13.389 INFO:teuthology.orchestra.run.smithi104.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy/main amd64 usb-modeswitch-data all 20191128-4 [33.2 kB] 2024-03-28T14:06:13.389 INFO:teuthology.orchestra.run.smithi104.stdout:Get:43 http://archive.ubuntu.com/ubuntu jammy/main amd64 usb-modeswitch amd64 2.6.1-3ubuntu2 [46.0 kB] 2024-03-28T14:06:13.520 INFO:teuthology.orchestra.run.smithi171.stdout:Selecting previously unselected package libfl-dev:amd64. 2024-03-28T14:06:13.536 INFO:teuthology.orchestra.run.smithi171.stdout:Preparing to unpack .../08-libfl-dev_2.6.4-8build2_amd64.deb ... 2024-03-28T14:06:13.578 INFO:teuthology.orchestra.run.smithi171.stdout:Unpacking libfl-dev:amd64 (2.6.4-8build2) ... 2024-03-28T14:06:13.797 INFO:teuthology.orchestra.run.smithi104.stdout:Preconfiguring packages ... 2024-03-28T14:06:13.832 INFO:teuthology.orchestra.run.smithi171.stdout:Selecting previously unselected package libjson-perl. 2024-03-28T14:06:13.860 INFO:teuthology.orchestra.run.smithi171.stdout:Preparing to unpack .../09-libjson-perl_4.04000-1_all.deb ... 2024-03-28T14:06:13.898 INFO:teuthology.orchestra.run.smithi171.stdout:Unpacking libjson-perl (4.04000-1) ... 2024-03-28T14:06:13.925 INFO:teuthology.orchestra.run.smithi104.stdout:Fetched 55.7 MB in 1s (48.8 MB/s) 2024-03-28T14:06:14.219 INFO:teuthology.orchestra.run.smithi171.stdout:Selecting previously unselected package libllvm14:amd64. 2024-03-28T14:06:14.247 INFO:teuthology.orchestra.run.smithi171.stdout:Preparing to unpack .../10-libllvm14_1%3a14.0.0-1ubuntu1.1_amd64.deb ... 2024-03-28T14:06:14.285 INFO:teuthology.orchestra.run.smithi171.stdout:Unpacking libllvm14:amd64 (1:14.0.0-1ubuntu1.1) ... 2024-03-28T14:06:14.311 INFO:teuthology.orchestra.run.smithi104.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 173988 files and directories currently installed.) 2024-03-28T14:06:14.316 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../util-linux_2.37.2-4ubuntu3.3_amd64.deb ... 2024-03-28T14:06:14.460 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking util-linux (2.37.2-4ubuntu3.3) over (2.37.2-4ubuntu3) ... 2024-03-28T14:06:15.081 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up util-linux (2.37.2-4ubuntu3.3) ... 2024-03-28T14:06:16.014 INFO:teuthology.orchestra.run.smithi171.stdout:Selecting previously unselected package libmbim-glib4:amd64. 2024-03-28T14:06:16.042 INFO:teuthology.orchestra.run.smithi171.stdout:Preparing to unpack .../11-libmbim-glib4_1.28.0-1~ubuntu20.04.1_amd64.deb ... 2024-03-28T14:06:16.051 INFO:teuthology.orchestra.run.smithi104.stdout:fstrim.service is a disabled or a static unit not running, not starting it. 2024-03-28T14:06:16.097 INFO:teuthology.orchestra.run.smithi171.stdout:Unpacking libmbim-glib4:amd64 (1.28.0-1~ubuntu20.04.1) ... 2024-03-28T14:06:16.289 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package flex. 2024-03-28T14:06:16.359 INFO:teuthology.orchestra.run.smithi171.stdout:Selecting previously unselected package libmbim-proxy. 2024-03-28T14:06:16.375 INFO:teuthology.orchestra.run.smithi171.stdout:Preparing to unpack .../12-libmbim-proxy_1.28.0-1~ubuntu20.04.1_amd64.deb ... 2024-03-28T14:06:16.387 INFO:teuthology.orchestra.run.smithi104.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 173988 files and directories currently installed.) 2024-03-28T14:06:16.393 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../flex_2.6.4-8build2_amd64.deb ... 2024-03-28T14:06:16.417 INFO:teuthology.orchestra.run.smithi171.stdout:Unpacking libmbim-proxy (1.28.0-1~ubuntu20.04.1) ... 2024-03-28T14:06:16.437 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking flex (2.6.4-8build2) ... 2024-03-28T14:06:16.746 INFO:teuthology.orchestra.run.smithi171.stdout:Selecting previously unselected package libmm-glib0:amd64. 2024-03-28T14:06:16.774 INFO:teuthology.orchestra.run.smithi171.stdout:Preparing to unpack .../13-libmm-glib0_1.20.0-1~ubuntu22.04.3_amd64.deb ... 2024-03-28T14:06:16.813 INFO:teuthology.orchestra.run.smithi171.stdout:Unpacking libmm-glib0:amd64 (1.20.0-1~ubuntu22.04.3) ... 2024-03-28T14:06:16.944 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../libssl3_3.0.2-0ubuntu1.15_amd64.deb ... 2024-03-28T14:06:17.117 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking libssl3:amd64 (3.0.2-0ubuntu1.15) over (3.0.2-0ubuntu1.7) ... 2024-03-28T14:06:17.166 INFO:teuthology.orchestra.run.smithi171.stdout:Selecting previously unselected package libndp0:amd64. 2024-03-28T14:06:17.194 INFO:teuthology.orchestra.run.smithi171.stdout:Preparing to unpack .../14-libndp0_1.8-0ubuntu3_amd64.deb ... 2024-03-28T14:06:17.233 INFO:teuthology.orchestra.run.smithi171.stdout:Unpacking libndp0:amd64 (1.8-0ubuntu3) ... 2024-03-28T14:06:17.536 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up libssl3:amd64 (3.0.2-0ubuntu1.15) ... 2024-03-28T14:06:17.561 INFO:teuthology.orchestra.run.smithi171.stdout:Selecting previously unselected package libnm0:amd64. 2024-03-28T14:06:17.589 INFO:teuthology.orchestra.run.smithi171.stdout:Preparing to unpack .../15-libnm0_1.36.6-0ubuntu2_amd64.deb ... 2024-03-28T14:06:17.620 INFO:teuthology.orchestra.run.smithi171.stdout:Unpacking libnm0:amd64 (1.36.6-0ubuntu2) ... 2024-03-28T14:06:17.965 INFO:teuthology.orchestra.run.smithi171.stdout:Selecting previously unselected package libpq5:amd64. 2024-03-28T14:06:17.993 INFO:teuthology.orchestra.run.smithi171.stdout:Preparing to unpack .../16-libpq5_14.11-0ubuntu0.22.04.1_amd64.deb ... 2024-03-28T14:06:18.034 INFO:teuthology.orchestra.run.smithi171.stdout:Unpacking libpq5:amd64 (14.11-0ubuntu0.22.04.1) ... 2024-03-28T14:06:18.411 INFO:teuthology.orchestra.run.smithi171.stdout:Selecting previously unselected package libqmi-glib5:amd64. 2024-03-28T14:06:18.439 INFO:teuthology.orchestra.run.smithi171.stdout:Preparing to unpack .../17-libqmi-glib5_1.32.0-1ubuntu0.22.04.1_amd64.deb ... 2024-03-28T14:06:18.477 INFO:teuthology.orchestra.run.smithi171.stdout:Unpacking libqmi-glib5:amd64 (1.32.0-1ubuntu0.22.04.1) ... 2024-03-28T14:06:18.567 INFO:teuthology.orchestra.run.smithi104.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 174061 files and directories currently installed.) 2024-03-28T14:06:18.572 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../uuid-dev_2.37.2-4ubuntu3.3_amd64.deb ... 2024-03-28T14:06:18.701 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking uuid-dev:amd64 (2.37.2-4ubuntu3.3) over (2.37.2-4ubuntu3) ... 2024-03-28T14:06:18.797 INFO:teuthology.orchestra.run.smithi171.stdout:Selecting previously unselected package libqmi-proxy. 2024-03-28T14:06:18.825 INFO:teuthology.orchestra.run.smithi171.stdout:Preparing to unpack .../18-libqmi-proxy_1.32.0-1ubuntu0.22.04.1_amd64.deb ... 2024-03-28T14:06:18.864 INFO:teuthology.orchestra.run.smithi171.stdout:Unpacking libqmi-proxy (1.32.0-1ubuntu0.22.04.1) ... 2024-03-28T14:06:19.100 INFO:teuthology.orchestra.run.smithi171.stdout:Selecting previously unselected package libssl-dev:amd64. 2024-03-28T14:06:19.103 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../libuuid1_2.37.2-4ubuntu3.3_amd64.deb ... 2024-03-28T14:06:19.116 INFO:teuthology.orchestra.run.smithi171.stdout:Preparing to unpack .../19-libssl-dev_3.0.2-0ubuntu1.15_amd64.deb ... 2024-03-28T14:06:19.159 INFO:teuthology.orchestra.run.smithi171.stdout:Unpacking libssl-dev:amd64 (3.0.2-0ubuntu1.15) ... 2024-03-28T14:06:19.259 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking libuuid1:amd64 (2.37.2-4ubuntu3.3) over (2.37.2-4ubuntu3) ... 2024-03-28T14:06:19.638 INFO:teuthology.orchestra.run.smithi171.stdout:Selecting previously unselected package libteamdctl0:amd64. 2024-03-28T14:06:19.666 INFO:teuthology.orchestra.run.smithi171.stdout:Preparing to unpack .../20-libteamdctl0_1.31-1build2_amd64.deb ... 2024-03-28T14:06:19.669 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up libuuid1:amd64 (2.37.2-4ubuntu3.3) ... 2024-03-28T14:06:19.705 INFO:teuthology.orchestra.run.smithi171.stdout:Unpacking libteamdctl0:amd64 (1.31-1build2) ... 2024-03-28T14:06:20.027 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package bison. 2024-03-28T14:06:20.067 INFO:teuthology.orchestra.run.smithi171.stdout:Selecting previously unselected package modemmanager. 2024-03-28T14:06:20.095 INFO:teuthology.orchestra.run.smithi171.stdout:Preparing to unpack .../21-modemmanager_1.20.0-1~ubuntu22.04.3_amd64.deb ... 2024-03-28T14:06:20.127 INFO:teuthology.orchestra.run.smithi104.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 174061 files and directories currently installed.) 2024-03-28T14:06:20.132 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../00-bison_2%3a3.8.2+dfsg-1build1_amd64.deb ... 2024-03-28T14:06:20.142 INFO:teuthology.orchestra.run.smithi171.stdout:Unpacking modemmanager (1.20.0-1~ubuntu22.04.3) ... 2024-03-28T14:06:20.178 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking bison (2:3.8.2+dfsg-1build1) ... 2024-03-28T14:06:20.579 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package liblzo2-2:amd64. 2024-03-28T14:06:20.607 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../01-liblzo2-2_2.10-2build3_amd64.deb ... 2024-03-28T14:06:20.646 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking liblzo2-2:amd64 (2.10-2build3) ... 2024-03-28T14:06:20.792 INFO:teuthology.orchestra.run.smithi171.stdout:Selecting previously unselected package network-manager. 2024-03-28T14:06:20.820 INFO:teuthology.orchestra.run.smithi171.stdout:Preparing to unpack .../22-network-manager_1.36.6-0ubuntu2_amd64.deb ... 2024-03-28T14:06:20.867 INFO:teuthology.orchestra.run.smithi171.stdout:Unpacking network-manager (1.36.6-0ubuntu2) ... 2024-03-28T14:06:20.965 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package dump. 2024-03-28T14:06:20.993 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../02-dump_0.4b47-3_amd64.deb ... 2024-03-28T14:06:21.032 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking dump (0.4b47-3) ... 2024-03-28T14:06:21.310 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package indent. 2024-03-28T14:06:21.326 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../03-indent_2.2.12-1ubuntu0.22.04.1_amd64.deb ... 2024-03-28T14:06:21.368 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking indent (2.2.12-1ubuntu0.22.04.1) ... 2024-03-28T14:06:21.459 INFO:teuthology.orchestra.run.smithi171.stdout:Selecting previously unselected package ppp. 2024-03-28T14:06:21.487 INFO:teuthology.orchestra.run.smithi171.stdout:Preparing to unpack .../23-ppp_2.4.9-1+1ubuntu3_amd64.deb ... 2024-03-28T14:06:21.613 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package libaio-dev:amd64. 2024-03-28T14:06:21.641 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../04-libaio-dev_0.3.112-13build1_amd64.deb ... 2024-03-28T14:06:21.679 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking libaio-dev:amd64 (0.3.112-13build1) ... 2024-03-28T14:06:21.780 INFO:teuthology.orchestra.run.smithi171.stdout:Unpacking ppp (2.4.9-1+1ubuntu3) ... 2024-03-28T14:06:22.008 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package libbluetooth3:amd64. 2024-03-28T14:06:22.036 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../05-libbluetooth3_5.64-0ubuntu1.1_amd64.deb ... 2024-03-28T14:06:22.074 INFO:teuthology.orchestra.run.smithi171.stdout:Selecting previously unselected package pptp-linux. 2024-03-28T14:06:22.074 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking libbluetooth3:amd64 (5.64-0ubuntu1.1) ... 2024-03-28T14:06:22.101 INFO:teuthology.orchestra.run.smithi171.stdout:Preparing to unpack .../24-pptp-linux_1.10.0-1build3_amd64.deb ... 2024-03-28T14:06:22.140 INFO:teuthology.orchestra.run.smithi171.stdout:Unpacking pptp-linux (1.10.0-1build3) ... 2024-03-28T14:06:22.335 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package libelf-dev:amd64. 2024-03-28T14:06:22.352 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../06-libelf-dev_0.186-1build1_amd64.deb ... 2024-03-28T14:06:22.393 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking libelf-dev:amd64 (0.186-1build1) ... 2024-03-28T14:06:22.394 INFO:teuthology.orchestra.run.smithi171.stdout:Selecting previously unselected package network-manager-pptp. 2024-03-28T14:06:22.422 INFO:teuthology.orchestra.run.smithi171.stdout:Preparing to unpack .../25-network-manager-pptp_1.2.10-1_amd64.deb ... 2024-03-28T14:06:22.460 INFO:teuthology.orchestra.run.smithi171.stdout:Unpacking network-manager-pptp (1.2.10-1) ... 2024-03-28T14:06:22.705 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package libfl2:amd64. 2024-03-28T14:06:22.733 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../07-libfl2_2.6.4-8build2_amd64.deb ... 2024-03-28T14:06:22.764 INFO:teuthology.orchestra.run.smithi171.stdout:Selecting previously unselected package postgresql-client-common. 2024-03-28T14:06:22.772 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking libfl2:amd64 (2.6.4-8build2) ... 2024-03-28T14:06:22.793 INFO:teuthology.orchestra.run.smithi171.stdout:Preparing to unpack .../26-postgresql-client-common_238_all.deb ... 2024-03-28T14:06:22.831 INFO:teuthology.orchestra.run.smithi171.stdout:Unpacking postgresql-client-common (238) ... 2024-03-28T14:06:23.033 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package libfl-dev:amd64. 2024-03-28T14:06:23.061 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../08-libfl-dev_2.6.4-8build2_amd64.deb ... 2024-03-28T14:06:23.099 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking libfl-dev:amd64 (2.6.4-8build2) ... 2024-03-28T14:06:23.128 INFO:teuthology.orchestra.run.smithi171.stdout:Selecting previously unselected package postgresql-client-14. 2024-03-28T14:06:23.156 INFO:teuthology.orchestra.run.smithi171.stdout:Preparing to unpack .../27-postgresql-client-14_14.11-0ubuntu0.22.04.1_amd64.deb ... 2024-03-28T14:06:23.194 INFO:teuthology.orchestra.run.smithi171.stdout:Unpacking postgresql-client-14 (14.11-0ubuntu0.22.04.1) ... 2024-03-28T14:06:23.354 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package libjson-perl. 2024-03-28T14:06:23.380 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../09-libjson-perl_4.04000-1_all.deb ... 2024-03-28T14:06:23.418 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking libjson-perl (4.04000-1) ... 2024-03-28T14:06:23.747 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package libllvm14:amd64. 2024-03-28T14:06:23.775 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../10-libllvm14_1%3a14.0.0-1ubuntu1.1_amd64.deb ... 2024-03-28T14:06:23.792 INFO:teuthology.orchestra.run.smithi171.stdout:Selecting previously unselected package postgresql-common. 2024-03-28T14:06:23.805 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking libllvm14:amd64 (1:14.0.0-1ubuntu1.1) ... 2024-03-28T14:06:23.820 INFO:teuthology.orchestra.run.smithi171.stdout:Preparing to unpack .../28-postgresql-common_238_all.deb ... 2024-03-28T14:06:24.113 INFO:teuthology.orchestra.run.smithi171.stdout:Adding 'diversion of /usr/bin/pg_config to /usr/bin/pg_config.libpq-dev by postgresql-common' 2024-03-28T14:06:24.540 INFO:teuthology.orchestra.run.smithi171.stdout:Unpacking postgresql-common (238) ... 2024-03-28T14:06:25.009 INFO:teuthology.orchestra.run.smithi171.stdout:Selecting previously unselected package postgresql-14. 2024-03-28T14:06:25.037 INFO:teuthology.orchestra.run.smithi171.stdout:Preparing to unpack .../29-postgresql-14_14.11-0ubuntu0.22.04.1_amd64.deb ... 2024-03-28T14:06:25.088 INFO:teuthology.orchestra.run.smithi171.stdout:Unpacking postgresql-14 (14.11-0ubuntu0.22.04.1) ... 2024-03-28T14:06:25.475 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package libmbim-glib4:amd64. 2024-03-28T14:06:25.502 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../11-libmbim-glib4_1.28.0-1~ubuntu20.04.1_amd64.deb ... 2024-03-28T14:06:25.541 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking libmbim-glib4:amd64 (1.28.0-1~ubuntu20.04.1) ... 2024-03-28T14:06:25.802 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package libmbim-proxy. 2024-03-28T14:06:25.830 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../12-libmbim-proxy_1.28.0-1~ubuntu20.04.1_amd64.deb ... 2024-03-28T14:06:25.868 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking libmbim-proxy (1.28.0-1~ubuntu20.04.1) ... 2024-03-28T14:06:26.196 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package libmm-glib0:amd64. 2024-03-28T14:06:26.224 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../13-libmm-glib0_1.20.0-1~ubuntu22.04.3_amd64.deb ... 2024-03-28T14:06:26.263 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking libmm-glib0:amd64 (1.20.0-1~ubuntu22.04.3) ... 2024-03-28T14:06:26.390 INFO:teuthology.orchestra.run.smithi171.stdout:Selecting previously unselected package postgresql. 2024-03-28T14:06:26.421 INFO:teuthology.orchestra.run.smithi171.stdout:Preparing to unpack .../30-postgresql_14+238_all.deb ... 2024-03-28T14:06:26.457 INFO:teuthology.orchestra.run.smithi171.stdout:Unpacking postgresql (14+238) ... 2024-03-28T14:06:26.557 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package libndp0:amd64. 2024-03-28T14:06:26.585 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../14-libndp0_1.8-0ubuntu3_amd64.deb ... 2024-03-28T14:06:26.632 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking libndp0:amd64 (1.8-0ubuntu3) ... 2024-03-28T14:06:26.693 INFO:teuthology.orchestra.run.smithi171.stdout:Selecting previously unselected package postgresql-client. 2024-03-28T14:06:26.709 INFO:teuthology.orchestra.run.smithi171.stdout:Preparing to unpack .../31-postgresql-client_14+238_all.deb ... 2024-03-28T14:06:26.743 INFO:teuthology.orchestra.run.smithi171.stdout:Unpacking postgresql-client (14+238) ... 2024-03-28T14:06:26.937 INFO:teuthology.orchestra.run.smithi171.stdout:Selecting previously unselected package postgresql-contrib. 2024-03-28T14:06:26.953 INFO:teuthology.orchestra.run.smithi171.stdout:Preparing to unpack .../32-postgresql-contrib_14+238_all.deb ... 2024-03-28T14:06:26.968 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package libnm0:amd64. 2024-03-28T14:06:26.995 INFO:teuthology.orchestra.run.smithi171.stdout:Unpacking postgresql-contrib (14+238) ... 2024-03-28T14:06:26.996 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../15-libnm0_1.36.6-0ubuntu2_amd64.deb ... 2024-03-28T14:06:27.035 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking libnm0:amd64 (1.36.6-0ubuntu2) ... 2024-03-28T14:06:27.316 INFO:teuthology.orchestra.run.smithi171.stdout:Selecting previously unselected package tcl8.6. 2024-03-28T14:06:27.344 INFO:teuthology.orchestra.run.smithi171.stdout:Preparing to unpack .../33-tcl8.6_8.6.12+dfsg-1build1_amd64.deb ... 2024-03-28T14:06:27.382 INFO:teuthology.orchestra.run.smithi171.stdout:Unpacking tcl8.6 (8.6.12+dfsg-1build1) ... 2024-03-28T14:06:27.396 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package libpq5:amd64. 2024-03-28T14:06:27.424 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../16-libpq5_14.11-0ubuntu0.22.04.1_amd64.deb ... 2024-03-28T14:06:27.463 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking libpq5:amd64 (14.11-0ubuntu0.22.04.1) ... 2024-03-28T14:06:27.669 INFO:teuthology.orchestra.run.smithi171.stdout:Selecting previously unselected package tcl. 2024-03-28T14:06:27.697 INFO:teuthology.orchestra.run.smithi171.stdout:Preparing to unpack .../34-tcl_8.6.11+1build2_amd64.deb ... 2024-03-28T14:06:27.732 INFO:teuthology.orchestra.run.smithi171.stdout:Unpacking tcl (8.6.11+1build2) ... 2024-03-28T14:06:27.799 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package libqmi-glib5:amd64. 2024-03-28T14:06:27.827 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../17-libqmi-glib5_1.32.0-1ubuntu0.22.04.1_amd64.deb ... 2024-03-28T14:06:27.866 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking libqmi-glib5:amd64 (1.32.0-1ubuntu0.22.04.1) ... 2024-03-28T14:06:28.081 INFO:teuthology.orchestra.run.smithi171.stdout:Selecting previously unselected package wpasupplicant. 2024-03-28T14:06:28.109 INFO:teuthology.orchestra.run.smithi171.stdout:Preparing to unpack .../35-wpasupplicant_2%3a2.10-6ubuntu2_amd64.deb ... 2024-03-28T14:06:28.147 INFO:teuthology.orchestra.run.smithi171.stdout:Unpacking wpasupplicant (2:2.10-6ubuntu2) ... 2024-03-28T14:06:28.159 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package libqmi-proxy. 2024-03-28T14:06:28.187 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../18-libqmi-proxy_1.32.0-1ubuntu0.22.04.1_amd64.deb ... 2024-03-28T14:06:28.226 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking libqmi-proxy (1.32.0-1ubuntu0.22.04.1) ... 2024-03-28T14:06:28.503 INFO:teuthology.orchestra.run.smithi171.stdout:Selecting previously unselected package usb-modeswitch-data. 2024-03-28T14:06:28.512 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package libssl-dev:amd64. 2024-03-28T14:06:28.531 INFO:teuthology.orchestra.run.smithi171.stdout:Preparing to unpack .../36-usb-modeswitch-data_20191128-4_all.deb ... 2024-03-28T14:06:28.539 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../19-libssl-dev_3.0.2-0ubuntu1.15_amd64.deb ... 2024-03-28T14:06:28.569 INFO:teuthology.orchestra.run.smithi171.stdout:Unpacking usb-modeswitch-data (20191128-4) ... 2024-03-28T14:06:28.578 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking libssl-dev:amd64 (3.0.2-0ubuntu1.15) ... 2024-03-28T14:06:28.949 INFO:teuthology.orchestra.run.smithi171.stdout:Selecting previously unselected package usb-modeswitch. 2024-03-28T14:06:28.976 INFO:teuthology.orchestra.run.smithi171.stdout:Preparing to unpack .../37-usb-modeswitch_2.6.1-3ubuntu2_amd64.deb ... 2024-03-28T14:06:29.024 INFO:teuthology.orchestra.run.smithi171.stdout:Unpacking usb-modeswitch (2.6.1-3ubuntu2) ... 2024-03-28T14:06:29.090 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package libteamdctl0:amd64. 2024-03-28T14:06:29.118 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../20-libteamdctl0_1.31-1build2_amd64.deb ... 2024-03-28T14:06:29.157 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking libteamdctl0:amd64 (1.31-1build2) ... 2024-03-28T14:06:29.578 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package modemmanager. 2024-03-28T14:06:29.606 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../21-modemmanager_1.20.0-1~ubuntu22.04.3_amd64.deb ... 2024-03-28T14:06:29.652 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking modemmanager (1.20.0-1~ubuntu22.04.3) ... 2024-03-28T14:06:29.672 INFO:teuthology.orchestra.run.smithi171.stdout:Setting up postgresql-client-common (238) ... 2024-03-28T14:06:29.881 INFO:teuthology.orchestra.run.smithi171.stdout:Setting up flex (2.6.4-8build2) ... 2024-03-28T14:06:29.983 INFO:teuthology.orchestra.run.smithi171.stdout:Setting up tcl8.6 (8.6.12+dfsg-1build1) ... 2024-03-28T14:06:30.084 INFO:teuthology.orchestra.run.smithi171.stdout:Setting up libpq5:amd64 (14.11-0ubuntu0.22.04.1) ... 2024-03-28T14:06:30.210 INFO:teuthology.orchestra.run.smithi171.stdout:Setting up liblzo2-2:amd64 (2.10-2build3) ... 2024-03-28T14:06:30.292 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package network-manager. 2024-03-28T14:06:30.320 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../22-network-manager_1.36.6-0ubuntu2_amd64.deb ... 2024-03-28T14:06:30.336 INFO:teuthology.orchestra.run.smithi171.stdout:Setting up libaio-dev:amd64 (0.3.112-13build1) ... 2024-03-28T14:06:30.367 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking network-manager (1.36.6-0ubuntu2) ... 2024-03-28T14:06:30.463 INFO:teuthology.orchestra.run.smithi171.stdout:Setting up postgresql-client-14 (14.11-0ubuntu0.22.04.1) ... 2024-03-28T14:06:30.943 INFO:teuthology.orchestra.run.smithi171.stdout:update-alternatives: using /usr/share/postgresql/14/man/man1/psql.1.gz to provide /usr/share/man/man1/psql.1.gz (psql.1.gz) in auto mode 2024-03-28T14:06:30.999 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package ppp. 2024-03-28T14:06:31.027 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../23-ppp_2.4.9-1+1ubuntu3_amd64.deb ... 2024-03-28T14:06:31.028 INFO:teuthology.orchestra.run.smithi171.stdout:Setting up wpasupplicant (2:2.10-6ubuntu2) ... 2024-03-28T14:06:31.296 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking ppp (2.4.9-1+1ubuntu3) ... 2024-03-28T14:06:31.377 INFO:teuthology.orchestra.run.smithi171.stdout:Created symlink /etc/systemd/system/dbus-fi.w1.wpa_supplicant1.service -> /lib/systemd/system/wpa_supplicant.service. 2024-03-28T14:06:31.377 INFO:teuthology.orchestra.run.smithi171.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/wpa_supplicant.service -> /lib/systemd/system/wpa_supplicant.service. 2024-03-28T14:06:31.629 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package pptp-linux. 2024-03-28T14:06:31.657 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../24-pptp-linux_1.10.0-1build3_amd64.deb ... 2024-03-28T14:06:31.695 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking pptp-linux (1.10.0-1build3) ... 2024-03-28T14:06:31.898 INFO:teuthology.orchestra.run.smithi171.stdout:Setting up uuid-dev:amd64 (2.37.2-4ubuntu3.3) ... 2024-03-28T14:06:31.948 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package network-manager-pptp. 2024-03-28T14:06:31.964 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../25-network-manager-pptp_1.2.10-1_amd64.deb ... 2024-03-28T14:06:32.015 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking network-manager-pptp (1.2.10-1) ... 2024-03-28T14:06:32.025 INFO:teuthology.orchestra.run.smithi171.stdout:Setting up libteamdctl0:amd64 (1.31-1build2) ... 2024-03-28T14:06:32.143 INFO:teuthology.orchestra.run.smithi171.stdout:Setting up libssl-dev:amd64 (3.0.2-0ubuntu1.15) ... 2024-03-28T14:06:32.261 INFO:teuthology.orchestra.run.smithi171.stdout:Setting up usb-modeswitch-data (20191128-4) ... 2024-03-28T14:06:32.343 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package postgresql-client-common. 2024-03-28T14:06:32.370 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../26-postgresql-client-common_238_all.deb ... 2024-03-28T14:06:32.387 INFO:teuthology.orchestra.run.smithi171.stdout:Setting up libelf-dev:amd64 (0.186-1build1) ... 2024-03-28T14:06:32.409 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking postgresql-client-common (238) ... 2024-03-28T14:06:32.513 INFO:teuthology.orchestra.run.smithi171.stdout:Setting up libfl2:amd64 (2.6.4-8build2) ... 2024-03-28T14:06:32.640 INFO:teuthology.orchestra.run.smithi171.stdout:Setting up libnm0:amd64 (1.36.6-0ubuntu2) ... 2024-03-28T14:06:32.696 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package postgresql-client-14. 2024-03-28T14:06:32.724 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../27-postgresql-client-14_14.11-0ubuntu0.22.04.1_amd64.deb ... 2024-03-28T14:06:32.767 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking postgresql-client-14 (14.11-0ubuntu0.22.04.1) ... 2024-03-28T14:06:32.768 INFO:teuthology.orchestra.run.smithi171.stdout:Setting up ppp (2.4.9-1+1ubuntu3) ... 2024-03-28T14:06:33.359 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package postgresql-common. 2024-03-28T14:06:33.387 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../28-postgresql-common_238_all.deb ... 2024-03-28T14:06:33.443 INFO:teuthology.orchestra.run.smithi104.stdout:Adding 'diversion of /usr/bin/pg_config to /usr/bin/pg_config.libpq-dev by postgresql-common' 2024-03-28T14:06:33.473 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking postgresql-common (238) ... 2024-03-28T14:06:33.517 INFO:teuthology.orchestra.run.smithi171.stdout:Setting up libllvm14:amd64 (1:14.0.0-1ubuntu1.1) ... 2024-03-28T14:06:33.635 INFO:teuthology.orchestra.run.smithi171.stdout:Setting up dump (0.4b47-3) ... 2024-03-28T14:06:33.712 INFO:teuthology.orchestra.run.smithi171.stdout:update-alternatives: using /usr/sbin/rmt-dump to provide /usr/sbin/rmt (rmt) in auto mode 2024-03-28T14:06:33.778 INFO:teuthology.orchestra.run.smithi171.stdout:Setting up libmm-glib0:amd64 (1.20.0-1~ubuntu22.04.3) ... 2024-03-28T14:06:33.882 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package postgresql-14. 2024-03-28T14:06:33.905 INFO:teuthology.orchestra.run.smithi171.stdout:Setting up libbluetooth3:amd64 (5.64-0ubuntu1.1) ... 2024-03-28T14:06:33.910 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../29-postgresql-14_14.11-0ubuntu0.22.04.1_amd64.deb ... 2024-03-28T14:06:33.943 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking postgresql-14 (14.11-0ubuntu0.22.04.1) ... 2024-03-28T14:06:34.048 INFO:teuthology.orchestra.run.smithi171.stdout:Setting up tcl (8.6.11+1build2) ... 2024-03-28T14:06:34.174 INFO:teuthology.orchestra.run.smithi171.stdout:Setting up libjson-perl (4.04000-1) ... 2024-03-28T14:06:34.300 INFO:teuthology.orchestra.run.smithi171.stdout:Setting up bison (2:3.8.2+dfsg-1build1) ... 2024-03-28T14:06:34.382 INFO:teuthology.orchestra.run.smithi171.stdout:update-alternatives: using /usr/bin/bison.yacc to provide /usr/bin/yacc (yacc) in auto mode 2024-03-28T14:06:34.444 INFO:teuthology.orchestra.run.smithi171.stdout:Setting up indent (2.2.12-1ubuntu0.22.04.1) ... 2024-03-28T14:06:34.572 INFO:teuthology.orchestra.run.smithi171.stdout:Setting up usb-modeswitch (2.6.1-3ubuntu2) ... 2024-03-28T14:06:34.747 INFO:teuthology.orchestra.run.smithi171.stdout:Setting up libndp0:amd64 (1.8-0ubuntu3) ... 2024-03-28T14:06:34.873 INFO:teuthology.orchestra.run.smithi171.stdout:Setting up libmbim-glib4:amd64 (1.28.0-1~ubuntu20.04.1) ... 2024-03-28T14:06:35.000 INFO:teuthology.orchestra.run.smithi171.stdout:Setting up pptp-linux (1.10.0-1build3) ... 2024-03-28T14:06:35.160 INFO:teuthology.orchestra.run.smithi171.stdout:Setting up postgresql-client (14+238) ... 2024-03-28T14:06:35.261 INFO:teuthology.orchestra.run.smithi171.stdout:Setting up network-manager (1.36.6-0ubuntu2) ... 2024-03-28T14:06:35.427 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package postgresql. 2024-03-28T14:06:35.453 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../30-postgresql_14+238_all.deb ... 2024-03-28T14:06:35.493 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking postgresql (14+238) ... 2024-03-28T14:06:35.746 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-03-28T14:06:35.746 INFO:teuthology.orchestra.run.smithi171.stdout:The following network interfaces were found in /etc/network/interfaces 2024-03-28T14:06:35.746 INFO:teuthology.orchestra.run.smithi171.stdout:which means they are currently configured by ifupdown: 2024-03-28T14:06:35.746 INFO:teuthology.orchestra.run.smithi171.stdout:- enp3s0f1 2024-03-28T14:06:35.746 INFO:teuthology.orchestra.run.smithi171.stdout:If you want to manage those interfaces with NetworkManager instead 2024-03-28T14:06:35.746 INFO:teuthology.orchestra.run.smithi171.stdout:remove their configuration from /etc/network/interfaces. 2024-03-28T14:06:35.746 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-03-28T14:06:35.747 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package postgresql-client. 2024-03-28T14:06:35.774 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../31-postgresql-client_14+238_all.deb ... 2024-03-28T14:06:35.812 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking postgresql-client (14+238) ... 2024-03-28T14:06:35.873 INFO:teuthology.orchestra.run.smithi171.stdout:Created symlink /etc/systemd/system/dbus-org.freedesktop.nm-dispatcher.service -> /lib/systemd/system/NetworkManager-dispatcher.service. 2024-03-28T14:06:36.056 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package postgresql-contrib. 2024-03-28T14:06:36.074 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../32-postgresql-contrib_14+238_all.deb ... 2024-03-28T14:06:36.345 INFO:teuthology.orchestra.run.smithi171.stdout:Created symlink /etc/systemd/system/network-online.target.wants/NetworkManager-wait-online.service -> /lib/systemd/system/NetworkManager-wait-online.service. 2024-03-28T14:06:36.345 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking postgresql-contrib (14+238) ... 2024-03-28T14:06:36.400 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package tcl8.6. 2024-03-28T14:06:36.428 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../33-tcl8.6_8.6.12+dfsg-1build1_amd64.deb ... 2024-03-28T14:06:36.466 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking tcl8.6 (8.6.12+dfsg-1build1) ... 2024-03-28T14:06:36.560 INFO:teuthology.orchestra.run.smithi171.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/NetworkManager.service -> /lib/systemd/system/NetworkManager.service. 2024-03-28T14:06:36.719 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package tcl. 2024-03-28T14:06:36.747 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../34-tcl_8.6.11+1build2_amd64.deb ... 2024-03-28T14:06:36.791 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking tcl (8.6.11+1build2) ... 2024-03-28T14:06:37.065 INFO:teuthology.orchestra.run.smithi171.stdout:Setting up libmbim-proxy (1.28.0-1~ubuntu20.04.1) ... 2024-03-28T14:06:37.130 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package wpasupplicant. 2024-03-28T14:06:37.158 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../35-wpasupplicant_2%3a2.10-6ubuntu2_amd64.deb ... 2024-03-28T14:06:37.196 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking wpasupplicant (2:2.10-6ubuntu2) ... 2024-03-28T14:06:37.224 INFO:teuthology.orchestra.run.smithi171.stdout:Setting up libfl-dev:amd64 (2.6.4-8build2) ... 2024-03-28T14:06:37.342 INFO:teuthology.orchestra.run.smithi171.stdout:Setting up network-manager-pptp (1.2.10-1) ... 2024-03-28T14:06:37.452 INFO:teuthology.orchestra.run.smithi171.stdout:Setting up postgresql-common (238) ... 2024-03-28T14:06:37.592 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package usb-modeswitch-data. 2024-03-28T14:06:37.621 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../36-usb-modeswitch-data_20191128-4_all.deb ... 2024-03-28T14:06:37.658 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking usb-modeswitch-data (20191128-4) ... 2024-03-28T14:06:37.995 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package usb-modeswitch. 2024-03-28T14:06:38.023 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../37-usb-modeswitch_2.6.1-3ubuntu2_amd64.deb ... 2024-03-28T14:06:38.061 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking usb-modeswitch (2.6.1-3ubuntu2) ... 2024-03-28T14:06:38.332 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up postgresql-client-common (238) ... 2024-03-28T14:06:38.336 INFO:teuthology.orchestra.run.smithi171.stdout:Adding user postgres to group ssl-cert 2024-03-28T14:06:38.542 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up flex (2.6.4-8build2) ... 2024-03-28T14:06:38.559 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-03-28T14:06:38.559 INFO:teuthology.orchestra.run.smithi171.stdout:Creating config file /etc/postgresql-common/createcluster.conf with new version 2024-03-28T14:06:38.668 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up tcl8.6 (8.6.12+dfsg-1build1) ... 2024-03-28T14:06:38.737 INFO:teuthology.orchestra.run.smithi171.stdout:Building PostgreSQL dictionaries from installed myspell/hunspell packages... 2024-03-28T14:06:38.737 INFO:teuthology.orchestra.run.smithi171.stdout:Removing obsolete dictionary files: 2024-03-28T14:06:38.819 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up libpq5:amd64 (14.11-0ubuntu0.22.04.1) ... 2024-03-28T14:06:38.945 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up liblzo2-2:amd64 (2.10-2build3) ... 2024-03-28T14:06:39.071 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up libaio-dev:amd64 (0.3.112-13build1) ... 2024-03-28T14:06:39.197 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up postgresql-client-14 (14.11-0ubuntu0.22.04.1) ... 2024-03-28T14:06:39.300 INFO:teuthology.orchestra.run.smithi171.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/postgresql.service -> /lib/systemd/system/postgresql.service. 2024-03-28T14:06:39.668 INFO:teuthology.orchestra.run.smithi104.stdout:update-alternatives: using /usr/share/postgresql/14/man/man1/psql.1.gz to provide /usr/share/man/man1/psql.1.gz (psql.1.gz) in auto mode 2024-03-28T14:06:39.673 INFO:teuthology.orchestra.run.smithi171.stdout:Setting up postgresql-14 (14.11-0ubuntu0.22.04.1) ... 2024-03-28T14:06:39.761 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up wpasupplicant (2:2.10-6ubuntu2) ... 2024-03-28T14:06:39.986 INFO:teuthology.orchestra.run.smithi171.stdout:Creating new PostgreSQL cluster 14/main ... 2024-03-28T14:06:40.008 INFO:teuthology.orchestra.run.smithi171.stdout:/usr/lib/postgresql/14/bin/initdb -D /var/lib/postgresql/14/main --auth-local peer --auth-host scram-sha-256 --no-instructions 2024-03-28T14:06:40.065 INFO:teuthology.orchestra.run.smithi171.stdout:The files belonging to this database system will be owned by user "postgres". 2024-03-28T14:06:40.065 INFO:teuthology.orchestra.run.smithi171.stdout:This user must also own the server process. 2024-03-28T14:06:40.065 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-03-28T14:06:40.065 INFO:teuthology.orchestra.run.smithi171.stdout:The database cluster will be initialized with locale "en_US". 2024-03-28T14:06:40.065 INFO:teuthology.orchestra.run.smithi171.stdout:The default database encoding has accordingly been set to "LATIN1". 2024-03-28T14:06:40.065 INFO:teuthology.orchestra.run.smithi171.stdout:The default text search configuration will be set to "english". 2024-03-28T14:06:40.065 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-03-28T14:06:40.065 INFO:teuthology.orchestra.run.smithi171.stdout:Data page checksums are disabled. 2024-03-28T14:06:40.065 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-03-28T14:06:40.066 INFO:teuthology.orchestra.run.smithi171.stdout:fixing permissions on existing directory /var/lib/postgresql/14/main ... ok 2024-03-28T14:06:40.066 INFO:teuthology.orchestra.run.smithi171.stdout:creating subdirectories ... ok 2024-03-28T14:06:40.066 INFO:teuthology.orchestra.run.smithi171.stdout:selecting dynamic shared memory implementation ... posix 2024-03-28T14:06:40.081 INFO:teuthology.orchestra.run.smithi171.stdout:selecting default max_connections ... 100 2024-03-28T14:06:40.101 INFO:teuthology.orchestra.run.smithi171.stdout:selecting default shared_buffers ... 128MB 2024-03-28T14:06:40.110 INFO:teuthology.orchestra.run.smithi171.stdout:selecting default time zone ... Etc/UTC 2024-03-28T14:06:40.117 INFO:teuthology.orchestra.run.smithi171.stdout:creating configuration files ... ok 2024-03-28T14:06:40.146 INFO:teuthology.orchestra.run.smithi104.stdout:Created symlink /etc/systemd/system/dbus-fi.w1.wpa_supplicant1.service -> /lib/systemd/system/wpa_supplicant.service. 2024-03-28T14:06:40.147 INFO:teuthology.orchestra.run.smithi104.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/wpa_supplicant.service -> /lib/systemd/system/wpa_supplicant.service. 2024-03-28T14:06:40.228 INFO:teuthology.orchestra.run.smithi171.stdout:running bootstrap script ... ok 2024-03-28T14:06:40.639 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up uuid-dev:amd64 (2.37.2-4ubuntu3.3) ... 2024-03-28T14:06:40.765 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up libteamdctl0:amd64 (1.31-1build2) ... 2024-03-28T14:06:40.867 INFO:teuthology.orchestra.run.smithi171.stdout:performing post-bootstrap initialization ... ok 2024-03-28T14:06:40.891 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up libssl-dev:amd64 (3.0.2-0ubuntu1.15) ... 2024-03-28T14:06:41.033 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up usb-modeswitch-data (20191128-4) ... 2024-03-28T14:06:41.159 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up libelf-dev:amd64 (0.186-1build1) ... 2024-03-28T14:06:41.294 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up libfl2:amd64 (2.6.4-8build2) ... 2024-03-28T14:06:41.444 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up libnm0:amd64 (1.36.6-0ubuntu2) ... 2024-03-28T14:06:41.579 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up ppp (2.4.9-1+1ubuntu3) ... 2024-03-28T14:06:41.796 INFO:teuthology.orchestra.run.smithi171.stdout:syncing data to disk ... ok 2024-03-28T14:06:42.191 INFO:teuthology.orchestra.run.smithi171.stdout:update-alternatives: using /usr/share/postgresql/14/man/man1/postmaster.1.gz to provide /usr/share/man/man1/postmaster.1.gz (postmaster.1.gz) in auto mode 2024-03-28T14:06:42.234 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up libllvm14:amd64 (1:14.0.0-1ubuntu1.1) ... 2024-03-28T14:06:42.360 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up dump (0.4b47-3) ... 2024-03-28T14:06:42.539 INFO:teuthology.orchestra.run.smithi104.stdout:update-alternatives: using /usr/sbin/rmt-dump to provide /usr/sbin/rmt (rmt) in auto mode 2024-03-28T14:06:42.678 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up libmm-glib0:amd64 (1.20.0-1~ubuntu22.04.3) ... 2024-03-28T14:06:42.813 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up libbluetooth3:amd64 (5.64-0ubuntu1.1) ... 2024-03-28T14:06:42.939 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up tcl (8.6.11+1build2) ... 2024-03-28T14:06:43.065 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up libjson-perl (4.04000-1) ... 2024-03-28T14:06:43.191 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up bison (2:3.8.2+dfsg-1build1) ... 2024-03-28T14:06:43.281 INFO:teuthology.orchestra.run.smithi104.stdout:update-alternatives: using /usr/bin/bison.yacc to provide /usr/bin/yacc (yacc) in auto mode 2024-03-28T14:06:43.350 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up indent (2.2.12-1ubuntu0.22.04.1) ... 2024-03-28T14:06:43.477 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up usb-modeswitch (2.6.1-3ubuntu2) ... 2024-03-28T14:06:43.661 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up libndp0:amd64 (1.8-0ubuntu3) ... 2024-03-28T14:06:43.820 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up libmbim-glib4:amd64 (1.28.0-1~ubuntu20.04.1) ... 2024-03-28T14:06:43.946 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up pptp-linux (1.10.0-1build3) ... 2024-03-28T14:06:44.114 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up postgresql-client (14+238) ... 2024-03-28T14:06:44.240 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up network-manager (1.36.6-0ubuntu2) ... 2024-03-28T14:06:44.703 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-03-28T14:06:44.703 INFO:teuthology.orchestra.run.smithi104.stdout:The following network interfaces were found in /etc/network/interfaces 2024-03-28T14:06:44.704 INFO:teuthology.orchestra.run.smithi104.stdout:which means they are currently configured by ifupdown: 2024-03-28T14:06:44.704 INFO:teuthology.orchestra.run.smithi104.stdout:- enp3s0f1 2024-03-28T14:06:44.704 INFO:teuthology.orchestra.run.smithi104.stdout:If you want to manage those interfaces with NetworkManager instead 2024-03-28T14:06:44.704 INFO:teuthology.orchestra.run.smithi104.stdout:remove their configuration from /etc/network/interfaces. 2024-03-28T14:06:44.704 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-03-28T14:06:44.829 INFO:teuthology.orchestra.run.smithi104.stdout:Created symlink /etc/systemd/system/dbus-org.freedesktop.nm-dispatcher.service -> /lib/systemd/system/NetworkManager-dispatcher.service. 2024-03-28T14:06:44.976 INFO:teuthology.orchestra.run.smithi171.stdout:Setting up libqmi-glib5:amd64 (1.32.0-1ubuntu0.22.04.1) ... 2024-03-28T14:06:45.119 INFO:teuthology.orchestra.run.smithi104.stdout:Created symlink /etc/systemd/system/network-online.target.wants/NetworkManager-wait-online.service -> /lib/systemd/system/NetworkManager-wait-online.service. 2024-03-28T14:06:45.127 INFO:teuthology.orchestra.run.smithi171.stdout:Setting up postgresql-contrib (14+238) ... 2024-03-28T14:06:45.262 INFO:teuthology.orchestra.run.smithi171.stdout:Setting up postgresql (14+238) ... 2024-03-28T14:06:45.388 INFO:teuthology.orchestra.run.smithi171.stdout:Setting up libqmi-proxy (1.32.0-1ubuntu0.22.04.1) ... 2024-03-28T14:06:45.426 INFO:teuthology.orchestra.run.smithi104.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/NetworkManager.service -> /lib/systemd/system/NetworkManager.service. 2024-03-28T14:06:45.522 INFO:teuthology.orchestra.run.smithi171.stdout:Setting up modemmanager (1.20.0-1~ubuntu22.04.3) ... 2024-03-28T14:06:45.801 INFO:teuthology.orchestra.run.smithi171.stdout:Created symlink /etc/systemd/system/dbus-org.freedesktop.ModemManager1.service -> /lib/systemd/system/ModemManager.service. 2024-03-28T14:06:45.801 INFO:teuthology.orchestra.run.smithi171.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ModemManager.service -> /lib/systemd/system/ModemManager.service. 2024-03-28T14:06:45.901 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up libmbim-proxy (1.28.0-1~ubuntu20.04.1) ... 2024-03-28T14:06:46.052 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up libfl-dev:amd64 (2.6.4-8build2) ... 2024-03-28T14:06:46.187 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up network-manager-pptp (1.2.10-1) ... 2024-03-28T14:06:46.313 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up postgresql-common (238) ... 2024-03-28T14:06:46.376 INFO:teuthology.orchestra.run.smithi171.stdout:Processing triggers for mailcap (3.70+nmu1ubuntu1) ... 2024-03-28T14:06:46.485 INFO:teuthology.orchestra.run.smithi171.stdout:Processing triggers for hicolor-icon-theme (0.17-2) ... 2024-03-28T14:06:46.805 INFO:teuthology.orchestra.run.smithi171.stdout:Processing triggers for libc-bin (2.35-0ubuntu3.1) ... 2024-03-28T14:06:46.965 INFO:teuthology.orchestra.run.smithi171.stdout:Processing triggers for man-db (2.10.2-1) ... 2024-03-28T14:06:47.243 INFO:teuthology.orchestra.run.smithi104.stdout:Adding user postgres to group ssl-cert 2024-03-28T14:06:47.483 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-03-28T14:06:47.483 INFO:teuthology.orchestra.run.smithi104.stdout:Creating config file /etc/postgresql-common/createcluster.conf with new version 2024-03-28T14:06:47.690 INFO:teuthology.orchestra.run.smithi104.stdout:Building PostgreSQL dictionaries from installed myspell/hunspell packages... 2024-03-28T14:06:47.690 INFO:teuthology.orchestra.run.smithi104.stdout:Removing obsolete dictionary files: 2024-03-28T14:06:48.242 INFO:teuthology.orchestra.run.smithi104.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/postgresql.service -> /lib/systemd/system/postgresql.service. 2024-03-28T14:06:48.354 INFO:teuthology.orchestra.run.smithi171.stdout:Processing triggers for dbus (1.12.20-2ubuntu4.1) ... 2024-03-28T14:06:48.455 INFO:teuthology.orchestra.run.smithi171.stdout:Processing triggers for install-info (6.8-4build1) ... 2024-03-28T14:06:48.622 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up postgresql-14 (14.11-0ubuntu0.22.04.1) ... 2024-03-28T14:06:48.932 INFO:teuthology.orchestra.run.smithi104.stdout:Creating new PostgreSQL cluster 14/main ... 2024-03-28T14:06:48.947 INFO:teuthology.orchestra.run.smithi104.stdout:/usr/lib/postgresql/14/bin/initdb -D /var/lib/postgresql/14/main --auth-local peer --auth-host scram-sha-256 --no-instructions 2024-03-28T14:06:48.963 INFO:teuthology.orchestra.run.smithi171.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2024-03-28T14:06:48.969 DEBUG:teuthology.parallel:result is None 2024-03-28T14:06:49.006 INFO:teuthology.orchestra.run.smithi104.stdout:The files belonging to this database system will be owned by user "postgres". 2024-03-28T14:06:49.006 INFO:teuthology.orchestra.run.smithi104.stdout:This user must also own the server process. 2024-03-28T14:06:49.006 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-03-28T14:06:49.006 INFO:teuthology.orchestra.run.smithi104.stdout:The database cluster will be initialized with locale "en_US". 2024-03-28T14:06:49.006 INFO:teuthology.orchestra.run.smithi104.stdout:The default database encoding has accordingly been set to "LATIN1". 2024-03-28T14:06:49.006 INFO:teuthology.orchestra.run.smithi104.stdout:The default text search configuration will be set to "english". 2024-03-28T14:06:49.006 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-03-28T14:06:49.006 INFO:teuthology.orchestra.run.smithi104.stdout:Data page checksums are disabled. 2024-03-28T14:06:49.006 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-03-28T14:06:49.006 INFO:teuthology.orchestra.run.smithi104.stdout:fixing permissions on existing directory /var/lib/postgresql/14/main ... ok 2024-03-28T14:06:49.006 INFO:teuthology.orchestra.run.smithi104.stdout:creating subdirectories ... ok 2024-03-28T14:06:49.007 INFO:teuthology.orchestra.run.smithi104.stdout:selecting dynamic shared memory implementation ... posix 2024-03-28T14:06:49.022 INFO:teuthology.orchestra.run.smithi104.stdout:selecting default max_connections ... 100 2024-03-28T14:06:49.041 INFO:teuthology.orchestra.run.smithi104.stdout:selecting default shared_buffers ... 128MB 2024-03-28T14:06:49.052 INFO:teuthology.orchestra.run.smithi104.stdout:selecting default time zone ... Etc/UTC 2024-03-28T14:06:49.052 INFO:teuthology.orchestra.run.smithi104.stdout:creating configuration files ... ok 2024-03-28T14:06:49.167 INFO:teuthology.orchestra.run.smithi104.stdout:running bootstrap script ... ok 2024-03-28T14:06:49.781 INFO:teuthology.orchestra.run.smithi104.stdout:performing post-bootstrap initialization ... ok 2024-03-28T14:06:50.458 INFO:teuthology.orchestra.run.smithi104.stdout:syncing data to disk ... ok 2024-03-28T14:06:50.848 INFO:teuthology.orchestra.run.smithi104.stdout:update-alternatives: using /usr/share/postgresql/14/man/man1/postmaster.1.gz to provide /usr/share/man/man1/postmaster.1.gz (postmaster.1.gz) in auto mode 2024-03-28T14:06:53.603 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up libqmi-glib5:amd64 (1.32.0-1ubuntu0.22.04.1) ... 2024-03-28T14:06:53.729 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up postgresql-contrib (14+238) ... 2024-03-28T14:06:53.863 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up postgresql (14+238) ... 2024-03-28T14:06:54.039 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up libqmi-proxy (1.32.0-1ubuntu0.22.04.1) ... 2024-03-28T14:06:54.174 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up modemmanager (1.20.0-1~ubuntu22.04.3) ... 2024-03-28T14:06:54.425 INFO:teuthology.orchestra.run.smithi104.stdout:Created symlink /etc/systemd/system/dbus-org.freedesktop.ModemManager1.service -> /lib/systemd/system/ModemManager.service. 2024-03-28T14:06:54.425 INFO:teuthology.orchestra.run.smithi104.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ModemManager.service -> /lib/systemd/system/ModemManager.service. 2024-03-28T14:06:54.968 INFO:teuthology.orchestra.run.smithi104.stdout:Processing triggers for mailcap (3.70+nmu1ubuntu1) ... 2024-03-28T14:06:55.060 INFO:teuthology.orchestra.run.smithi104.stdout:Processing triggers for hicolor-icon-theme (0.17-2) ... 2024-03-28T14:06:55.387 INFO:teuthology.orchestra.run.smithi104.stdout:Processing triggers for libc-bin (2.35-0ubuntu3.1) ... 2024-03-28T14:06:55.540 INFO:teuthology.orchestra.run.smithi104.stdout:Processing triggers for man-db (2.10.2-1) ... 2024-03-28T14:06:56.876 INFO:teuthology.orchestra.run.smithi104.stdout:Processing triggers for dbus (1.12.20-2ubuntu4.1) ... 2024-03-28T14:06:56.983 INFO:teuthology.orchestra.run.smithi104.stdout:Processing triggers for install-info (6.8-4build1) ... 2024-03-28T14:06:57.473 INFO:teuthology.orchestra.run.smithi104.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2024-03-28T14:06:57.477 DEBUG:teuthology.parallel:result is None 2024-03-28T14:06:57.478 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=e0e48ba6ee8c8b4c8f38d3b879d6eafbbb1efc15 2024-03-28T14:06:57.695 DEBUG:teuthology.orchestra.run.smithi104:> dpkg-query -W -f '${Version}' ceph 2024-03-28T14:06:57.714 INFO:teuthology.orchestra.run.smithi104.stdout:19.0.0-2499-ge0e48ba6-1jammy 2024-03-28T14:06:57.715 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-2499-ge0e48ba6-1jammy 2024-03-28T14:06:57.715 INFO:teuthology.task.install:The correct ceph version 19.0.0-2499-ge0e48ba6-1jammy is installed. 2024-03-28T14:06:57.717 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=e0e48ba6ee8c8b4c8f38d3b879d6eafbbb1efc15 2024-03-28T14:06:57.865 DEBUG:teuthology.orchestra.run.smithi171:> dpkg-query -W -f '${Version}' ceph 2024-03-28T14:06:57.885 INFO:teuthology.orchestra.run.smithi171.stdout:19.0.0-2499-ge0e48ba6-1jammy 2024-03-28T14:06:57.886 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-2499-ge0e48ba6-1jammy 2024-03-28T14:06:57.886 INFO:teuthology.task.install:The correct ceph version 19.0.0-2499-ge0e48ba6-1jammy is installed. 2024-03-28T14:06:57.888 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-03-28T14:06:57.888 DEBUG:teuthology.orchestra.run.smithi104:> set -ex 2024-03-28T14:06:57.888 DEBUG:teuthology.orchestra.run.smithi104:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-28T14:06:57.902 DEBUG:teuthology.orchestra.run.smithi171:> set -ex 2024-03-28T14:06:57.903 DEBUG:teuthology.orchestra.run.smithi171:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-28T14:06:57.940 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-03-28T14:06:57.940 DEBUG:teuthology.orchestra.run.smithi104:> set -ex 2024-03-28T14:06:57.940 DEBUG:teuthology.orchestra.run.smithi104:> sudo dd of=/usr/bin/daemon-helper 2024-03-28T14:06:57.958 DEBUG:teuthology.orchestra.run.smithi104:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-28T14:06:58.014 DEBUG:teuthology.orchestra.run.smithi171:> set -ex 2024-03-28T14:06:58.014 DEBUG:teuthology.orchestra.run.smithi171:> sudo dd of=/usr/bin/daemon-helper 2024-03-28T14:06:58.027 DEBUG:teuthology.orchestra.run.smithi171:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-28T14:06:58.079 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-03-28T14:06:58.079 DEBUG:teuthology.orchestra.run.smithi104:> set -ex 2024-03-28T14:06:58.079 DEBUG:teuthology.orchestra.run.smithi104:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-28T14:06:58.092 DEBUG:teuthology.orchestra.run.smithi104:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-28T14:06:58.146 DEBUG:teuthology.orchestra.run.smithi171:> set -ex 2024-03-28T14:06:58.146 DEBUG:teuthology.orchestra.run.smithi171:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-28T14:06:58.159 DEBUG:teuthology.orchestra.run.smithi171:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-28T14:06:58.211 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-03-28T14:06:58.211 DEBUG:teuthology.orchestra.run.smithi104:> set -ex 2024-03-28T14:06:58.211 DEBUG:teuthology.orchestra.run.smithi104:> sudo dd of=/usr/bin/stdin-killer 2024-03-28T14:06:58.224 DEBUG:teuthology.orchestra.run.smithi104:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-28T14:06:58.278 DEBUG:teuthology.orchestra.run.smithi171:> set -ex 2024-03-28T14:06:58.278 DEBUG:teuthology.orchestra.run.smithi171:> sudo dd of=/usr/bin/stdin-killer 2024-03-28T14:06:58.291 DEBUG:teuthology.orchestra.run.smithi171:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-28T14:06:58.347 INFO:teuthology.run_tasks:Running task ceph... 2024-03-28T14:06:58.449 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2024-03-28T14:06:58.450 DEBUG:teuthology.orchestra.run.smithi104:> sudo chmod 777 /var/log/ceph 2024-03-28T14:06:58.452 DEBUG:teuthology.orchestra.run.smithi171:> sudo chmod 777 /var/log/ceph 2024-03-28T14:06:58.465 INFO:tasks.ceph:Disabling ceph logrotate... 2024-03-28T14:06:58.465 DEBUG:teuthology.orchestra.run.smithi104:> sudo rm -f -- /etc/logrotate.d/ceph 2024-03-28T14:06:58.508 DEBUG:teuthology.orchestra.run.smithi171:> sudo rm -f -- /etc/logrotate.d/ceph 2024-03-28T14:06:58.522 INFO:tasks.ceph:Creating extra log directories... 2024-03-28T14:06:58.522 DEBUG:teuthology.orchestra.run.smithi104:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-03-28T14:06:58.564 DEBUG:teuthology.orchestra.run.smithi171:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-03-28T14:06:58.586 INFO:tasks.ceph:Setting up log rotation with {'ceph-mds': '10G', 'ceph-osd': '10G'} 2024-03-28T14:06:58.586 INFO:tasks.ceph:writing logrotate stanza for ceph-mds 2024-03-28T14:06:58.587 INFO:tasks.ceph:writing logrotate stanza for ceph-osd 2024-03-28T14:06:58.587 DEBUG:teuthology.orchestra.run.smithi104:> set -ex 2024-03-28T14:06:58.587 DEBUG:teuthology.orchestra.run.smithi104:> dd of=/home/ubuntu/cephtest/logrotate.ceph-test.conf 2024-03-28T14:06:58.623 DEBUG:teuthology.orchestra.run.smithi104:> sudo mv /home/ubuntu/cephtest/logrotate.ceph-test.conf /etc/logrotate.d/ceph-test.conf && sudo chmod 0644 /etc/logrotate.d/ceph-test.conf && sudo chown root.root /etc/logrotate.d/ceph-test.conf 2024-03-28T14:06:58.699 DEBUG:teuthology.orchestra.run.smithi171:> set -ex 2024-03-28T14:06:58.699 DEBUG:teuthology.orchestra.run.smithi171:> dd of=/home/ubuntu/cephtest/logrotate.ceph-test.conf 2024-03-28T14:06:58.705 DEBUG:teuthology.orchestra.run.smithi171:> sudo mv /home/ubuntu/cephtest/logrotate.ceph-test.conf /etc/logrotate.d/ceph-test.conf && sudo chmod 0644 /etc/logrotate.d/ceph-test.conf && sudo chown root.root /etc/logrotate.d/ceph-test.conf 2024-03-28T14:06:58.784 INFO:tasks.ceph:Creating ceph cluster ceph... 2024-03-28T14:06:58.785 INFO:tasks.ceph:config {'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'pid file': '', 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug client': 20, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon down mkfs grace': 300, 'mon op complaint time': 120}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'FS_DEGRADED', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'MDS_UP_LESS_THAN_MAX', 'filesystem is online with fewer MDS than max_mds', 'POOL_APP_NOT_ENABLED', 'overall HEALTH_', 'Replacing daemon', 'deprecated feature inline_data', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'PG_AVAILABILITY', 'PG_DEGRADED', 'Reduced data availability'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2024-03-28T14:06:58.785 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/pdonnell-2024-03-28_07:14:07-fs-wip-batrick-testing-20240327.230800-distro-default-smithi/7627919', 'branch': 'wip-batrick-testing-20240327.230800', 'description': 'fs/shell/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1-mds-1-client-coloc conf/{client mds mgr mon osd} distro/ubuntu_latest mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile pg_health} tasks/cephfs-shell}', 'email': 'pdonnell@redhat.com', 'first_in_suite': False, 'job_id': '7627919', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'ktype': 'distro', 'last_in_suite': False, 'log-rotate': {'ceph-mds': '10G', 'ceph-osd': '10G'}, 'machine_type': 'smithi', 'mgrmodules': {'sequential': [{'print': 'Enabling mgr modules'}]}, 'name': 'pdonnell-2024-03-28_07:14:07-fs-wip-batrick-testing-20240327.230800-distro-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'openstack': [{'volumes': {'count': 4, 'size': 20}}, {'machine': {'disk': 200}}], 'os_type': 'ubuntu', 'os_version': '22.04', 'overrides': {'admin_socket': {'branch': 'wip-batrick-testing-20240327.230800'}, 'ceph': {'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'pid file': '', 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug client': 20, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon down mkfs grace': 300, 'mon op complaint time': 120}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'FS_DEGRADED', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'MDS_UP_LESS_THAN_MAX', 'filesystem is online with fewer MDS than max_mds', 'POOL_APP_NOT_ENABLED', 'overall HEALTH_', 'Replacing daemon', 'deprecated feature inline_data', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'PG_AVAILABILITY', 'PG_DEGRADED', 'Reduced data availability'], 'sha1': 'e0e48ba6ee8c8b4c8f38d3b879d6eafbbb1efc15'}, 'ceph-deploy': {'bluestore': True, 'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs'}, 'install': {'ceph': {'flavor': 'default', 'sha1': 'e0e48ba6ee8c8b4c8f38d3b879d6eafbbb1efc15'}}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'wip-batrick-testing-20240327.230800', 'sha1': 'e0e48ba6ee8c8b4c8f38d3b879d6eafbbb1efc15'}}, 'owner': 'scheduled_pdonnell@teuthology', 'priority': 75, 'repo': 'https://git.ceph.com/ceph-ci.git', 'roles': [['mon.a', 'mgr.y', 'mds.a', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0'], ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'osd.4', 'osd.5', 'osd.6', 'osd.7']], 'seed': 7752, 'sha1': 'e0e48ba6ee8c8b4c8f38d3b879d6eafbbb1efc15', 'sleep_before_teardown': 0, 'subset': '1/512', 'suite': 'fs', 'suite_branch': 'wip-batrick-testing-20240327.230800', 'suite_path': '/home/teuthworker/src/git.ceph.com_ceph-c_e0e48ba6ee8c8b4c8f38d3b879d6eafbbb1efc15/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://git.ceph.com/ceph-ci.git', 'suite_sha1': 'e0e48ba6ee8c8b4c8f38d3b879d6eafbbb1efc15', 'syslog': {'ignorelist': ['WARNING*.*check_session_state', 'WARNING*.*__ceph_remove_cap']}, 'targets': {'smithi104.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBGXlb930Fp5uCSc+HMfKppybY3GP8wFgyECDsGraqfNuCbbML8e2DjaXRdJia3zcv5XKdUADEYo0s8OGWouN7lM=', 'smithi171.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBBOgEufC093sZkDP4SZ4QwXTRVg3TWB3hAP05pEnLzCzkgNvfFQrEf22R9X6bBjKRSoeKd/zi5NHter+nJ2py2c='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'flavor': 'default', 'sha1': 'e0e48ba6ee8c8b4c8f38d3b879d6eafbbb1efc15'}}, {'ceph': {'create_rbd_pool': False, 'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'pid file': '', 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug client': 20, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon down mkfs grace': 300, 'mon op complaint time': 120}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'FS_DEGRADED', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'MDS_UP_LESS_THAN_MAX', 'filesystem is online with fewer MDS than max_mds', 'POOL_APP_NOT_ENABLED', 'overall HEALTH_', 'Replacing daemon', 'deprecated feature inline_data', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'PG_AVAILABILITY', 'PG_DEGRADED', 'Reduced data availability'], 'sha1': 'e0e48ba6ee8c8b4c8f38d3b879d6eafbbb1efc15', 'cluster': 'ceph'}}, {'sequential': ['mgrmodules']}, {'ceph-fuse': None}, {'cephfs_test_runner': {'modules': ['tasks.cephfs.test_cephfs_shell']}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': ['local function is_kupstream() return false end', 'local function is_kdistro() return false end', 'local function is_fuse() return true end', "local function syntax_version() return '' end"]}, 'teuthology_branch': 'main', 'teuthology_sha1': 'e691533f9cbb33d85b2187bba20d7102f098636d', 'timestamp': '2024-03-28_07:14:07', 'tube': 'smithi', 'user': 'pdonnell', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885'} 2024-03-28T14:06:58.785 DEBUG:teuthology.orchestra.run.smithi104:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-03-28T14:06:58.788 DEBUG:teuthology.orchestra.run.smithi171:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-03-28T14:06:58.833 DEBUG:teuthology.orchestra.run.smithi104:> sudo install -d -m0777 -- /var/run/ceph 2024-03-28T14:06:58.835 DEBUG:teuthology.orchestra.run.smithi171:> sudo install -d -m0777 -- /var/run/ceph 2024-03-28T14:06:58.888 DEBUG:teuthology.orchestra.run.smithi104:> set -ex 2024-03-28T14:06:58.889 DEBUG:teuthology.orchestra.run.smithi104:> dd if=/scratch_devs of=/dev/stdout 2024-03-28T14:06:58.894 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-28T14:06:58.895 DEBUG:teuthology.orchestra.run.smithi104:> stat /dev/vg_nvme/lv_1 2024-03-28T14:06:58.943 INFO:teuthology.orchestra.run.smithi104.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-28T14:06:58.943 INFO:teuthology.orchestra.run.smithi104.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-28T14:06:58.943 INFO:teuthology.orchestra.run.smithi104.stdout:Device: 5h/5d Inode: 865 Links: 1 2024-03-28T14:06:58.943 INFO:teuthology.orchestra.run.smithi104.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-28T14:06:58.943 INFO:teuthology.orchestra.run.smithi104.stdout:Access: 2024-03-28 14:00:43.457380059 +0000 2024-03-28T14:06:58.943 INFO:teuthology.orchestra.run.smithi104.stdout:Modify: 2024-03-28 14:00:43.181382521 +0000 2024-03-28T14:06:58.944 INFO:teuthology.orchestra.run.smithi104.stdout:Change: 2024-03-28 14:00:43.181382521 +0000 2024-03-28T14:06:58.944 INFO:teuthology.orchestra.run.smithi104.stdout: Birth: - 2024-03-28T14:06:58.944 DEBUG:teuthology.orchestra.run.smithi104:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-28T14:06:58.997 INFO:teuthology.orchestra.run.smithi104.stderr:1+0 records in 2024-03-28T14:06:58.998 INFO:teuthology.orchestra.run.smithi104.stderr:1+0 records out 2024-03-28T14:06:58.998 INFO:teuthology.orchestra.run.smithi104.stderr:512 bytes copied, 0.000362316 s, 1.4 MB/s 2024-03-28T14:06:58.999 DEBUG:teuthology.orchestra.run.smithi104:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-28T14:06:59.048 DEBUG:teuthology.orchestra.run.smithi104:> stat /dev/vg_nvme/lv_2 2024-03-28T14:06:59.094 INFO:teuthology.orchestra.run.smithi104.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-28T14:06:59.094 INFO:teuthology.orchestra.run.smithi104.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-28T14:06:59.095 INFO:teuthology.orchestra.run.smithi104.stdout:Device: 5h/5d Inode: 887 Links: 1 2024-03-28T14:06:59.095 INFO:teuthology.orchestra.run.smithi104.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-28T14:06:59.095 INFO:teuthology.orchestra.run.smithi104.stdout:Access: 2024-03-28 14:00:43.937375776 +0000 2024-03-28T14:06:59.095 INFO:teuthology.orchestra.run.smithi104.stdout:Modify: 2024-03-28 14:00:43.693377953 +0000 2024-03-28T14:06:59.095 INFO:teuthology.orchestra.run.smithi104.stdout:Change: 2024-03-28 14:00:43.693377953 +0000 2024-03-28T14:06:59.095 INFO:teuthology.orchestra.run.smithi104.stdout: Birth: - 2024-03-28T14:06:59.095 DEBUG:teuthology.orchestra.run.smithi104:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-28T14:06:59.149 INFO:teuthology.orchestra.run.smithi104.stderr:1+0 records in 2024-03-28T14:06:59.149 INFO:teuthology.orchestra.run.smithi104.stderr:1+0 records out 2024-03-28T14:06:59.149 INFO:teuthology.orchestra.run.smithi104.stderr:512 bytes copied, 0.000424095 s, 1.2 MB/s 2024-03-28T14:06:59.150 DEBUG:teuthology.orchestra.run.smithi104:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-28T14:06:59.202 DEBUG:teuthology.orchestra.run.smithi104:> stat /dev/vg_nvme/lv_3 2024-03-28T14:06:59.251 INFO:teuthology.orchestra.run.smithi104.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-28T14:06:59.251 INFO:teuthology.orchestra.run.smithi104.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-28T14:06:59.251 INFO:teuthology.orchestra.run.smithi104.stdout:Device: 5h/5d Inode: 899 Links: 1 2024-03-28T14:06:59.251 INFO:teuthology.orchestra.run.smithi104.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-28T14:06:59.251 INFO:teuthology.orchestra.run.smithi104.stdout:Access: 2024-03-28 14:00:44.441371280 +0000 2024-03-28T14:06:59.251 INFO:teuthology.orchestra.run.smithi104.stdout:Modify: 2024-03-28 14:00:44.189373529 +0000 2024-03-28T14:06:59.251 INFO:teuthology.orchestra.run.smithi104.stdout:Change: 2024-03-28 14:00:44.189373529 +0000 2024-03-28T14:06:59.252 INFO:teuthology.orchestra.run.smithi104.stdout: Birth: - 2024-03-28T14:06:59.252 DEBUG:teuthology.orchestra.run.smithi104:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-28T14:06:59.305 INFO:teuthology.orchestra.run.smithi104.stderr:1+0 records in 2024-03-28T14:06:59.305 INFO:teuthology.orchestra.run.smithi104.stderr:1+0 records out 2024-03-28T14:06:59.305 INFO:teuthology.orchestra.run.smithi104.stderr:512 bytes copied, 0.000431563 s, 1.2 MB/s 2024-03-28T14:06:59.307 DEBUG:teuthology.orchestra.run.smithi104:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-28T14:06:59.356 DEBUG:teuthology.orchestra.run.smithi104:> stat /dev/vg_nvme/lv_4 2024-03-28T14:06:59.402 INFO:teuthology.orchestra.run.smithi104.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-28T14:06:59.402 INFO:teuthology.orchestra.run.smithi104.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-28T14:06:59.403 INFO:teuthology.orchestra.run.smithi104.stdout:Device: 5h/5d Inode: 921 Links: 1 2024-03-28T14:06:59.403 INFO:teuthology.orchestra.run.smithi104.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-28T14:06:59.403 INFO:teuthology.orchestra.run.smithi104.stdout:Access: 2024-03-28 14:00:44.933366892 +0000 2024-03-28T14:06:59.403 INFO:teuthology.orchestra.run.smithi104.stdout:Modify: 2024-03-28 14:00:44.689369068 +0000 2024-03-28T14:06:59.403 INFO:teuthology.orchestra.run.smithi104.stdout:Change: 2024-03-28 14:00:44.689369068 +0000 2024-03-28T14:06:59.403 INFO:teuthology.orchestra.run.smithi104.stdout: Birth: - 2024-03-28T14:06:59.403 DEBUG:teuthology.orchestra.run.smithi104:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-28T14:06:59.457 INFO:teuthology.orchestra.run.smithi104.stderr:1+0 records in 2024-03-28T14:06:59.458 INFO:teuthology.orchestra.run.smithi104.stderr:1+0 records out 2024-03-28T14:06:59.458 INFO:teuthology.orchestra.run.smithi104.stderr:512 bytes copied, 0.0003898 s, 1.3 MB/s 2024-03-28T14:06:59.459 DEBUG:teuthology.orchestra.run.smithi104:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-28T14:06:59.508 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-28T14:06:59.508 DEBUG:teuthology.orchestra.run.smithi171:> set -ex 2024-03-28T14:06:59.508 DEBUG:teuthology.orchestra.run.smithi171:> dd if=/scratch_devs of=/dev/stdout 2024-03-28T14:06:59.515 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-28T14:06:59.515 DEBUG:teuthology.orchestra.run.smithi171:> stat /dev/vg_nvme/lv_1 2024-03-28T14:06:59.561 INFO:teuthology.orchestra.run.smithi171.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-28T14:06:59.561 INFO:teuthology.orchestra.run.smithi171.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-28T14:06:59.561 INFO:teuthology.orchestra.run.smithi171.stdout:Device: 5h/5d Inode: 873 Links: 1 2024-03-28T14:06:59.561 INFO:teuthology.orchestra.run.smithi171.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-28T14:06:59.561 INFO:teuthology.orchestra.run.smithi171.stdout:Access: 2024-03-28 14:00:43.446728690 +0000 2024-03-28T14:06:59.561 INFO:teuthology.orchestra.run.smithi171.stdout:Modify: 2024-03-28 14:00:43.186731009 +0000 2024-03-28T14:06:59.561 INFO:teuthology.orchestra.run.smithi171.stdout:Change: 2024-03-28 14:00:43.186731009 +0000 2024-03-28T14:06:59.561 INFO:teuthology.orchestra.run.smithi171.stdout: Birth: - 2024-03-28T14:06:59.561 DEBUG:teuthology.orchestra.run.smithi171:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-28T14:06:59.617 INFO:teuthology.orchestra.run.smithi171.stderr:1+0 records in 2024-03-28T14:06:59.618 INFO:teuthology.orchestra.run.smithi171.stderr:1+0 records out 2024-03-28T14:06:59.618 INFO:teuthology.orchestra.run.smithi171.stderr:512 bytes copied, 0.000431033 s, 1.2 MB/s 2024-03-28T14:06:59.622 DEBUG:teuthology.orchestra.run.smithi171:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-28T14:06:59.686 DEBUG:teuthology.orchestra.run.smithi171:> stat /dev/vg_nvme/lv_2 2024-03-28T14:06:59.745 INFO:teuthology.orchestra.run.smithi171.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-28T14:06:59.745 INFO:teuthology.orchestra.run.smithi171.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-28T14:06:59.745 INFO:teuthology.orchestra.run.smithi171.stdout:Device: 5h/5d Inode: 891 Links: 1 2024-03-28T14:06:59.745 INFO:teuthology.orchestra.run.smithi171.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-28T14:06:59.745 INFO:teuthology.orchestra.run.smithi171.stdout:Access: 2024-03-28 14:00:43.970724015 +0000 2024-03-28T14:06:59.745 INFO:teuthology.orchestra.run.smithi171.stdout:Modify: 2024-03-28 14:00:43.682726584 +0000 2024-03-28T14:06:59.745 INFO:teuthology.orchestra.run.smithi171.stdout:Change: 2024-03-28 14:00:43.682726584 +0000 2024-03-28T14:06:59.745 INFO:teuthology.orchestra.run.smithi171.stdout: Birth: - 2024-03-28T14:06:59.746 DEBUG:teuthology.orchestra.run.smithi171:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-28T14:06:59.799 INFO:teuthology.orchestra.run.smithi171.stderr:1+0 records in 2024-03-28T14:06:59.800 INFO:teuthology.orchestra.run.smithi171.stderr:1+0 records out 2024-03-28T14:06:59.800 INFO:teuthology.orchestra.run.smithi171.stderr:512 bytes copied, 0.00042088 s, 1.2 MB/s 2024-03-28T14:06:59.801 DEBUG:teuthology.orchestra.run.smithi171:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-28T14:06:59.850 DEBUG:teuthology.orchestra.run.smithi171:> stat /dev/vg_nvme/lv_3 2024-03-28T14:06:59.907 INFO:teuthology.orchestra.run.smithi171.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-28T14:06:59.908 INFO:teuthology.orchestra.run.smithi171.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-28T14:06:59.908 INFO:teuthology.orchestra.run.smithi171.stdout:Device: 5h/5d Inode: 913 Links: 1 2024-03-28T14:06:59.908 INFO:teuthology.orchestra.run.smithi171.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-28T14:06:59.908 INFO:teuthology.orchestra.run.smithi171.stdout:Access: 2024-03-28 14:00:44.442719804 +0000 2024-03-28T14:06:59.908 INFO:teuthology.orchestra.run.smithi171.stdout:Modify: 2024-03-28 14:00:44.202721945 +0000 2024-03-28T14:06:59.908 INFO:teuthology.orchestra.run.smithi171.stdout:Change: 2024-03-28 14:00:44.202721945 +0000 2024-03-28T14:06:59.908 INFO:teuthology.orchestra.run.smithi171.stdout: Birth: - 2024-03-28T14:06:59.909 DEBUG:teuthology.orchestra.run.smithi171:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-28T14:06:59.974 INFO:teuthology.orchestra.run.smithi171.stderr:1+0 records in 2024-03-28T14:06:59.974 INFO:teuthology.orchestra.run.smithi171.stderr:1+0 records out 2024-03-28T14:06:59.974 INFO:teuthology.orchestra.run.smithi171.stderr:512 bytes copied, 0.000312123 s, 1.6 MB/s 2024-03-28T14:06:59.976 DEBUG:teuthology.orchestra.run.smithi171:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-28T14:07:00.030 DEBUG:teuthology.orchestra.run.smithi171:> stat /dev/vg_nvme/lv_4 2024-03-28T14:07:00.080 INFO:teuthology.orchestra.run.smithi171.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-28T14:07:00.080 INFO:teuthology.orchestra.run.smithi171.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-28T14:07:00.080 INFO:teuthology.orchestra.run.smithi171.stdout:Device: 5h/5d Inode: 931 Links: 1 2024-03-28T14:07:00.081 INFO:teuthology.orchestra.run.smithi171.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-28T14:07:00.081 INFO:teuthology.orchestra.run.smithi171.stdout:Access: 2024-03-28 14:00:44.678717699 +0000 2024-03-28T14:07:00.081 INFO:teuthology.orchestra.run.smithi171.stdout:Modify: 2024-03-28 14:00:44.674717735 +0000 2024-03-28T14:07:00.081 INFO:teuthology.orchestra.run.smithi171.stdout:Change: 2024-03-28 14:00:44.674717735 +0000 2024-03-28T14:07:00.081 INFO:teuthology.orchestra.run.smithi171.stdout: Birth: - 2024-03-28T14:07:00.081 DEBUG:teuthology.orchestra.run.smithi171:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-28T14:07:00.143 INFO:teuthology.orchestra.run.smithi171.stderr:1+0 records in 2024-03-28T14:07:00.143 INFO:teuthology.orchestra.run.smithi171.stderr:1+0 records out 2024-03-28T14:07:00.143 INFO:teuthology.orchestra.run.smithi171.stderr:512 bytes copied, 0.000382867 s, 1.3 MB/s 2024-03-28T14:07:00.144 DEBUG:teuthology.orchestra.run.smithi171:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-28T14:07:00.197 INFO:tasks.ceph:osd dev map: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-28T14:07:00.198 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi104.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi171.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-03-28T14:07:00.198 INFO:tasks.ceph:Generating config... 2024-03-28T14:07:00.199 INFO:tasks.ceph:[client] client mount timeout = 600 2024-03-28T14:07:00.199 INFO:tasks.ceph:[client] debug client = 20 2024-03-28T14:07:00.199 INFO:tasks.ceph:[client] debug ms = 1 2024-03-28T14:07:00.199 INFO:tasks.ceph:[client] pid file = 2024-03-28T14:07:00.199 INFO:tasks.ceph:[client] rados mon op timeout = 900 2024-03-28T14:07:00.199 INFO:tasks.ceph:[client] rados osd op timeout = 900 2024-03-28T14:07:00.199 INFO:tasks.ceph:[mds] debug mds = 20 2024-03-28T14:07:00.199 INFO:tasks.ceph:[mds] debug mds balancer = 20 2024-03-28T14:07:00.200 INFO:tasks.ceph:[mds] debug ms = 1 2024-03-28T14:07:00.200 INFO:tasks.ceph:[mds] mds debug frag = True 2024-03-28T14:07:00.200 INFO:tasks.ceph:[mds] mds debug scatterstat = True 2024-03-28T14:07:00.200 INFO:tasks.ceph:[mds] mds op complaint time = 180 2024-03-28T14:07:00.200 INFO:tasks.ceph:[mds] mds verify scatter = True 2024-03-28T14:07:00.200 INFO:tasks.ceph:[mds] osd op complaint time = 180 2024-03-28T14:07:00.200 INFO:tasks.ceph:[mds] rados mon op timeout = 900 2024-03-28T14:07:00.200 INFO:tasks.ceph:[mds] rados osd op timeout = 900 2024-03-28T14:07:00.200 INFO:tasks.ceph:[mgr] debug client = 20 2024-03-28T14:07:00.200 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-03-28T14:07:00.200 INFO:tasks.ceph:[mgr] debug ms = 1 2024-03-28T14:07:00.200 INFO:tasks.ceph:[mon] debug mon = 20 2024-03-28T14:07:00.200 INFO:tasks.ceph:[mon] debug ms = 1 2024-03-28T14:07:00.200 INFO:tasks.ceph:[mon] debug paxos = 20 2024-03-28T14:07:00.201 INFO:tasks.ceph:[mon] mon down mkfs grace = 300 2024-03-28T14:07:00.201 INFO:tasks.ceph:[mon] mon op complaint time = 120 2024-03-28T14:07:00.201 INFO:tasks.ceph:[osd] bdev async discard = True 2024-03-28T14:07:00.201 INFO:tasks.ceph:[osd] bdev enable discard = True 2024-03-28T14:07:00.201 INFO:tasks.ceph:[osd] bluestore allocator = bitmap 2024-03-28T14:07:00.201 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2024-03-28T14:07:00.201 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2024-03-28T14:07:00.201 INFO:tasks.ceph:[osd] debug bluefs = 1/20 2024-03-28T14:07:00.201 INFO:tasks.ceph:[osd] debug bluestore = 1/20 2024-03-28T14:07:00.201 INFO:tasks.ceph:[osd] debug ms = 1 2024-03-28T14:07:00.201 INFO:tasks.ceph:[osd] debug osd = 20 2024-03-28T14:07:00.201 INFO:tasks.ceph:[osd] debug rocksdb = 4/10 2024-03-28T14:07:00.201 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2024-03-28T14:07:00.201 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2024-03-28T14:07:00.202 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2024-03-28T14:07:00.202 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2024-03-28T14:07:00.202 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2024-03-28T14:07:00.202 INFO:tasks.ceph:[osd] osd op complaint time = 180 2024-03-28T14:07:00.202 INFO:tasks.ceph:Setting up mon.a... 2024-03-28T14:07:00.202 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2024-03-28T14:07:00.237 INFO:teuthology.orchestra.run.smithi104.stdout:creating /etc/ceph/ceph.keyring 2024-03-28T14:07:00.248 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2024-03-28T14:07:00.329 DEBUG:teuthology.orchestra.run.smithi104:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-03-28T14:07:00.383 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '172.21.15.104'), ('mon.b', '172.21.15.171'), ('mon.c', '[v2:172.21.15.171:3301,v1:172.21.15.171:6790]')] 2024-03-28T14:07:00.384 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.104,172.21.15.171,[v2:172.21.15.171:3301,v1:172.21.15.171:6790]'}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false', 'debug client': 20}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon down mkfs grace': 300, 'mon op complaint time': 120}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'pid file': '', 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mon.a': {}, 'mon.b': {}, 'mon.c': {}, 'mds.a': {}, 'mds.b': {}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}} 2024-03-28T14:07:00.385 DEBUG:teuthology.orchestra.run.smithi104:> set -ex 2024-03-28T14:07:00.385 DEBUG:teuthology.orchestra.run.smithi104:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-03-28T14:07:00.438 DEBUG:teuthology.orchestra.run.smithi104:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.104 --add b 172.21.15.171 --addv c '[v2:172.21.15.171:3301,v1:172.21.15.171:6790]' --print /home/ubuntu/cephtest/ceph.monmap 2024-03-28T14:07:00.524 INFO:teuthology.orchestra.run.smithi104.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2024-03-28T14:07:00.524 INFO:teuthology.orchestra.run.smithi104.stdout:monmaptool: generated fsid 428a90f2-3351-435f-8fcf-f7e66c78a13e 2024-03-28T14:07:00.524 INFO:teuthology.orchestra.run.smithi104.stdout:setting min_mon_release = quincy 2024-03-28T14:07:00.524 INFO:teuthology.orchestra.run.smithi104.stdout:epoch 0 2024-03-28T14:07:00.524 INFO:teuthology.orchestra.run.smithi104.stdout:fsid 428a90f2-3351-435f-8fcf-f7e66c78a13e 2024-03-28T14:07:00.525 INFO:teuthology.orchestra.run.smithi104.stdout:last_changed 2024-03-28T14:07:00.527348+0000 2024-03-28T14:07:00.525 INFO:teuthology.orchestra.run.smithi104.stdout:created 2024-03-28T14:07:00.527348+0000 2024-03-28T14:07:00.525 INFO:teuthology.orchestra.run.smithi104.stdout:min_mon_release 17 (quincy) 2024-03-28T14:07:00.525 INFO:teuthology.orchestra.run.smithi104.stdout:election_strategy: 1 2024-03-28T14:07:00.525 INFO:teuthology.orchestra.run.smithi104.stdout:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-03-28T14:07:00.525 INFO:teuthology.orchestra.run.smithi104.stdout:1: [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] mon.b 2024-03-28T14:07:00.525 INFO:teuthology.orchestra.run.smithi104.stdout:2: [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] mon.c 2024-03-28T14:07:00.525 INFO:teuthology.orchestra.run.smithi104.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (3 monitors) 2024-03-28T14:07:00.526 DEBUG:teuthology.orchestra.run.smithi104:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-03-28T14:07:00.579 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID 428a90f2-3351-435f-8fcf-f7e66c78a13e... 2024-03-28T14:07:00.580 DEBUG:teuthology.orchestra.run.smithi104:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-03-28T14:07:00.629 DEBUG:teuthology.orchestra.run.smithi171:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-03-28T14:07:00.658 INFO:teuthology.orchestra.run.smithi104.stdout:[global] 2024-03-28T14:07:00.658 INFO:teuthology.orchestra.run.smithi104.stdout: chdir = "" 2024-03-28T14:07:00.658 INFO:teuthology.orchestra.run.smithi104.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-03-28T14:07:00.659 INFO:teuthology.orchestra.run.smithi104.stdout: auth supported = cephx 2024-03-28T14:07:00.659 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-03-28T14:07:00.659 INFO:teuthology.orchestra.run.smithi104.stdout: filestore xattr use omap = true 2024-03-28T14:07:00.659 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-03-28T14:07:00.659 INFO:teuthology.orchestra.run.smithi104.stdout: mon clock drift allowed = 1.000 2024-03-28T14:07:00.659 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-03-28T14:07:00.659 INFO:teuthology.orchestra.run.smithi104.stdout: osd crush chooseleaf type = 0 2024-03-28T14:07:00.660 INFO:teuthology.orchestra.run.smithi104.stdout: auth debug = true 2024-03-28T14:07:00.660 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-03-28T14:07:00.660 INFO:teuthology.orchestra.run.smithi104.stdout: ms die on old message = true 2024-03-28T14:07:00.660 INFO:teuthology.orchestra.run.smithi104.stdout: ms die on bug = true 2024-03-28T14:07:00.660 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-03-28T14:07:00.660 INFO:teuthology.orchestra.run.smithi104.stdout: mon max pg per osd = 10000 # >= luminous 2024-03-28T14:07:00.661 INFO:teuthology.orchestra.run.smithi104.stdout: mon pg warn max object skew = 0 2024-03-28T14:07:00.661 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-03-28T14:07:00.661 INFO:teuthology.orchestra.run.smithi104.stdout: # disable pg_autoscaler by default for new pools 2024-03-28T14:07:00.661 INFO:teuthology.orchestra.run.smithi104.stdout: osd_pool_default_pg_autoscale_mode = off 2024-03-28T14:07:00.661 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-03-28T14:07:00.661 INFO:teuthology.orchestra.run.smithi104.stdout: osd pool default size = 2 2024-03-28T14:07:00.661 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-03-28T14:07:00.662 INFO:teuthology.orchestra.run.smithi104.stdout: mon osd allow primary affinity = true 2024-03-28T14:07:00.662 INFO:teuthology.orchestra.run.smithi104.stdout: mon osd allow pg remap = true 2024-03-28T14:07:00.662 INFO:teuthology.orchestra.run.smithi104.stdout: mon warn on legacy crush tunables = false 2024-03-28T14:07:00.662 INFO:teuthology.orchestra.run.smithi104.stdout: mon warn on crush straw calc version zero = false 2024-03-28T14:07:00.662 INFO:teuthology.orchestra.run.smithi104.stdout: mon warn on no sortbitwise = false 2024-03-28T14:07:00.662 INFO:teuthology.orchestra.run.smithi104.stdout: mon warn on osd down out interval zero = false 2024-03-28T14:07:00.662 INFO:teuthology.orchestra.run.smithi104.stdout: mon warn on too few osds = false 2024-03-28T14:07:00.663 INFO:teuthology.orchestra.run.smithi104.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-03-28T14:07:00.663 INFO:teuthology.orchestra.run.smithi104.stdout: mon_warn_on_pool_no_redundancy = false 2024-03-28T14:07:00.663 INFO:teuthology.orchestra.run.smithi104.stdout: mon_allow_pool_size_one = true 2024-03-28T14:07:00.663 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-03-28T14:07:00.663 INFO:teuthology.orchestra.run.smithi104.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-03-28T14:07:00.663 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-03-28T14:07:00.663 INFO:teuthology.orchestra.run.smithi104.stdout: osd default data pool replay window = 5 2024-03-28T14:07:00.663 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-03-28T14:07:00.664 INFO:teuthology.orchestra.run.smithi104.stdout: mon allow pool delete = true 2024-03-28T14:07:00.664 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-03-28T14:07:00.664 INFO:teuthology.orchestra.run.smithi104.stdout: mon cluster log file level = debug 2024-03-28T14:07:00.664 INFO:teuthology.orchestra.run.smithi104.stdout: debug asserts on shutdown = true 2024-03-28T14:07:00.664 INFO:teuthology.orchestra.run.smithi104.stdout: mon health detail to clog = false 2024-03-28T14:07:00.664 INFO:teuthology.orchestra.run.smithi104.stdout: mon host = "172.21.15.104,172.21.15.171,[v2:172.21.15.171:3301,v1:172.21.15.171:6790]" 2024-03-28T14:07:00.664 INFO:teuthology.orchestra.run.smithi104.stdout: fsid = 428a90f2-3351-435f-8fcf-f7e66c78a13e 2024-03-28T14:07:00.664 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-03-28T14:07:00.664 INFO:teuthology.orchestra.run.smithi104.stdout:[osd] 2024-03-28T14:07:00.665 INFO:teuthology.orchestra.run.smithi104.stdout: osd journal size = 100 2024-03-28T14:07:00.665 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-03-28T14:07:00.665 INFO:teuthology.orchestra.run.smithi104.stdout: osd scrub load threshold = 5.0 2024-03-28T14:07:00.665 INFO:teuthology.orchestra.run.smithi104.stdout: osd scrub max interval = 600 2024-03-28T14:07:00.665 INFO:teuthology.orchestra.run.smithi104.stdout: osd mclock profile = high_recovery_ops 2024-03-28T14:07:00.665 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-03-28T14:07:00.666 INFO:teuthology.orchestra.run.smithi104.stdout: osd recover clone overlap = true 2024-03-28T14:07:00.666 INFO:teuthology.orchestra.run.smithi104.stdout: osd recovery max chunk = 1048576 2024-03-28T14:07:00.666 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-03-28T14:07:00.666 INFO:teuthology.orchestra.run.smithi104.stdout: osd debug shutdown = true 2024-03-28T14:07:00.666 INFO:teuthology.orchestra.run.smithi104.stdout: osd debug op order = true 2024-03-28T14:07:00.667 INFO:teuthology.orchestra.run.smithi104.stdout: osd debug verify stray on activate = true 2024-03-28T14:07:00.667 INFO:teuthology.orchestra.run.smithi104.stdout: osd debug trim objects = true 2024-03-28T14:07:00.667 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-03-28T14:07:00.667 INFO:teuthology.orchestra.run.smithi104.stdout: osd open classes on start = true 2024-03-28T14:07:00.667 INFO:teuthology.orchestra.run.smithi104.stdout: osd debug pg log writeout = true 2024-03-28T14:07:00.667 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-03-28T14:07:00.667 INFO:teuthology.orchestra.run.smithi104.stdout: osd deep scrub update digest min age = 30 2024-03-28T14:07:00.667 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-03-28T14:07:00.667 INFO:teuthology.orchestra.run.smithi104.stdout: osd map max advance = 10 2024-03-28T14:07:00.667 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-03-28T14:07:00.667 INFO:teuthology.orchestra.run.smithi104.stdout: journal zero on create = true 2024-03-28T14:07:00.668 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-03-28T14:07:00.668 INFO:teuthology.orchestra.run.smithi104.stdout: filestore ondisk finisher threads = 3 2024-03-28T14:07:00.668 INFO:teuthology.orchestra.run.smithi104.stdout: filestore apply finisher threads = 3 2024-03-28T14:07:00.668 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-03-28T14:07:00.668 INFO:teuthology.orchestra.run.smithi104.stdout: bdev debug aio = true 2024-03-28T14:07:00.668 INFO:teuthology.orchestra.run.smithi104.stdout: osd debug misdirected ops = true 2024-03-28T14:07:00.668 INFO:teuthology.orchestra.run.smithi104.stdout: bdev async discard = True 2024-03-28T14:07:00.668 INFO:teuthology.orchestra.run.smithi104.stdout: bdev enable discard = True 2024-03-28T14:07:00.668 INFO:teuthology.orchestra.run.smithi104.stdout: bluestore allocator = bitmap 2024-03-28T14:07:00.669 INFO:teuthology.orchestra.run.smithi104.stdout: bluestore block size = 96636764160 2024-03-28T14:07:00.669 INFO:teuthology.orchestra.run.smithi104.stdout: bluestore fsck on mount = True 2024-03-28T14:07:00.669 INFO:teuthology.orchestra.run.smithi104.stdout: debug bluefs = 1/20 2024-03-28T14:07:00.669 INFO:teuthology.orchestra.run.smithi104.stdout: debug bluestore = 1/20 2024-03-28T14:07:00.669 INFO:teuthology.orchestra.run.smithi104.stdout: debug ms = 1 2024-03-28T14:07:00.669 INFO:teuthology.orchestra.run.smithi104.stdout: debug osd = 20 2024-03-28T14:07:00.669 INFO:teuthology.orchestra.run.smithi104.stdout: debug rocksdb = 4/10 2024-03-28T14:07:00.669 INFO:teuthology.orchestra.run.smithi104.stdout: mon osd backfillfull_ratio = 0.85 2024-03-28T14:07:00.669 INFO:teuthology.orchestra.run.smithi104.stdout: mon osd full ratio = 0.9 2024-03-28T14:07:00.669 INFO:teuthology.orchestra.run.smithi104.stdout: mon osd nearfull ratio = 0.8 2024-03-28T14:07:00.670 INFO:teuthology.orchestra.run.smithi104.stdout: osd failsafe full ratio = 0.95 2024-03-28T14:07:00.670 INFO:teuthology.orchestra.run.smithi104.stdout: osd objectstore = bluestore 2024-03-28T14:07:00.670 INFO:teuthology.orchestra.run.smithi104.stdout: osd op complaint time = 180 2024-03-28T14:07:00.670 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-03-28T14:07:00.670 INFO:teuthology.orchestra.run.smithi104.stdout:[mgr] 2024-03-28T14:07:00.670 INFO:teuthology.orchestra.run.smithi104.stdout: debug ms = 1 2024-03-28T14:07:00.670 INFO:teuthology.orchestra.run.smithi104.stdout: debug mgr = 20 2024-03-28T14:07:00.670 INFO:teuthology.orchestra.run.smithi104.stdout: debug mon = 20 2024-03-28T14:07:00.671 INFO:teuthology.orchestra.run.smithi104.stdout: debug auth = 20 2024-03-28T14:07:00.671 INFO:teuthology.orchestra.run.smithi104.stdout: mon reweight min pgs per osd = 4 2024-03-28T14:07:00.671 INFO:teuthology.orchestra.run.smithi104.stdout: mon reweight min bytes per osd = 10 2024-03-28T14:07:00.671 INFO:teuthology.orchestra.run.smithi104.stdout: mgr/telemetry/nag = false 2024-03-28T14:07:00.671 INFO:teuthology.orchestra.run.smithi104.stdout: debug client = 20 2024-03-28T14:07:00.671 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-03-28T14:07:00.671 INFO:teuthology.orchestra.run.smithi104.stdout:[mon] 2024-03-28T14:07:00.671 INFO:teuthology.orchestra.run.smithi104.stdout: debug ms = 1 2024-03-28T14:07:00.671 INFO:teuthology.orchestra.run.smithi104.stdout: debug mon = 20 2024-03-28T14:07:00.672 INFO:teuthology.orchestra.run.smithi104.stdout: debug paxos = 20 2024-03-28T14:07:00.672 INFO:teuthology.orchestra.run.smithi104.stdout: debug auth = 20 2024-03-28T14:07:00.672 INFO:teuthology.orchestra.run.smithi104.stdout: mon data avail warn = 5 2024-03-28T14:07:00.672 INFO:teuthology.orchestra.run.smithi104.stdout: mon mgr mkfs grace = 240 2024-03-28T14:07:00.672 INFO:teuthology.orchestra.run.smithi104.stdout: mon reweight min pgs per osd = 4 2024-03-28T14:07:00.672 INFO:teuthology.orchestra.run.smithi104.stdout: mon osd reporter subtree level = osd 2024-03-28T14:07:00.672 INFO:teuthology.orchestra.run.smithi104.stdout: mon osd prime pg temp = true 2024-03-28T14:07:00.672 INFO:teuthology.orchestra.run.smithi104.stdout: mon reweight min bytes per osd = 10 2024-03-28T14:07:00.672 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-03-28T14:07:00.672 INFO:teuthology.orchestra.run.smithi104.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-03-28T14:07:00.673 INFO:teuthology.orchestra.run.smithi104.stdout: auth mon ticket ttl = 660 # 11m 2024-03-28T14:07:00.673 INFO:teuthology.orchestra.run.smithi104.stdout: auth service ticket ttl = 240 # 4m 2024-03-28T14:07:00.673 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-03-28T14:07:00.673 INFO:teuthology.orchestra.run.smithi104.stdout: # don't complain about insecure global_id in the test suite 2024-03-28T14:07:00.674 INFO:teuthology.orchestra.run.smithi104.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-03-28T14:07:00.674 INFO:teuthology.orchestra.run.smithi104.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-03-28T14:07:00.674 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-03-28T14:07:00.674 INFO:teuthology.orchestra.run.smithi104.stdout: # 1m isn't quite enough 2024-03-28T14:07:00.674 INFO:teuthology.orchestra.run.smithi104.stdout: mon_down_mkfs_grace = 2m 2024-03-28T14:07:00.674 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-03-28T14:07:00.674 INFO:teuthology.orchestra.run.smithi104.stdout: mon_warn_on_filestore_osds = false 2024-03-28T14:07:00.674 INFO:teuthology.orchestra.run.smithi104.stdout: mon down mkfs grace = 300 2024-03-28T14:07:00.674 INFO:teuthology.orchestra.run.smithi104.stdout: mon op complaint time = 120 2024-03-28T14:07:00.674 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-03-28T14:07:00.675 INFO:teuthology.orchestra.run.smithi104.stdout:[client] 2024-03-28T14:07:00.675 INFO:teuthology.orchestra.run.smithi104.stdout: rgw cache enabled = true 2024-03-28T14:07:00.675 INFO:teuthology.orchestra.run.smithi104.stdout: rgw enable ops log = true 2024-03-28T14:07:00.675 INFO:teuthology.orchestra.run.smithi104.stdout: rgw enable usage log = true 2024-03-28T14:07:00.675 INFO:teuthology.orchestra.run.smithi104.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-03-28T14:07:00.675 INFO:teuthology.orchestra.run.smithi104.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-03-28T14:07:00.675 INFO:teuthology.orchestra.run.smithi104.stdout: client mount timeout = 600 2024-03-28T14:07:00.675 INFO:teuthology.orchestra.run.smithi104.stdout: debug client = 20 2024-03-28T14:07:00.676 INFO:teuthology.orchestra.run.smithi104.stdout: debug ms = 1 2024-03-28T14:07:00.676 INFO:teuthology.orchestra.run.smithi104.stdout: pid file = "" 2024-03-28T14:07:00.676 INFO:teuthology.orchestra.run.smithi104.stdout: rados mon op timeout = 900 2024-03-28T14:07:00.676 INFO:teuthology.orchestra.run.smithi104.stdout: rados osd op timeout = 900 2024-03-28T14:07:00.676 INFO:teuthology.orchestra.run.smithi104.stdout:[mon.a] 2024-03-28T14:07:00.676 INFO:teuthology.orchestra.run.smithi104.stdout:[mon.b] 2024-03-28T14:07:00.676 INFO:teuthology.orchestra.run.smithi104.stdout:[mon.c] 2024-03-28T14:07:00.677 INFO:teuthology.orchestra.run.smithi104.stdout:[mds.a] 2024-03-28T14:07:00.677 INFO:teuthology.orchestra.run.smithi104.stdout:[mds.b] 2024-03-28T14:07:00.677 INFO:teuthology.orchestra.run.smithi104.stdout:[mds] 2024-03-28T14:07:00.677 INFO:teuthology.orchestra.run.smithi104.stdout: debug mds = 20 2024-03-28T14:07:00.677 INFO:teuthology.orchestra.run.smithi104.stdout: debug mds balancer = 20 2024-03-28T14:07:00.677 INFO:teuthology.orchestra.run.smithi104.stdout: debug ms = 1 2024-03-28T14:07:00.677 INFO:teuthology.orchestra.run.smithi104.stdout: mds debug frag = True 2024-03-28T14:07:00.677 INFO:teuthology.orchestra.run.smithi104.stdout: mds debug scatterstat = True 2024-03-28T14:07:00.677 INFO:teuthology.orchestra.run.smithi104.stdout: mds op complaint time = 180 2024-03-28T14:07:00.677 INFO:teuthology.orchestra.run.smithi104.stdout: mds verify scatter = True 2024-03-28T14:07:00.677 INFO:teuthology.orchestra.run.smithi104.stdout: osd op complaint time = 180 2024-03-28T14:07:00.677 INFO:teuthology.orchestra.run.smithi104.stdout: rados mon op timeout = 900 2024-03-28T14:07:00.677 INFO:teuthology.orchestra.run.smithi104.stdout: rados osd op timeout = 900 2024-03-28T14:07:00.679 INFO:teuthology.orchestra.run.smithi171.stdout:[global] 2024-03-28T14:07:00.680 INFO:teuthology.orchestra.run.smithi171.stdout: chdir = "" 2024-03-28T14:07:00.680 INFO:teuthology.orchestra.run.smithi171.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-03-28T14:07:00.680 INFO:teuthology.orchestra.run.smithi171.stdout: auth supported = cephx 2024-03-28T14:07:00.680 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-03-28T14:07:00.680 INFO:teuthology.orchestra.run.smithi171.stdout: filestore xattr use omap = true 2024-03-28T14:07:00.680 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-03-28T14:07:00.680 INFO:teuthology.orchestra.run.smithi171.stdout: mon clock drift allowed = 1.000 2024-03-28T14:07:00.680 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-03-28T14:07:00.681 INFO:teuthology.orchestra.run.smithi171.stdout: osd crush chooseleaf type = 0 2024-03-28T14:07:00.681 INFO:teuthology.orchestra.run.smithi171.stdout: auth debug = true 2024-03-28T14:07:00.681 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-03-28T14:07:00.681 INFO:teuthology.orchestra.run.smithi171.stdout: ms die on old message = true 2024-03-28T14:07:00.681 INFO:teuthology.orchestra.run.smithi171.stdout: ms die on bug = true 2024-03-28T14:07:00.681 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-03-28T14:07:00.681 INFO:teuthology.orchestra.run.smithi171.stdout: mon max pg per osd = 10000 # >= luminous 2024-03-28T14:07:00.681 INFO:teuthology.orchestra.run.smithi171.stdout: mon pg warn max object skew = 0 2024-03-28T14:07:00.681 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-03-28T14:07:00.681 INFO:teuthology.orchestra.run.smithi171.stdout: # disable pg_autoscaler by default for new pools 2024-03-28T14:07:00.681 INFO:teuthology.orchestra.run.smithi171.stdout: osd_pool_default_pg_autoscale_mode = off 2024-03-28T14:07:00.681 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-03-28T14:07:00.682 INFO:teuthology.orchestra.run.smithi171.stdout: osd pool default size = 2 2024-03-28T14:07:00.682 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-03-28T14:07:00.682 INFO:teuthology.orchestra.run.smithi171.stdout: mon osd allow primary affinity = true 2024-03-28T14:07:00.682 INFO:teuthology.orchestra.run.smithi171.stdout: mon osd allow pg remap = true 2024-03-28T14:07:00.682 INFO:teuthology.orchestra.run.smithi171.stdout: mon warn on legacy crush tunables = false 2024-03-28T14:07:00.682 INFO:teuthology.orchestra.run.smithi171.stdout: mon warn on crush straw calc version zero = false 2024-03-28T14:07:00.682 INFO:teuthology.orchestra.run.smithi171.stdout: mon warn on no sortbitwise = false 2024-03-28T14:07:00.682 INFO:teuthology.orchestra.run.smithi171.stdout: mon warn on osd down out interval zero = false 2024-03-28T14:07:00.682 INFO:teuthology.orchestra.run.smithi171.stdout: mon warn on too few osds = false 2024-03-28T14:07:00.682 INFO:teuthology.orchestra.run.smithi171.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-03-28T14:07:00.682 INFO:teuthology.orchestra.run.smithi171.stdout: mon_warn_on_pool_no_redundancy = false 2024-03-28T14:07:00.682 INFO:teuthology.orchestra.run.smithi171.stdout: mon_allow_pool_size_one = true 2024-03-28T14:07:00.683 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-03-28T14:07:00.683 INFO:teuthology.orchestra.run.smithi171.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-03-28T14:07:00.683 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-03-28T14:07:00.683 INFO:teuthology.orchestra.run.smithi171.stdout: osd default data pool replay window = 5 2024-03-28T14:07:00.683 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-03-28T14:07:00.683 INFO:teuthology.orchestra.run.smithi171.stdout: mon allow pool delete = true 2024-03-28T14:07:00.683 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-03-28T14:07:00.683 INFO:teuthology.orchestra.run.smithi171.stdout: mon cluster log file level = debug 2024-03-28T14:07:00.683 INFO:teuthology.orchestra.run.smithi171.stdout: debug asserts on shutdown = true 2024-03-28T14:07:00.683 INFO:teuthology.orchestra.run.smithi171.stdout: mon health detail to clog = false 2024-03-28T14:07:00.683 INFO:teuthology.orchestra.run.smithi171.stdout: mon host = "172.21.15.104,172.21.15.171,[v2:172.21.15.171:3301,v1:172.21.15.171:6790]" 2024-03-28T14:07:00.683 INFO:teuthology.orchestra.run.smithi171.stdout: fsid = 428a90f2-3351-435f-8fcf-f7e66c78a13e 2024-03-28T14:07:00.684 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-03-28T14:07:00.684 INFO:teuthology.orchestra.run.smithi171.stdout:[osd] 2024-03-28T14:07:00.684 INFO:teuthology.orchestra.run.smithi171.stdout: osd journal size = 100 2024-03-28T14:07:00.684 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-03-28T14:07:00.684 INFO:teuthology.orchestra.run.smithi171.stdout: osd scrub load threshold = 5.0 2024-03-28T14:07:00.684 INFO:teuthology.orchestra.run.smithi171.stdout: osd scrub max interval = 600 2024-03-28T14:07:00.684 INFO:teuthology.orchestra.run.smithi171.stdout: osd mclock profile = high_recovery_ops 2024-03-28T14:07:00.684 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-03-28T14:07:00.684 INFO:teuthology.orchestra.run.smithi171.stdout: osd recover clone overlap = true 2024-03-28T14:07:00.684 INFO:teuthology.orchestra.run.smithi171.stdout: osd recovery max chunk = 1048576 2024-03-28T14:07:00.684 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-03-28T14:07:00.684 INFO:teuthology.orchestra.run.smithi171.stdout: osd debug shutdown = true 2024-03-28T14:07:00.684 INFO:teuthology.orchestra.run.smithi171.stdout: osd debug op order = true 2024-03-28T14:07:00.685 INFO:teuthology.orchestra.run.smithi171.stdout: osd debug verify stray on activate = true 2024-03-28T14:07:00.685 INFO:teuthology.orchestra.run.smithi171.stdout: osd debug trim objects = true 2024-03-28T14:07:00.685 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-03-28T14:07:00.685 INFO:teuthology.orchestra.run.smithi171.stdout: osd open classes on start = true 2024-03-28T14:07:00.685 INFO:teuthology.orchestra.run.smithi171.stdout: osd debug pg log writeout = true 2024-03-28T14:07:00.685 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-03-28T14:07:00.685 INFO:teuthology.orchestra.run.smithi171.stdout: osd deep scrub update digest min age = 30 2024-03-28T14:07:00.685 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-03-28T14:07:00.685 INFO:teuthology.orchestra.run.smithi171.stdout: osd map max advance = 10 2024-03-28T14:07:00.685 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-03-28T14:07:00.685 INFO:teuthology.orchestra.run.smithi171.stdout: journal zero on create = true 2024-03-28T14:07:00.685 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-03-28T14:07:00.685 INFO:teuthology.orchestra.run.smithi171.stdout: filestore ondisk finisher threads = 3 2024-03-28T14:07:00.685 INFO:teuthology.orchestra.run.smithi171.stdout: filestore apply finisher threads = 3 2024-03-28T14:07:00.685 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-03-28T14:07:00.686 INFO:teuthology.orchestra.run.smithi171.stdout: bdev debug aio = true 2024-03-28T14:07:00.686 INFO:teuthology.orchestra.run.smithi171.stdout: osd debug misdirected ops = true 2024-03-28T14:07:00.686 INFO:teuthology.orchestra.run.smithi171.stdout: bdev async discard = True 2024-03-28T14:07:00.686 INFO:teuthology.orchestra.run.smithi171.stdout: bdev enable discard = True 2024-03-28T14:07:00.686 INFO:teuthology.orchestra.run.smithi171.stdout: bluestore allocator = bitmap 2024-03-28T14:07:00.686 INFO:teuthology.orchestra.run.smithi171.stdout: bluestore block size = 96636764160 2024-03-28T14:07:00.686 INFO:teuthology.orchestra.run.smithi171.stdout: bluestore fsck on mount = True 2024-03-28T14:07:00.686 INFO:teuthology.orchestra.run.smithi171.stdout: debug bluefs = 1/20 2024-03-28T14:07:00.686 INFO:teuthology.orchestra.run.smithi171.stdout: debug bluestore = 1/20 2024-03-28T14:07:00.686 INFO:teuthology.orchestra.run.smithi171.stdout: debug ms = 1 2024-03-28T14:07:00.686 INFO:teuthology.orchestra.run.smithi171.stdout: debug osd = 20 2024-03-28T14:07:00.686 INFO:teuthology.orchestra.run.smithi171.stdout: debug rocksdb = 4/10 2024-03-28T14:07:00.687 INFO:teuthology.orchestra.run.smithi171.stdout: mon osd backfillfull_ratio = 0.85 2024-03-28T14:07:00.687 INFO:teuthology.orchestra.run.smithi171.stdout: mon osd full ratio = 0.9 2024-03-28T14:07:00.687 INFO:teuthology.orchestra.run.smithi171.stdout: mon osd nearfull ratio = 0.8 2024-03-28T14:07:00.687 INFO:teuthology.orchestra.run.smithi171.stdout: osd failsafe full ratio = 0.95 2024-03-28T14:07:00.687 INFO:teuthology.orchestra.run.smithi171.stdout: osd objectstore = bluestore 2024-03-28T14:07:00.687 INFO:teuthology.orchestra.run.smithi171.stdout: osd op complaint time = 180 2024-03-28T14:07:00.687 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-03-28T14:07:00.687 INFO:teuthology.orchestra.run.smithi171.stdout:[mgr] 2024-03-28T14:07:00.687 INFO:teuthology.orchestra.run.smithi171.stdout: debug ms = 1 2024-03-28T14:07:00.687 INFO:teuthology.orchestra.run.smithi171.stdout: debug mgr = 20 2024-03-28T14:07:00.687 INFO:teuthology.orchestra.run.smithi171.stdout: debug mon = 20 2024-03-28T14:07:00.688 INFO:teuthology.orchestra.run.smithi171.stdout: debug auth = 20 2024-03-28T14:07:00.688 INFO:teuthology.orchestra.run.smithi171.stdout: mon reweight min pgs per osd = 4 2024-03-28T14:07:00.688 INFO:teuthology.orchestra.run.smithi171.stdout: mon reweight min bytes per osd = 10 2024-03-28T14:07:00.688 INFO:teuthology.orchestra.run.smithi171.stdout: mgr/telemetry/nag = false 2024-03-28T14:07:00.688 INFO:teuthology.orchestra.run.smithi171.stdout: debug client = 20 2024-03-28T14:07:00.688 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-03-28T14:07:00.688 INFO:teuthology.orchestra.run.smithi171.stdout:[mon] 2024-03-28T14:07:00.688 INFO:teuthology.orchestra.run.smithi171.stdout: debug ms = 1 2024-03-28T14:07:00.688 INFO:teuthology.orchestra.run.smithi171.stdout: debug mon = 20 2024-03-28T14:07:00.688 INFO:teuthology.orchestra.run.smithi171.stdout: debug paxos = 20 2024-03-28T14:07:00.688 INFO:teuthology.orchestra.run.smithi171.stdout: debug auth = 20 2024-03-28T14:07:00.688 INFO:teuthology.orchestra.run.smithi171.stdout: mon data avail warn = 5 2024-03-28T14:07:00.689 INFO:teuthology.orchestra.run.smithi171.stdout: mon mgr mkfs grace = 240 2024-03-28T14:07:00.689 INFO:teuthology.orchestra.run.smithi171.stdout: mon reweight min pgs per osd = 4 2024-03-28T14:07:00.689 INFO:teuthology.orchestra.run.smithi171.stdout: mon osd reporter subtree level = osd 2024-03-28T14:07:00.689 INFO:teuthology.orchestra.run.smithi171.stdout: mon osd prime pg temp = true 2024-03-28T14:07:00.689 INFO:teuthology.orchestra.run.smithi171.stdout: mon reweight min bytes per osd = 10 2024-03-28T14:07:00.689 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-03-28T14:07:00.689 INFO:teuthology.orchestra.run.smithi171.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-03-28T14:07:00.689 INFO:teuthology.orchestra.run.smithi171.stdout: auth mon ticket ttl = 660 # 11m 2024-03-28T14:07:00.689 INFO:teuthology.orchestra.run.smithi171.stdout: auth service ticket ttl = 240 # 4m 2024-03-28T14:07:00.689 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-03-28T14:07:00.689 INFO:teuthology.orchestra.run.smithi171.stdout: # don't complain about insecure global_id in the test suite 2024-03-28T14:07:00.689 INFO:teuthology.orchestra.run.smithi171.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-03-28T14:07:00.689 INFO:teuthology.orchestra.run.smithi171.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-03-28T14:07:00.689 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-03-28T14:07:00.690 INFO:teuthology.orchestra.run.smithi171.stdout: # 1m isn't quite enough 2024-03-28T14:07:00.690 INFO:teuthology.orchestra.run.smithi171.stdout: mon_down_mkfs_grace = 2m 2024-03-28T14:07:00.690 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-03-28T14:07:00.690 INFO:teuthology.orchestra.run.smithi171.stdout: mon_warn_on_filestore_osds = false 2024-03-28T14:07:00.690 INFO:teuthology.orchestra.run.smithi171.stdout: mon down mkfs grace = 300 2024-03-28T14:07:00.690 INFO:teuthology.orchestra.run.smithi171.stdout: mon op complaint time = 120 2024-03-28T14:07:00.690 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-03-28T14:07:00.690 INFO:teuthology.orchestra.run.smithi171.stdout:[client] 2024-03-28T14:07:00.690 INFO:teuthology.orchestra.run.smithi171.stdout: rgw cache enabled = true 2024-03-28T14:07:00.690 INFO:teuthology.orchestra.run.smithi171.stdout: rgw enable ops log = true 2024-03-28T14:07:00.690 INFO:teuthology.orchestra.run.smithi171.stdout: rgw enable usage log = true 2024-03-28T14:07:00.690 INFO:teuthology.orchestra.run.smithi171.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-03-28T14:07:00.691 INFO:teuthology.orchestra.run.smithi171.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-03-28T14:07:00.691 INFO:teuthology.orchestra.run.smithi171.stdout: client mount timeout = 600 2024-03-28T14:07:00.691 INFO:teuthology.orchestra.run.smithi171.stdout: debug client = 20 2024-03-28T14:07:00.691 INFO:teuthology.orchestra.run.smithi171.stdout: debug ms = 1 2024-03-28T14:07:00.691 INFO:teuthology.orchestra.run.smithi171.stdout: pid file = "" 2024-03-28T14:07:00.691 INFO:teuthology.orchestra.run.smithi171.stdout: rados mon op timeout = 900 2024-03-28T14:07:00.691 INFO:teuthology.orchestra.run.smithi171.stdout: rados osd op timeout = 900 2024-03-28T14:07:00.691 INFO:teuthology.orchestra.run.smithi171.stdout:[mon.a] 2024-03-28T14:07:00.691 INFO:teuthology.orchestra.run.smithi171.stdout:[mon.b] 2024-03-28T14:07:00.691 INFO:teuthology.orchestra.run.smithi171.stdout:[mon.c] 2024-03-28T14:07:00.691 INFO:teuthology.orchestra.run.smithi171.stdout:[mds.a] 2024-03-28T14:07:00.691 INFO:teuthology.orchestra.run.smithi171.stdout:[mds.b] 2024-03-28T14:07:00.692 INFO:teuthology.orchestra.run.smithi171.stdout:[mds] 2024-03-28T14:07:00.692 INFO:teuthology.orchestra.run.smithi171.stdout: debug mds = 20 2024-03-28T14:07:00.692 INFO:teuthology.orchestra.run.smithi171.stdout: debug mds balancer = 20 2024-03-28T14:07:00.692 INFO:teuthology.orchestra.run.smithi171.stdout: debug ms = 1 2024-03-28T14:07:00.692 INFO:teuthology.orchestra.run.smithi171.stdout: mds debug frag = True 2024-03-28T14:07:00.692 INFO:teuthology.orchestra.run.smithi171.stdout: mds debug scatterstat = True 2024-03-28T14:07:00.692 INFO:teuthology.orchestra.run.smithi171.stdout: mds op complaint time = 180 2024-03-28T14:07:00.692 INFO:teuthology.orchestra.run.smithi171.stdout: mds verify scatter = True 2024-03-28T14:07:00.692 INFO:teuthology.orchestra.run.smithi171.stdout: osd op complaint time = 180 2024-03-28T14:07:00.693 INFO:teuthology.orchestra.run.smithi171.stdout: rados mon op timeout = 900 2024-03-28T14:07:00.693 INFO:teuthology.orchestra.run.smithi171.stdout: rados osd op timeout = 900 2024-03-28T14:07:00.693 INFO:tasks.ceph:Creating admin key on mon.a... 2024-03-28T14:07:00.693 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2024-03-28T14:07:00.788 INFO:tasks.ceph:Copying monmap to all nodes... 2024-03-28T14:07:00.788 DEBUG:teuthology.orchestra.run.smithi104:> set -ex 2024-03-28T14:07:00.788 DEBUG:teuthology.orchestra.run.smithi104:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2024-03-28T14:07:00.835 DEBUG:teuthology.orchestra.run.smithi104:> set -ex 2024-03-28T14:07:00.835 DEBUG:teuthology.orchestra.run.smithi104:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2024-03-28T14:07:00.887 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi104.front.sepia.ceph.com 2024-03-28T14:07:00.887 DEBUG:teuthology.orchestra.run.smithi104:> set -ex 2024-03-28T14:07:00.887 DEBUG:teuthology.orchestra.run.smithi104:> sudo dd of=/etc/ceph/ceph.keyring 2024-03-28T14:07:00.888 DEBUG:teuthology.orchestra.run.smithi104:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-03-28T14:07:00.951 DEBUG:teuthology.orchestra.run.smithi104:> set -ex 2024-03-28T14:07:00.952 DEBUG:teuthology.orchestra.run.smithi104:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-03-28T14:07:01.011 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi171.front.sepia.ceph.com 2024-03-28T14:07:01.011 DEBUG:teuthology.orchestra.run.smithi171:> set -ex 2024-03-28T14:07:01.011 DEBUG:teuthology.orchestra.run.smithi171:> sudo dd of=/etc/ceph/ceph.keyring 2024-03-28T14:07:01.011 DEBUG:teuthology.orchestra.run.smithi171:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-03-28T14:07:01.035 DEBUG:teuthology.orchestra.run.smithi171:> set -ex 2024-03-28T14:07:01.035 DEBUG:teuthology.orchestra.run.smithi171:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-03-28T14:07:01.092 INFO:tasks.ceph:Setting up mon nodes... 2024-03-28T14:07:01.093 INFO:tasks.ceph:Setting up mgr nodes... 2024-03-28T14:07:01.093 DEBUG:teuthology.orchestra.run.smithi104:> sudo mkdir -p /var/lib/ceph/mgr/ceph-y && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.y /var/lib/ceph/mgr/ceph-y/keyring 2024-03-28T14:07:01.139 INFO:teuthology.orchestra.run.smithi104.stdout:creating /var/lib/ceph/mgr/ceph-y/keyring 2024-03-28T14:07:01.141 DEBUG:teuthology.orchestra.run.smithi171:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2024-03-28T14:07:01.183 INFO:teuthology.orchestra.run.smithi171.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2024-03-28T14:07:01.194 INFO:tasks.ceph:Setting up mds nodes... 2024-03-28T14:07:01.195 DEBUG:teuthology.orchestra.run.smithi104:> sudo mkdir -p /var/lib/ceph/mds/ceph-a && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.a /var/lib/ceph/mds/ceph-a/keyring 2024-03-28T14:07:01.235 INFO:teuthology.orchestra.run.smithi104.stdout:creating /var/lib/ceph/mds/ceph-a/keyring 2024-03-28T14:07:01.237 DEBUG:teuthology.orchestra.run.smithi104:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-a 2024-03-28T14:07:01.291 DEBUG:teuthology.orchestra.run.smithi171:> sudo mkdir -p /var/lib/ceph/mds/ceph-b && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.b /var/lib/ceph/mds/ceph-b/keyring 2024-03-28T14:07:01.338 INFO:teuthology.orchestra.run.smithi171.stdout:creating /var/lib/ceph/mds/ceph-b/keyring 2024-03-28T14:07:01.341 DEBUG:teuthology.orchestra.run.smithi171:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-b 2024-03-28T14:07:01.398 INFO:tasks.ceph_client:Setting up client nodes... 2024-03-28T14:07:01.398 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-03-28T14:07:01.435 INFO:teuthology.orchestra.run.smithi104.stdout:creating /etc/ceph/ceph.client.0.keyring 2024-03-28T14:07:01.446 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-03-28T14:07:01.446 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi104.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi171.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-03-28T14:07:01.446 DEBUG:teuthology.orchestra.run.smithi104:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2024-03-28T14:07:01.504 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-28T14:07:01.504 INFO:tasks.ceph:role: osd.0 2024-03-28T14:07:01.504 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi104.front.sepia.ceph.com 2024-03-28T14:07:01.505 DEBUG:teuthology.orchestra.run.smithi104:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-03-28T14:07:01.572 INFO:teuthology.orchestra.run.smithi104.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-03-28T14:07:01.573 INFO:teuthology.orchestra.run.smithi104.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-28T14:07:01.573 INFO:teuthology.orchestra.run.smithi104.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-28T14:07:01.573 INFO:teuthology.orchestra.run.smithi104.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-03-28T14:07:01.573 INFO:teuthology.orchestra.run.smithi104.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-28T14:07:01.573 INFO:teuthology.orchestra.run.smithi104.stdout: = sunit=0 swidth=0 blks 2024-03-28T14:07:01.573 INFO:teuthology.orchestra.run.smithi104.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-28T14:07:01.573 INFO:teuthology.orchestra.run.smithi104.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-03-28T14:07:01.573 INFO:teuthology.orchestra.run.smithi104.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-28T14:07:01.573 INFO:teuthology.orchestra.run.smithi104.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-28T14:07:02.048 INFO:teuthology.orchestra.run.smithi104.stdout:Discarding blocks...Done. 2024-03-28T14:07:02.049 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi104.front.sepia.ceph.com -o noatime 2024-03-28T14:07:02.049 DEBUG:teuthology.orchestra.run.smithi104:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2024-03-28T14:07:02.220 DEBUG:teuthology.orchestra.run.smithi104:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2024-03-28T14:07:02.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-28T14:07:02.236 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: /sbin/restorecon: command not found 2024-03-28T14:07:02.237 DEBUG:teuthology.orchestra.run.smithi104:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2024-03-28T14:07:02.298 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-28T14:07:02.299 INFO:tasks.ceph:role: osd.1 2024-03-28T14:07:02.299 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi104.front.sepia.ceph.com 2024-03-28T14:07:02.299 DEBUG:teuthology.orchestra.run.smithi104:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-03-28T14:07:02.362 INFO:teuthology.orchestra.run.smithi104.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-03-28T14:07:02.363 INFO:teuthology.orchestra.run.smithi104.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-28T14:07:02.363 INFO:teuthology.orchestra.run.smithi104.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-28T14:07:02.363 INFO:teuthology.orchestra.run.smithi104.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-03-28T14:07:02.363 INFO:teuthology.orchestra.run.smithi104.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-28T14:07:02.363 INFO:teuthology.orchestra.run.smithi104.stdout: = sunit=0 swidth=0 blks 2024-03-28T14:07:02.363 INFO:teuthology.orchestra.run.smithi104.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-28T14:07:02.363 INFO:teuthology.orchestra.run.smithi104.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-03-28T14:07:02.363 INFO:teuthology.orchestra.run.smithi104.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-28T14:07:02.363 INFO:teuthology.orchestra.run.smithi104.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-28T14:07:02.810 INFO:teuthology.orchestra.run.smithi104.stdout:Discarding blocks...Done. 2024-03-28T14:07:02.812 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi104.front.sepia.ceph.com -o noatime 2024-03-28T14:07:02.812 DEBUG:teuthology.orchestra.run.smithi104:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2024-03-28T14:07:02.896 DEBUG:teuthology.orchestra.run.smithi104:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2024-03-28T14:07:02.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-28T14:07:02.949 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: /sbin/restorecon: command not found 2024-03-28T14:07:02.950 DEBUG:teuthology.orchestra.run.smithi104:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2024-03-28T14:07:03.002 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-28T14:07:03.003 INFO:tasks.ceph:role: osd.2 2024-03-28T14:07:03.003 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi104.front.sepia.ceph.com 2024-03-28T14:07:03.003 DEBUG:teuthology.orchestra.run.smithi104:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-03-28T14:07:03.065 INFO:teuthology.orchestra.run.smithi104.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-03-28T14:07:03.066 INFO:teuthology.orchestra.run.smithi104.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-28T14:07:03.066 INFO:teuthology.orchestra.run.smithi104.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-28T14:07:03.066 INFO:teuthology.orchestra.run.smithi104.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-03-28T14:07:03.066 INFO:teuthology.orchestra.run.smithi104.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-28T14:07:03.066 INFO:teuthology.orchestra.run.smithi104.stdout: = sunit=0 swidth=0 blks 2024-03-28T14:07:03.066 INFO:teuthology.orchestra.run.smithi104.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-28T14:07:03.066 INFO:teuthology.orchestra.run.smithi104.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-03-28T14:07:03.066 INFO:teuthology.orchestra.run.smithi104.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-28T14:07:03.066 INFO:teuthology.orchestra.run.smithi104.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-28T14:07:03.513 INFO:teuthology.orchestra.run.smithi104.stdout:Discarding blocks...Done. 2024-03-28T14:07:03.515 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi104.front.sepia.ceph.com -o noatime 2024-03-28T14:07:03.515 DEBUG:teuthology.orchestra.run.smithi104:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2024-03-28T14:07:03.655 DEBUG:teuthology.orchestra.run.smithi104:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2024-03-28T14:07:03.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-28T14:07:03.666 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: /sbin/restorecon: command not found 2024-03-28T14:07:03.667 DEBUG:teuthology.orchestra.run.smithi104:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2024-03-28T14:07:03.723 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-28T14:07:03.723 INFO:tasks.ceph:role: osd.3 2024-03-28T14:07:03.723 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi104.front.sepia.ceph.com 2024-03-28T14:07:03.723 DEBUG:teuthology.orchestra.run.smithi104:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-03-28T14:07:03.787 INFO:teuthology.orchestra.run.smithi104.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-03-28T14:07:03.787 INFO:teuthology.orchestra.run.smithi104.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-28T14:07:03.787 INFO:teuthology.orchestra.run.smithi104.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-28T14:07:03.787 INFO:teuthology.orchestra.run.smithi104.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-03-28T14:07:03.787 INFO:teuthology.orchestra.run.smithi104.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-28T14:07:03.787 INFO:teuthology.orchestra.run.smithi104.stdout: = sunit=0 swidth=0 blks 2024-03-28T14:07:03.787 INFO:teuthology.orchestra.run.smithi104.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-28T14:07:03.788 INFO:teuthology.orchestra.run.smithi104.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-03-28T14:07:03.788 INFO:teuthology.orchestra.run.smithi104.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-28T14:07:03.788 INFO:teuthology.orchestra.run.smithi104.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-28T14:07:04.289 INFO:teuthology.orchestra.run.smithi104.stdout:Discarding blocks...Done. 2024-03-28T14:07:04.291 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi104.front.sepia.ceph.com -o noatime 2024-03-28T14:07:04.292 DEBUG:teuthology.orchestra.run.smithi104:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-3 2024-03-28T14:07:04.431 DEBUG:teuthology.orchestra.run.smithi104:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2024-03-28T14:07:04.441 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: /sbin/restorecon: command not found 2024-03-28T14:07:04.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-28T14:07:04.443 DEBUG:teuthology.orchestra.run.smithi104:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-28T14:07:04.558 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:04.561+0000 7f3aa8b547c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2024-03-28T14:07:04.564 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:04.565+0000 7f3aa8b547c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2024-03-28T14:07:04.564 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:04.565+0000 7f3aa8b547c0 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-03-28T14:07:04.564 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:04.565+0000 7f3aa8b547c0 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-03-28T14:07:04.564 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:04.565+0000 7f3aa8b547c0 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-03-28T14:07:04.564 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:04.565+0000 7f3aa8b547c0 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2024-03-28T14:07:07.563 DEBUG:teuthology.orchestra.run.smithi104:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-03-28T14:07:07.577 DEBUG:teuthology.orchestra.run.smithi104:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-28T14:07:07.657 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:07.657+0000 7f54403fa7c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2024-03-28T14:07:07.658 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:07.657+0000 7f54403fa7c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2024-03-28T14:07:07.658 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:07.657+0000 7f54403fa7c0 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-03-28T14:07:07.658 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:07.657+0000 7f54403fa7c0 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-03-28T14:07:07.658 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:07.657+0000 7f54403fa7c0 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-03-28T14:07:07.658 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:07.661+0000 7f54403fa7c0 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2024-03-28T14:07:10.658 DEBUG:teuthology.orchestra.run.smithi104:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-03-28T14:07:10.716 DEBUG:teuthology.orchestra.run.smithi104:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-28T14:07:10.797 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:10.797+0000 7f3e88efa7c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2024-03-28T14:07:10.797 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:10.797+0000 7f3e88efa7c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2024-03-28T14:07:10.798 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:10.797+0000 7f3e88efa7c0 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-03-28T14:07:10.798 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:10.797+0000 7f3e88efa7c0 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-03-28T14:07:10.798 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:10.797+0000 7f3e88efa7c0 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-03-28T14:07:10.798 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:10.797+0000 7f3e88efa7c0 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2024-03-28T14:07:13.765 DEBUG:teuthology.orchestra.run.smithi104:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-03-28T14:07:13.818 DEBUG:teuthology.orchestra.run.smithi104:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-28T14:07:13.899 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:13.901+0000 7ff49677a7c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2024-03-28T14:07:13.900 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:13.901+0000 7ff49677a7c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2024-03-28T14:07:13.900 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:13.901+0000 7ff49677a7c0 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-03-28T14:07:13.900 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:13.901+0000 7ff49677a7c0 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-03-28T14:07:13.900 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:13.901+0000 7ff49677a7c0 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-03-28T14:07:13.901 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:13.901+0000 7ff49677a7c0 -1 bluestore(/var/lib/ceph/osd/ceph-3) _read_fsid unparsable uuid 2024-03-28T14:07:16.935 DEBUG:teuthology.orchestra.run.smithi104:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-03-28T14:07:16.996 DEBUG:teuthology.orchestra.run.smithi171:> sudo mkdir -p /var/lib/ceph/osd/ceph-4 2024-03-28T14:07:17.011 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-28T14:07:17.011 INFO:tasks.ceph:role: osd.4 2024-03-28T14:07:17.011 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi171.front.sepia.ceph.com 2024-03-28T14:07:17.011 DEBUG:teuthology.orchestra.run.smithi171:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-03-28T14:07:17.075 INFO:teuthology.orchestra.run.smithi171.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-03-28T14:07:17.075 INFO:teuthology.orchestra.run.smithi171.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-28T14:07:17.075 INFO:teuthology.orchestra.run.smithi171.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-28T14:07:17.075 INFO:teuthology.orchestra.run.smithi171.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-03-28T14:07:17.075 INFO:teuthology.orchestra.run.smithi171.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-28T14:07:17.075 INFO:teuthology.orchestra.run.smithi171.stdout: = sunit=0 swidth=0 blks 2024-03-28T14:07:17.075 INFO:teuthology.orchestra.run.smithi171.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-28T14:07:17.075 INFO:teuthology.orchestra.run.smithi171.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-03-28T14:07:17.076 INFO:teuthology.orchestra.run.smithi171.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-28T14:07:17.076 INFO:teuthology.orchestra.run.smithi171.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-28T14:07:17.558 INFO:teuthology.orchestra.run.smithi171.stdout:Discarding blocks...Done. 2024-03-28T14:07:17.560 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi171.front.sepia.ceph.com -o noatime 2024-03-28T14:07:17.560 DEBUG:teuthology.orchestra.run.smithi171:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-4 2024-03-28T14:07:17.685 DEBUG:teuthology.orchestra.run.smithi171:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-4 2024-03-28T14:07:17.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-28T14:07:17.696 INFO:teuthology.orchestra.run.smithi171.stderr:sudo: /sbin/restorecon: command not found 2024-03-28T14:07:17.697 DEBUG:teuthology.orchestra.run.smithi171:> sudo mkdir -p /var/lib/ceph/osd/ceph-5 2024-03-28T14:07:17.751 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-28T14:07:17.751 INFO:tasks.ceph:role: osd.5 2024-03-28T14:07:17.752 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi171.front.sepia.ceph.com 2024-03-28T14:07:17.752 DEBUG:teuthology.orchestra.run.smithi171:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-03-28T14:07:17.816 INFO:teuthology.orchestra.run.smithi171.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-03-28T14:07:17.816 INFO:teuthology.orchestra.run.smithi171.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-28T14:07:17.816 INFO:teuthology.orchestra.run.smithi171.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-28T14:07:17.816 INFO:teuthology.orchestra.run.smithi171.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-03-28T14:07:17.816 INFO:teuthology.orchestra.run.smithi171.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-28T14:07:17.816 INFO:teuthology.orchestra.run.smithi171.stdout: = sunit=0 swidth=0 blks 2024-03-28T14:07:17.816 INFO:teuthology.orchestra.run.smithi171.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-28T14:07:17.816 INFO:teuthology.orchestra.run.smithi171.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-03-28T14:07:17.816 INFO:teuthology.orchestra.run.smithi171.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-28T14:07:17.816 INFO:teuthology.orchestra.run.smithi171.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-28T14:07:18.244 INFO:teuthology.orchestra.run.smithi171.stdout:Discarding blocks...Done. 2024-03-28T14:07:18.246 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi171.front.sepia.ceph.com -o noatime 2024-03-28T14:07:18.246 DEBUG:teuthology.orchestra.run.smithi171:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-5 2024-03-28T14:07:18.309 DEBUG:teuthology.orchestra.run.smithi171:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-5 2024-03-28T14:07:18.361 INFO:teuthology.orchestra.run.smithi171.stderr:sudo: /sbin/restorecon: command not found 2024-03-28T14:07:18.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-28T14:07:18.362 DEBUG:teuthology.orchestra.run.smithi171:> sudo mkdir -p /var/lib/ceph/osd/ceph-6 2024-03-28T14:07:18.414 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-28T14:07:18.415 INFO:tasks.ceph:role: osd.6 2024-03-28T14:07:18.415 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi171.front.sepia.ceph.com 2024-03-28T14:07:18.415 DEBUG:teuthology.orchestra.run.smithi171:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-03-28T14:07:18.476 INFO:teuthology.orchestra.run.smithi171.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-03-28T14:07:18.476 INFO:teuthology.orchestra.run.smithi171.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-28T14:07:18.476 INFO:teuthology.orchestra.run.smithi171.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-28T14:07:18.476 INFO:teuthology.orchestra.run.smithi171.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-03-28T14:07:18.476 INFO:teuthology.orchestra.run.smithi171.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-28T14:07:18.476 INFO:teuthology.orchestra.run.smithi171.stdout: = sunit=0 swidth=0 blks 2024-03-28T14:07:18.476 INFO:teuthology.orchestra.run.smithi171.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-28T14:07:18.476 INFO:teuthology.orchestra.run.smithi171.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-03-28T14:07:18.476 INFO:teuthology.orchestra.run.smithi171.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-28T14:07:18.477 INFO:teuthology.orchestra.run.smithi171.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-28T14:07:18.951 INFO:teuthology.orchestra.run.smithi171.stdout:Discarding blocks...Done. 2024-03-28T14:07:18.953 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi171.front.sepia.ceph.com -o noatime 2024-03-28T14:07:18.953 DEBUG:teuthology.orchestra.run.smithi171:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-6 2024-03-28T14:07:19.041 DEBUG:teuthology.orchestra.run.smithi171:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-6 2024-03-28T14:07:19.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-28T14:07:19.097 INFO:teuthology.orchestra.run.smithi171.stderr:sudo: /sbin/restorecon: command not found 2024-03-28T14:07:19.097 DEBUG:teuthology.orchestra.run.smithi171:> sudo mkdir -p /var/lib/ceph/osd/ceph-7 2024-03-28T14:07:19.149 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-28T14:07:19.149 INFO:tasks.ceph:role: osd.7 2024-03-28T14:07:19.150 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi171.front.sepia.ceph.com 2024-03-28T14:07:19.150 DEBUG:teuthology.orchestra.run.smithi171:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-03-28T14:07:19.209 INFO:teuthology.orchestra.run.smithi171.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-03-28T14:07:19.209 INFO:teuthology.orchestra.run.smithi171.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-28T14:07:19.209 INFO:teuthology.orchestra.run.smithi171.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-28T14:07:19.209 INFO:teuthology.orchestra.run.smithi171.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-03-28T14:07:19.209 INFO:teuthology.orchestra.run.smithi171.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-28T14:07:19.209 INFO:teuthology.orchestra.run.smithi171.stdout: = sunit=0 swidth=0 blks 2024-03-28T14:07:19.209 INFO:teuthology.orchestra.run.smithi171.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-28T14:07:19.209 INFO:teuthology.orchestra.run.smithi171.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-03-28T14:07:19.209 INFO:teuthology.orchestra.run.smithi171.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-28T14:07:19.209 INFO:teuthology.orchestra.run.smithi171.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-28T14:07:19.703 INFO:teuthology.orchestra.run.smithi171.stdout:Discarding blocks...Done. 2024-03-28T14:07:19.705 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi171.front.sepia.ceph.com -o noatime 2024-03-28T14:07:19.705 DEBUG:teuthology.orchestra.run.smithi171:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-7 2024-03-28T14:07:19.847 DEBUG:teuthology.orchestra.run.smithi171:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-7 2024-03-28T14:07:19.858 INFO:teuthology.orchestra.run.smithi171.stderr:sudo: /sbin/restorecon: command not found 2024-03-28T14:07:19.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-28T14:07:19.859 DEBUG:teuthology.orchestra.run.smithi171:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 4 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-28T14:07:19.973 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:19.971+0000 7fdb2b3c27c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-4/keyring: can't open /var/lib/ceph/osd/ceph-4/keyring: (2) No such file or directory 2024-03-28T14:07:19.976 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:19.975+0000 7fdb2b3c27c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-4/keyring 2024-03-28T14:07:19.976 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:19.975+0000 7fdb2b3c27c0 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-03-28T14:07:19.977 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:19.975+0000 7fdb2b3c27c0 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-03-28T14:07:19.977 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:19.975+0000 7fdb2b3c27c0 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-03-28T14:07:19.986 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:19.983+0000 7fdb2b3c27c0 -1 bluestore(/var/lib/ceph/osd/ceph-4) _read_fsid unparsable uuid 2024-03-28T14:07:22.960 DEBUG:teuthology.orchestra.run.smithi171:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-03-28T14:07:22.973 DEBUG:teuthology.orchestra.run.smithi171:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 5 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-28T14:07:23.053 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:23.051+0000 7fc4b57287c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-5/keyring: can't open /var/lib/ceph/osd/ceph-5/keyring: (2) No such file or directory 2024-03-28T14:07:23.054 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:23.051+0000 7fc4b57287c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-5/keyring 2024-03-28T14:07:23.054 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:23.051+0000 7fc4b57287c0 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-03-28T14:07:23.054 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:23.051+0000 7fc4b57287c0 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-03-28T14:07:23.054 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:23.051+0000 7fc4b57287c0 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-03-28T14:07:23.054 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:23.051+0000 7fc4b57287c0 -1 bluestore(/var/lib/ceph/osd/ceph-5) _read_fsid unparsable uuid 2024-03-28T14:07:26.045 DEBUG:teuthology.orchestra.run.smithi171:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-03-28T14:07:26.100 DEBUG:teuthology.orchestra.run.smithi171:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 6 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-28T14:07:26.182 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:26.179+0000 7fad0d0a37c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-6/keyring: can't open /var/lib/ceph/osd/ceph-6/keyring: (2) No such file or directory 2024-03-28T14:07:26.183 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:26.179+0000 7fad0d0a37c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-6/keyring 2024-03-28T14:07:26.183 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:26.179+0000 7fad0d0a37c0 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-03-28T14:07:26.183 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:26.179+0000 7fad0d0a37c0 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-03-28T14:07:26.184 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:26.179+0000 7fad0d0a37c0 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-03-28T14:07:26.184 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:26.183+0000 7fad0d0a37c0 -1 bluestore(/var/lib/ceph/osd/ceph-6) _read_fsid unparsable uuid 2024-03-28T14:07:28.789 DEBUG:teuthology.orchestra.run.smithi104:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-28T14:07:28.792 DEBUG:teuthology.orchestra.run.smithi171:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-28T14:07:29.159 DEBUG:teuthology.orchestra.run.smithi171:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-03-28T14:07:29.172 DEBUG:teuthology.orchestra.run.smithi171:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 7 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-28T14:07:29.253 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:29.251+0000 7f9d2144d7c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-7/keyring: can't open /var/lib/ceph/osd/ceph-7/keyring: (2) No such file or directory 2024-03-28T14:07:29.253 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:29.251+0000 7f9d2144d7c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-7/keyring 2024-03-28T14:07:29.253 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:29.251+0000 7f9d2144d7c0 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-03-28T14:07:29.253 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:29.251+0000 7f9d2144d7c0 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-03-28T14:07:29.253 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:29.251+0000 7f9d2144d7c0 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-03-28T14:07:29.254 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:29.251+0000 7f9d2144d7c0 -1 bluestore(/var/lib/ceph/osd/ceph-7) _read_fsid unparsable uuid 2024-03-28T14:07:32.226 DEBUG:teuthology.orchestra.run.smithi171:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-03-28T14:07:32.284 INFO:tasks.ceph:Reading keys from all nodes... 2024-03-28T14:07:32.284 DEBUG:teuthology.orchestra.run.smithi104:> set -ex 2024-03-28T14:07:32.284 DEBUG:teuthology.orchestra.run.smithi104:> sudo dd if=/var/lib/ceph/mgr/ceph-y/keyring of=/dev/stdout 2024-03-28T14:07:32.299 DEBUG:teuthology.orchestra.run.smithi104:> set -ex 2024-03-28T14:07:32.299 DEBUG:teuthology.orchestra.run.smithi104:> sudo dd if=/var/lib/ceph/mds/ceph-a/keyring of=/dev/stdout 2024-03-28T14:07:32.355 DEBUG:teuthology.orchestra.run.smithi104:> set -ex 2024-03-28T14:07:32.355 DEBUG:teuthology.orchestra.run.smithi104:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2024-03-28T14:07:32.410 DEBUG:teuthology.orchestra.run.smithi104:> set -ex 2024-03-28T14:07:32.410 DEBUG:teuthology.orchestra.run.smithi104:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2024-03-28T14:07:32.466 DEBUG:teuthology.orchestra.run.smithi104:> set -ex 2024-03-28T14:07:32.466 DEBUG:teuthology.orchestra.run.smithi104:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2024-03-28T14:07:32.523 DEBUG:teuthology.orchestra.run.smithi104:> set -ex 2024-03-28T14:07:32.523 DEBUG:teuthology.orchestra.run.smithi104:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2024-03-28T14:07:32.578 DEBUG:teuthology.orchestra.run.smithi171:> set -ex 2024-03-28T14:07:32.579 DEBUG:teuthology.orchestra.run.smithi171:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2024-03-28T14:07:32.593 DEBUG:teuthology.orchestra.run.smithi171:> set -ex 2024-03-28T14:07:32.593 DEBUG:teuthology.orchestra.run.smithi171:> sudo dd if=/var/lib/ceph/mds/ceph-b/keyring of=/dev/stdout 2024-03-28T14:07:32.652 DEBUG:teuthology.orchestra.run.smithi171:> set -ex 2024-03-28T14:07:32.652 DEBUG:teuthology.orchestra.run.smithi171:> sudo dd if=/var/lib/ceph/osd/ceph-4/keyring of=/dev/stdout 2024-03-28T14:07:32.709 DEBUG:teuthology.orchestra.run.smithi171:> set -ex 2024-03-28T14:07:32.709 DEBUG:teuthology.orchestra.run.smithi171:> sudo dd if=/var/lib/ceph/osd/ceph-5/keyring of=/dev/stdout 2024-03-28T14:07:32.766 DEBUG:teuthology.orchestra.run.smithi171:> set -ex 2024-03-28T14:07:32.766 DEBUG:teuthology.orchestra.run.smithi171:> sudo dd if=/var/lib/ceph/osd/ceph-6/keyring of=/dev/stdout 2024-03-28T14:07:32.820 DEBUG:teuthology.orchestra.run.smithi171:> set -ex 2024-03-28T14:07:32.820 DEBUG:teuthology.orchestra.run.smithi171:> sudo dd if=/var/lib/ceph/osd/ceph-7/keyring of=/dev/stdout 2024-03-28T14:07:32.878 DEBUG:teuthology.orchestra.run.smithi104:> set -ex 2024-03-28T14:07:32.878 DEBUG:teuthology.orchestra.run.smithi104:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2024-03-28T14:07:32.884 INFO:tasks.ceph:Adding keys to all mons... 2024-03-28T14:07:32.885 DEBUG:teuthology.orchestra.run.smithi104:> sudo tee -a /etc/ceph/ceph.keyring 2024-03-28T14:07:32.929 DEBUG:teuthology.orchestra.run.smithi171:> sudo tee -a /etc/ceph/ceph.keyring 2024-03-28T14:07:32.938 INFO:teuthology.orchestra.run.smithi104.stdout:[mgr.y] 2024-03-28T14:07:32.939 INFO:teuthology.orchestra.run.smithi104.stdout: key = AQCFeQVmIL12CBAAuJ8i8ChzCtQXwKFxiyzaRg== 2024-03-28T14:07:32.939 INFO:teuthology.orchestra.run.smithi104.stdout:[mds.a] 2024-03-28T14:07:32.939 INFO:teuthology.orchestra.run.smithi104.stdout: key = AQCFeQVmR5QyDhAAi9PSbldc6n0vgJYojFBc7w== 2024-03-28T14:07:32.939 INFO:teuthology.orchestra.run.smithi104.stdout:[osd.0] 2024-03-28T14:07:32.939 INFO:teuthology.orchestra.run.smithi104.stdout: key = AQCIeQVm8wh8IRAArenUIzBmf88Ujjqo019SbA== 2024-03-28T14:07:32.939 INFO:teuthology.orchestra.run.smithi104.stdout:[osd.1] 2024-03-28T14:07:32.939 INFO:teuthology.orchestra.run.smithi104.stdout: key = AQCLeQVmcj1ZJxAAyxCHsIh7x2ZT1LcQtV4efg== 2024-03-28T14:07:32.940 INFO:teuthology.orchestra.run.smithi104.stdout:[osd.2] 2024-03-28T14:07:32.940 INFO:teuthology.orchestra.run.smithi104.stdout: key = AQCOeQVmf3ixLxAAbuuAonIliKXnzp2IMc8gdg== 2024-03-28T14:07:32.940 INFO:teuthology.orchestra.run.smithi104.stdout:[osd.3] 2024-03-28T14:07:32.940 INFO:teuthology.orchestra.run.smithi104.stdout: key = AQCReQVmB9LPNRAAfUx+PjggnlhW1PIwHt5/Vw== 2024-03-28T14:07:32.940 INFO:teuthology.orchestra.run.smithi104.stdout:[mgr.x] 2024-03-28T14:07:32.940 INFO:teuthology.orchestra.run.smithi104.stdout: key = AQCFeQVmsescCxAA6RH0UHxvAzTjUsxKGN141g== 2024-03-28T14:07:32.940 INFO:teuthology.orchestra.run.smithi104.stdout:[mds.b] 2024-03-28T14:07:32.940 INFO:teuthology.orchestra.run.smithi104.stdout: key = AQCFeQVm9M5TFBAAP3Vf8lLVq8X5bUs8i6Ta0A== 2024-03-28T14:07:32.940 INFO:teuthology.orchestra.run.smithi104.stdout:[osd.4] 2024-03-28T14:07:32.940 INFO:teuthology.orchestra.run.smithi104.stdout: key = AQCXeQVmd+42OhAAERCUxkaXDdi1vpiHnyL2Rw== 2024-03-28T14:07:32.940 INFO:teuthology.orchestra.run.smithi104.stdout:[osd.5] 2024-03-28T14:07:32.940 INFO:teuthology.orchestra.run.smithi104.stdout: key = AQCbeQVm0eVgAxAAgQIiw5t2P7w6kNIiO0b+SA== 2024-03-28T14:07:32.940 INFO:teuthology.orchestra.run.smithi104.stdout:[osd.6] 2024-03-28T14:07:32.941 INFO:teuthology.orchestra.run.smithi104.stdout: key = AQCeeQVmJX0SCxAARorbp1LQBwG7Ro7YZvDM4Q== 2024-03-28T14:07:32.941 INFO:teuthology.orchestra.run.smithi104.stdout:[osd.7] 2024-03-28T14:07:32.941 INFO:teuthology.orchestra.run.smithi104.stdout: key = AQCheQVmQwxIDxAAKNWebD4FNjtqCSP12WLAKw== 2024-03-28T14:07:32.941 INFO:teuthology.orchestra.run.smithi104.stdout:[client.0] 2024-03-28T14:07:32.941 INFO:teuthology.orchestra.run.smithi104.stdout: key = AQCFeQVmkAgiGhAAUVl6HwWo1xt2L4bwVt1VWw== 2024-03-28T14:07:32.942 INFO:teuthology.orchestra.run.smithi171.stdout:[mgr.y] 2024-03-28T14:07:32.942 INFO:teuthology.orchestra.run.smithi171.stdout: key = AQCFeQVmIL12CBAAuJ8i8ChzCtQXwKFxiyzaRg== 2024-03-28T14:07:32.942 INFO:teuthology.orchestra.run.smithi171.stdout:[mds.a] 2024-03-28T14:07:32.942 INFO:teuthology.orchestra.run.smithi171.stdout: key = AQCFeQVmR5QyDhAAi9PSbldc6n0vgJYojFBc7w== 2024-03-28T14:07:32.942 INFO:teuthology.orchestra.run.smithi171.stdout:[osd.0] 2024-03-28T14:07:32.942 INFO:teuthology.orchestra.run.smithi171.stdout: key = AQCIeQVm8wh8IRAArenUIzBmf88Ujjqo019SbA== 2024-03-28T14:07:32.942 INFO:teuthology.orchestra.run.smithi171.stdout:[osd.1] 2024-03-28T14:07:32.942 INFO:teuthology.orchestra.run.smithi171.stdout: key = AQCLeQVmcj1ZJxAAyxCHsIh7x2ZT1LcQtV4efg== 2024-03-28T14:07:32.943 INFO:teuthology.orchestra.run.smithi171.stdout:[osd.2] 2024-03-28T14:07:32.943 INFO:teuthology.orchestra.run.smithi171.stdout: key = AQCOeQVmf3ixLxAAbuuAonIliKXnzp2IMc8gdg== 2024-03-28T14:07:32.943 INFO:teuthology.orchestra.run.smithi171.stdout:[osd.3] 2024-03-28T14:07:32.943 INFO:teuthology.orchestra.run.smithi171.stdout: key = AQCReQVmB9LPNRAAfUx+PjggnlhW1PIwHt5/Vw== 2024-03-28T14:07:32.943 INFO:teuthology.orchestra.run.smithi171.stdout:[mgr.x] 2024-03-28T14:07:32.943 INFO:teuthology.orchestra.run.smithi171.stdout: key = AQCFeQVmsescCxAA6RH0UHxvAzTjUsxKGN141g== 2024-03-28T14:07:32.943 INFO:teuthology.orchestra.run.smithi171.stdout:[mds.b] 2024-03-28T14:07:32.943 INFO:teuthology.orchestra.run.smithi171.stdout: key = AQCFeQVm9M5TFBAAP3Vf8lLVq8X5bUs8i6Ta0A== 2024-03-28T14:07:32.943 INFO:teuthology.orchestra.run.smithi171.stdout:[osd.4] 2024-03-28T14:07:32.943 INFO:teuthology.orchestra.run.smithi171.stdout: key = AQCXeQVmd+42OhAAERCUxkaXDdi1vpiHnyL2Rw== 2024-03-28T14:07:32.943 INFO:teuthology.orchestra.run.smithi171.stdout:[osd.5] 2024-03-28T14:07:32.944 INFO:teuthology.orchestra.run.smithi171.stdout: key = AQCbeQVm0eVgAxAAgQIiw5t2P7w6kNIiO0b+SA== 2024-03-28T14:07:32.944 INFO:teuthology.orchestra.run.smithi171.stdout:[osd.6] 2024-03-28T14:07:32.944 INFO:teuthology.orchestra.run.smithi171.stdout: key = AQCeeQVmJX0SCxAARorbp1LQBwG7Ro7YZvDM4Q== 2024-03-28T14:07:32.944 INFO:teuthology.orchestra.run.smithi171.stdout:[osd.7] 2024-03-28T14:07:32.944 INFO:teuthology.orchestra.run.smithi171.stdout: key = AQCheQVmQwxIDxAAKNWebD4FNjtqCSP12WLAKw== 2024-03-28T14:07:32.944 INFO:teuthology.orchestra.run.smithi171.stdout:[client.0] 2024-03-28T14:07:32.944 INFO:teuthology.orchestra.run.smithi171.stdout: key = AQCFeQVmkAgiGhAAUVl6HwWo1xt2L4bwVt1VWw== 2024-03-28T14:07:32.946 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-28T14:07:32.989 DEBUG:teuthology.orchestra.run.smithi171:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-28T14:07:33.035 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.a --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-03-28T14:07:33.077 DEBUG:teuthology.orchestra.run.smithi171:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.a --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-03-28T14:07:33.128 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-28T14:07:33.164 DEBUG:teuthology.orchestra.run.smithi171:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-28T14:07:33.207 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-28T14:07:33.209 DEBUG:teuthology.orchestra.run.smithi171:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-28T14:07:33.291 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-28T14:07:33.293 DEBUG:teuthology.orchestra.run.smithi171:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-28T14:07:33.375 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-28T14:07:33.377 DEBUG:teuthology.orchestra.run.smithi171:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-28T14:07:33.454 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-28T14:07:33.460 DEBUG:teuthology.orchestra.run.smithi171:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-28T14:07:33.538 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.b --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-03-28T14:07:33.541 DEBUG:teuthology.orchestra.run.smithi171:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.b --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-03-28T14:07:33.626 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-28T14:07:33.628 DEBUG:teuthology.orchestra.run.smithi171:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-28T14:07:33.667 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-28T14:07:33.709 DEBUG:teuthology.orchestra.run.smithi171:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-28T14:07:33.752 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-28T14:07:33.796 DEBUG:teuthology.orchestra.run.smithi171:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-28T14:07:33.842 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-28T14:07:33.884 DEBUG:teuthology.orchestra.run.smithi171:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-28T14:07:33.932 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-28T14:07:33.968 DEBUG:teuthology.orchestra.run.smithi171:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-28T14:07:34.017 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-03-28T14:07:34.017 DEBUG:teuthology.orchestra.run.smithi104:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2024-03-28T14:07:34.066 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-03-28T14:07:34.160 DEBUG:teuthology.orchestra.run.smithi104:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2024-03-28T14:07:34.170 DEBUG:teuthology.orchestra.run.smithi171:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2024-03-28T14:07:34.184 DEBUG:teuthology.orchestra.run.smithi171:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-03-28T14:07:34.277 DEBUG:teuthology.orchestra.run.smithi171:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2024-03-28T14:07:34.287 DEBUG:teuthology.orchestra.run.smithi171:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2024-03-28T14:07:34.341 DEBUG:teuthology.orchestra.run.smithi171:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-03-28T14:07:34.431 DEBUG:teuthology.orchestra.run.smithi171:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2024-03-28T14:07:34.442 DEBUG:teuthology.orchestra.run.smithi104:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-03-28T14:07:34.444 DEBUG:teuthology.orchestra.run.smithi171:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-03-28T14:07:34.488 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2024-03-28T14:07:34.488 INFO:tasks.ceph.mon.a:Restarting daemon 2024-03-28T14:07:34.488 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-03-28T14:07:34.492 INFO:tasks.ceph.mon.a:Started 2024-03-28T14:07:34.492 INFO:tasks.ceph.mon.b:Restarting daemon 2024-03-28T14:07:34.492 DEBUG:teuthology.orchestra.run.smithi171:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2024-03-28T14:07:34.537 INFO:tasks.ceph.mon.b:Started 2024-03-28T14:07:34.537 INFO:tasks.ceph.mon.c:Restarting daemon 2024-03-28T14:07:34.538 DEBUG:teuthology.orchestra.run.smithi171:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2024-03-28T14:07:34.540 INFO:tasks.ceph.mon.c:Started 2024-03-28T14:07:34.540 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2024-03-28T14:07:34.540 INFO:tasks.ceph.mgr.y:Restarting daemon 2024-03-28T14:07:34.540 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i y 2024-03-28T14:07:34.543 INFO:tasks.ceph.mgr.y:Started 2024-03-28T14:07:34.544 INFO:tasks.ceph.mgr.x:Restarting daemon 2024-03-28T14:07:34.544 DEBUG:teuthology.orchestra.run.smithi171:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2024-03-28T14:07:34.547 INFO:tasks.ceph.mgr.x:Started 2024-03-28T14:07:34.547 INFO:tasks.ceph:Setting crush tunables to default 2024-03-28T14:07:34.547 DEBUG:teuthology.orchestra.run.smithi104:> sudo ceph --cluster ceph osd crush tunables default 2024-03-28T14:07:34.660 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:34.660+0000 7f03dce36640 1 Processor -- start 2024-03-28T14:07:34.660 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:34.660+0000 7f03dce36640 1 -- start start 2024-03-28T14:07:34.662 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:34.660+0000 7f03dce36640 1 --2- >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f03d8151040 0x7f03d8153430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:07:34.662 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:34.660+0000 7f03dce36640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f03d8073210 con 0x7f03d806c940 2024-03-28T14:07:34.662 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:34.660+0000 7f03dce36640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f03d8073790 con 0x7f03d806cd20 2024-03-28T14:07:34.662 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:34.660+0000 7f03dce36640 1 -- --> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] -- mon_getmap magic: 0 -- 0x7f03d8147970 con 0x7f03d8151040 2024-03-28T14:07:34.662 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:34.660+0000 7f03d659c640 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f03d806c940 0x7f03d8147e00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:40220/0 (socket says 172.21.15.104:40220) 2024-03-28T14:07:34.662 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:34.660+0000 7f03d659c640 1 -- 172.21.15.104:0/3746185163 learned_addr learned my addr 172.21.15.104:0/3746185163 (peer_addr_for_me v1:172.21.15.104:0/0) 2024-03-28T14:07:34.662 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:34.660+0000 7f03d6d9d640 1 --2- 172.21.15.104:0/3746185163 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f03d8151040 0x7f03d8153430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:07:34.662 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:34.660+0000 7f03d6d9d640 1 -- 172.21.15.104:0/3746185163 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f03d8151040 msgr2=0x7f03d8153430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-03-28T14:07:34.662 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:34.660+0000 7f03d6d9d640 1 -- 172.21.15.104:0/3746185163 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f03d8151040 msgr2=0x7f03d8153430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-03-28T14:07:34.662 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:34.660+0000 7f03d6d9d640 1 --2- 172.21.15.104:0/3746185163 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f03d8151040 0x7f03d8153430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-28T14:07:34.662 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:34.660+0000 7f03d6d9d640 1 --2- 172.21.15.104:0/3746185163 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f03d8151040 0x7f03d8153430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-28T14:07:34.863 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:34.864+0000 7f03d6d9d640 1 --2- 172.21.15.104:0/3746185163 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f03d8151040 0x7f03d8153430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:07:34.863 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:34.864+0000 7f03d6d9d640 1 -- 172.21.15.104:0/3746185163 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f03d8151040 msgr2=0x7f03d8153430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-03-28T14:07:34.863 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:34.864+0000 7f03d6d9d640 1 -- 172.21.15.104:0/3746185163 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f03d8151040 msgr2=0x7f03d8153430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-03-28T14:07:34.863 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:34.864+0000 7f03d6d9d640 1 --2- 172.21.15.104:0/3746185163 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f03d8151040 0x7f03d8153430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-28T14:07:34.864 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:34.864+0000 7f03d6d9d640 1 --2- 172.21.15.104:0/3746185163 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f03d8151040 0x7f03d8153430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-28T14:07:34.889 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:34.888+0000 7f03d5d9b640 1 -- 172.21.15.104:0/3746185163 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2327266473 0 0) 0x7f03d8073210 con 0x7f03d806c940 2024-03-28T14:07:34.889 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:34.888+0000 7f03d5d9b640 1 -- 172.21.15.104:0/3746185163 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f03c4003530 con 0x7f03d806c940 2024-03-28T14:07:34.891 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:34.892+0000 7f03d5d9b640 1 -- 172.21.15.104:0/3746185163 <== mon.0 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1780445502 0 0) 0x7f03c4003530 con 0x7f03d806c940 2024-03-28T14:07:34.891 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:34.892+0000 7f03d5d9b640 1 -- 172.21.15.104:0/3746185163 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f03d8151040 msgr2=0x7f03d8153430 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-28T14:07:34.892 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:34.892+0000 7f03d5d9b640 1 --2- 172.21.15.104:0/3746185163 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f03d8151040 0x7f03d8153430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:07:34.892 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:34.892+0000 7f03d5d9b640 1 -- 172.21.15.104:0/3746185163 >> v1:172.21.15.171:6789/0 conn(0x7f03d806cd20 legacy=0x7f03d8141600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:07:34.892 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:34.892+0000 7f03d5d9b640 1 -- 172.21.15.104:0/3746185163 --> v1:172.21.15.104:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f03d80734b0 con 0x7f03d806c940 2024-03-28T14:07:34.892 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:34.892+0000 7f03d5d9b640 1 -- 172.21.15.104:0/3746185163 <== mon.0 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3745580849 0 0) 0x7f03c80016a0 con 0x7f03d806c940 2024-03-28T14:07:34.892 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:34.892+0000 7f03d5d9b640 1 -- 172.21.15.104:0/3746185163 >> v1:172.21.15.104:6789/0 conn(0x7f03d806c940 legacy=0x7f03d8147e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:07:34.892 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:34.892+0000 7f03d5d9b640 1 --2- 172.21.15.104:0/3746185163 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f03c40039f0 0x7f03c4003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:07:34.892 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:34.892+0000 7f03d5d9b640 1 --2- 172.21.15.104:0/3746185163 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f03c4004810 0x7f03c4004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:07:34.892 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:34.892+0000 7f03d5d9b640 1 --2- 172.21.15.104:0/3746185163 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f03d8151040 0x7f03c4005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:07:34.892 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:34.892+0000 7f03d5d9b640 1 -- 172.21.15.104:0/3746185163 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f03d8147970 con 0x7f03c4004810 2024-03-28T14:07:34.892 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:34.892+0000 7f03d5d9b640 1 -- 172.21.15.104:0/3746185163 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f03c400bb00 con 0x7f03c40039f0 2024-03-28T14:07:34.892 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:34.892+0000 7f03d5d9b640 1 -- 172.21.15.104:0/3746185163 --> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] -- mon_getmap magic: 0 -- 0x7f03c400bd00 con 0x7f03d8151040 2024-03-28T14:07:34.892 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:34.892+0000 7f03cdd9b640 1 --2- 172.21.15.104:0/3746185163 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f03c4004810 0x7f03c4004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:07:34.892 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:34.892+0000 7f03d6d9d640 1 --2- 172.21.15.104:0/3746185163 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f03d8151040 0x7f03c4005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:07:34.893 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:34.892+0000 7f03d659c640 1 --2- 172.21.15.104:0/3746185163 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f03c40039f0 0x7f03c4003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:07:34.893 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:34.892+0000 7f03cdd9b640 1 -- 172.21.15.104:0/3746185163 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f03d8151040 msgr2=0x7f03c4005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T14:07:34.893 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:34.892+0000 7f03cdd9b640 1 --2- 172.21.15.104:0/3746185163 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f03d8151040 0x7f03c4005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:07:34.893 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:34.892+0000 7f03cdd9b640 1 -- 172.21.15.104:0/3746185163 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f03c40039f0 msgr2=0x7f03c4003e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T14:07:34.893 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:34.892+0000 7f03cdd9b640 1 --2- 172.21.15.104:0/3746185163 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f03c40039f0 0x7f03c4003e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:07:34.893 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:34.892+0000 7f03cdd9b640 1 -- 172.21.15.104:0/3746185163 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f03c4003530 con 0x7f03c4004810 2024-03-28T14:07:34.896 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:34.896+0000 7f03cdd9b640 1 --2- 172.21.15.104:0/3746185163 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f03c4004810 0x7f03c4004c90 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f03c0001750 tx=0x7f03c001c3e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=102e182f43b39e7b server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:07:34.897 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:34.896+0000 7f03d5d9b640 1 -- 172.21.15.104:0/3746185163 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03c00013a0 con 0x7f03c4004810 2024-03-28T14:07:34.897 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:34.896+0000 7f03d5d9b640 1 -- 172.21.15.104:0/3746185163 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f03c00052f0 con 0x7f03c4004810 2024-03-28T14:07:34.897 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:34.896+0000 7f03d5d9b640 1 -- 172.21.15.104:0/3746185163 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03c0003840 con 0x7f03c4004810 2024-03-28T14:07:34.899 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:34.900+0000 7f03dce36640 1 -- 172.21.15.104:0/3746185163 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f03c4004810 msgr2=0x7f03c4004c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:07:34.899 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:34.900+0000 7f03dce36640 1 --2- 172.21.15.104:0/3746185163 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f03c4004810 0x7f03c4004c90 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f03c0001750 tx=0x7f03c001c3e0 comp rx=0 tx=0).stop 2024-03-28T14:07:34.899 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:34.900+0000 7f03d6d9d640 1 -- 172.21.15.104:0/3746185163 reap_dead start 2024-03-28T14:07:34.899 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:34.900+0000 7f03dce36640 1 -- 172.21.15.104:0/3746185163 shutdown_connections 2024-03-28T14:07:34.899 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:34.900+0000 7f03dce36640 1 -- 172.21.15.104:0/3746185163 >> 172.21.15.104:0/3746185163 conn(0x7f03d80b7a10 msgr2=0x7f03d80b7e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T14:07:34.899 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:34.900+0000 7f03dce36640 1 -- 172.21.15.104:0/3746185163 shutdown_connections 2024-03-28T14:07:34.899 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:34.900+0000 7f03dce36640 1 -- 172.21.15.104:0/3746185163 wait complete. 2024-03-28T14:07:34.900 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:34.900+0000 7f03dce36640 1 Processor -- start 2024-03-28T14:07:34.900 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:34.900+0000 7f03dce36640 1 -- start start 2024-03-28T14:07:34.902 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:34.900+0000 7f03dce36640 1 --2- >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f03c4004810 0x7f03d813f090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:07:34.902 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:34.900+0000 7f03dce36640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f03d80a0b80 0x7f03d81e0e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:07:34.902 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:34.900+0000 7f03dce36640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f03d8138c50 0x7f03d81390d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:07:34.902 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:34.900+0000 7f03dce36640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f03d806d210 con 0x7f03d80a0b80 2024-03-28T14:07:34.902 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:34.900+0000 7f03dce36640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f03d8072850 con 0x7f03d8138c50 2024-03-28T14:07:34.902 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:34.900+0000 7f03dce36640 1 -- --> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] -- mon_getmap magic: 0 -- 0x7f03d8141030 con 0x7f03c4004810 2024-03-28T14:07:34.903 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:34.900+0000 7f03cdd9b640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f03d80a0b80 0x7f03d81e0e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:07:34.903 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:34.900+0000 7f03cdd9b640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f03d80a0b80 0x7f03d81e0e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:48916/0 (socket says 172.21.15.104:48916) 2024-03-28T14:07:34.903 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:34.900+0000 7f03cdd9b640 1 -- 172.21.15.104:0/4048823209 learned_addr learned my addr 172.21.15.104:0/4048823209 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-03-28T14:07:34.903 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:34.900+0000 7f03cdd9b640 1 -- 172.21.15.104:0/4048823209 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f03c4004810 msgr2=0x7f03d813f090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:07:34.903 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:34.900+0000 7f03cdd9b640 1 --2- 172.21.15.104:0/4048823209 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f03c4004810 0x7f03d813f090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:07:34.903 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:34.900+0000 7f03cdd9b640 1 -- 172.21.15.104:0/4048823209 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f03d8138c50 msgr2=0x7f03d81390d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:07:34.903 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:34.900+0000 7f03cdd9b640 1 --2- 172.21.15.104:0/4048823209 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f03d8138c50 0x7f03d81390d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:07:34.903 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:34.900+0000 7f03cdd9b640 1 -- 172.21.15.104:0/4048823209 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f03d8140b10 con 0x7f03d80a0b80 2024-03-28T14:07:34.903 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:34.904+0000 7f03cdd9b640 1 --2- 172.21.15.104:0/4048823209 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f03d80a0b80 0x7f03d81e0e40 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f03c001c8c0 tx=0x7f03c4005840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:07:34.904 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:34.904+0000 7f03cf7fe640 1 -- 172.21.15.104:0/4048823209 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03c00ba020 con 0x7f03d80a0b80 2024-03-28T14:07:34.904 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:34.904+0000 7f03cf7fe640 1 -- 172.21.15.104:0/4048823209 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f03c0000ee0 con 0x7f03d80a0b80 2024-03-28T14:07:34.904 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:34.904+0000 7f03dce36640 1 -- 172.21.15.104:0/4048823209 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f03d809f7c0 con 0x7f03d80a0b80 2024-03-28T14:07:34.904 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:34.904+0000 7f03cf7fe640 1 -- 172.21.15.104:0/4048823209 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03c0006a00 con 0x7f03d80a0b80 2024-03-28T14:07:34.905 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:34.904+0000 7f03dce36640 1 -- 172.21.15.104:0/4048823209 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f03d8140680 con 0x7f03d80a0b80 2024-03-28T14:07:34.905 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:34.904+0000 7f03cf7fe640 1 -- 172.21.15.104:0/4048823209 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f03c00bd020 con 0x7f03d80a0b80 2024-03-28T14:07:34.905 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:34.904+0000 7f03cf7fe640 1 -- 172.21.15.104:0/4048823209 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f03c0003dc0 con 0x7f03d80a0b80 2024-03-28T14:07:34.905 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:34.904+0000 7f03dce36640 1 -- 172.21.15.104:0/4048823209 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f03d802d560 con 0x7f03d80a0b80 2024-03-28T14:07:34.906 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:34.904+0000 7f03cf7fe640 1 -- 172.21.15.104:0/4048823209 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7f03c00c4030 con 0x7f03d80a0b80 2024-03-28T14:07:34.969 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:34.968+0000 7f03dce36640 1 -- 172.21.15.104:0/4048823209 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) -- 0x7f03d80687c0 con 0x7f03d80a0b80 2024-03-28T14:07:34.972 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:34.972+0000 7f03cf7fe640 1 -- 172.21.15.104:0/4048823209 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v2) ==== 124+0+0 (secure 0 0 0) 0x7f03c00160f0 con 0x7f03d80a0b80 2024-03-28T14:07:34.972 INFO:teuthology.orchestra.run.smithi104.stderr:adjusted tunables profile to default 2024-03-28T14:07:34.974 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:34.972+0000 7f03dce36640 1 -- 172.21.15.104:0/4048823209 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f03d80a0b80 msgr2=0x7f03d81e0e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:07:34.974 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:34.972+0000 7f03dce36640 1 --2- 172.21.15.104:0/4048823209 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f03d80a0b80 0x7f03d81e0e40 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f03c001c8c0 tx=0x7f03c4005840 comp rx=0 tx=0).stop 2024-03-28T14:07:34.974 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:34.972+0000 7f03dce36640 1 -- 172.21.15.104:0/4048823209 shutdown_connections 2024-03-28T14:07:34.974 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:34.972+0000 7f03dce36640 1 --2- 172.21.15.104:0/4048823209 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f03d8138c50 0x7f03d81390d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:07:34.975 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:34.972+0000 7f03dce36640 1 --2- 172.21.15.104:0/4048823209 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f03d80a0b80 0x7f03d81e0e40 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:07:34.975 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:34.972+0000 7f03dce36640 1 --2- 172.21.15.104:0/4048823209 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f03c4004810 0x7f03d813f090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:07:34.975 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:34.972+0000 7f03dce36640 1 -- 172.21.15.104:0/4048823209 >> 172.21.15.104:0/4048823209 conn(0x7f03d80b7a10 msgr2=0x7f03d8067330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T14:07:34.975 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:34.972+0000 7f03dce36640 1 -- 172.21.15.104:0/4048823209 shutdown_connections 2024-03-28T14:07:34.975 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:34.972+0000 7f03dce36640 1 -- 172.21.15.104:0/4048823209 wait complete. 2024-03-28T14:07:34.991 INFO:tasks.ceph:check_enable_crimson: False 2024-03-28T14:07:34.991 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2024-03-28T14:07:34.991 DEBUG:teuthology.orchestra.run.smithi104:> set -ex 2024-03-28T14:07:34.991 DEBUG:teuthology.orchestra.run.smithi104:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2024-03-28T14:07:35.003 DEBUG:teuthology.orchestra.run.smithi104:> set -ex 2024-03-28T14:07:35.003 DEBUG:teuthology.orchestra.run.smithi104:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2024-03-28T14:07:35.057 DEBUG:teuthology.orchestra.run.smithi104:> set -ex 2024-03-28T14:07:35.057 DEBUG:teuthology.orchestra.run.smithi104:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2024-03-28T14:07:35.110 DEBUG:teuthology.orchestra.run.smithi104:> set -ex 2024-03-28T14:07:35.111 DEBUG:teuthology.orchestra.run.smithi104:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2024-03-28T14:07:35.121 INFO:tasks.ceph.mgr.x.smithi171.stderr:2024-03-28T14:07:35.119+0000 7fa461f1b180 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-28T14:07:35.167 DEBUG:teuthology.orchestra.run.smithi171:> set -ex 2024-03-28T14:07:35.167 DEBUG:teuthology.orchestra.run.smithi171:> sudo dd if=/var/lib/ceph/osd/ceph-4/fsid of=/dev/stdout 2024-03-28T14:07:35.180 DEBUG:teuthology.orchestra.run.smithi171:> set -ex 2024-03-28T14:07:35.180 DEBUG:teuthology.orchestra.run.smithi171:> sudo dd if=/var/lib/ceph/osd/ceph-5/fsid of=/dev/stdout 2024-03-28T14:07:35.235 DEBUG:teuthology.orchestra.run.smithi171:> set -ex 2024-03-28T14:07:35.235 DEBUG:teuthology.orchestra.run.smithi171:> sudo dd if=/var/lib/ceph/osd/ceph-6/fsid of=/dev/stdout 2024-03-28T14:07:35.288 DEBUG:teuthology.orchestra.run.smithi171:> set -ex 2024-03-28T14:07:35.288 DEBUG:teuthology.orchestra.run.smithi171:> sudo dd if=/var/lib/ceph/osd/ceph-7/fsid of=/dev/stdout 2024-03-28T14:07:35.343 DEBUG:teuthology.orchestra.run.smithi171:> sudo ceph --cluster ceph osd new 1cb70540-4e42-402f-a446-aaec4ca83007 0 2024-03-28T14:07:35.504 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:35.503+0000 7f44bfdab640 1 Processor -- start 2024-03-28T14:07:35.505 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:35.503+0000 7f44bfdab640 1 -- start start 2024-03-28T14:07:35.505 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:35.503+0000 7f44bfdab640 1 --2- >> v2:172.21.15.171:3300/0 conn(0x7f44b8141ca0 0x7f44b8153100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:07:35.506 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:35.503+0000 7f44bfdab640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f44b8147770 con 0x7f44b80b9350 2024-03-28T14:07:35.506 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:35.503+0000 7f44bfdab640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f44b80b8fa0 con 0x7f44b809b2d0 2024-03-28T14:07:35.506 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:35.503+0000 7f44bfdab640 1 -- --> v2:172.21.15.171:3300/0 -- mon_getmap magic: 0 -- 0x7f44b806b680 con 0x7f44b8141ca0 2024-03-28T14:07:35.506 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:35.503+0000 7f44be348640 1 --2- >> v2:172.21.15.171:3300/0 conn(0x7f44b8141ca0 0x7f44b8153100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:07:35.506 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:35.503+0000 7f44be348640 1 --2- >> v2:172.21.15.171:3300/0 conn(0x7f44b8141ca0 0x7f44b8153100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.171:3300/0 says I am v2:172.21.15.171:42970/0 (socket says 172.21.15.171:42970) 2024-03-28T14:07:35.506 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:35.503+0000 7f44bdb47640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7f44b809b2d0 0x7f44b8147f10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:47254/0 (socket says 172.21.15.171:47254) 2024-03-28T14:07:35.506 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:35.503+0000 7f44be348640 1 -- 172.21.15.171:0/635048835 learned_addr learned my addr 172.21.15.171:0/635048835 (peer_addr_for_me v2:172.21.15.171:0/0) 2024-03-28T14:07:35.506 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:35.503+0000 7f44be348640 1 -- 172.21.15.171:0/635048835 >> v1:172.21.15.171:6789/0 conn(0x7f44b809b2d0 legacy=0x7f44b8147f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:07:35.506 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:35.503+0000 7f44be348640 1 -- 172.21.15.171:0/635048835 >> v1:172.21.15.104:6789/0 conn(0x7f44b80b9350 legacy=0x7f44b8141590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T14:07:35.506 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:35.503+0000 7f44be348640 1 -- 172.21.15.171:0/635048835 --> v2:172.21.15.171:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f44b809b6b0 con 0x7f44b8141ca0 2024-03-28T14:07:35.507 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:35.503+0000 7f44be348640 1 --2- 172.21.15.171:0/635048835 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f44b8141ca0 0x7f44b8153100 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f44a80097f0 tx=0x7f44a801c150 comp rx=0 tx=0).ready entity=mon.1 client_cookie=78aac16dc258f7d1 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:07:35.507 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:35.507+0000 7f44bcb45640 1 -- 172.21.15.171:0/635048835 <== mon.1 v2:172.21.15.171:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44a80b0070 con 0x7f44b8141ca0 2024-03-28T14:07:35.508 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:35.507+0000 7f44bcb45640 1 -- 172.21.15.171:0/635048835 <== mon.1 v2:172.21.15.171:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f44a80ab600 con 0x7f44b8141ca0 2024-03-28T14:07:35.508 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:35.507+0000 7f44bcb45640 1 -- 172.21.15.171:0/635048835 <== mon.1 v2:172.21.15.171:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44a80ab9d0 con 0x7f44b8141ca0 2024-03-28T14:07:35.508 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:35.507+0000 7f44bfdab640 1 -- 172.21.15.171:0/635048835 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f44b8141ca0 msgr2=0x7f44b8153100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:07:35.508 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:35.507+0000 7f44bfdab640 1 --2- 172.21.15.171:0/635048835 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f44b8141ca0 0x7f44b8153100 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f44a80097f0 tx=0x7f44a801c150 comp rx=0 tx=0).stop 2024-03-28T14:07:35.509 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:35.507+0000 7f44bfdab640 1 -- 172.21.15.171:0/635048835 shutdown_connections 2024-03-28T14:07:35.509 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:35.507+0000 7f44bfdab640 1 --2- 172.21.15.171:0/635048835 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f44b8141ca0 0x7f44b8153100 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:07:35.509 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:35.507+0000 7f44bfdab640 1 -- 172.21.15.171:0/635048835 >> 172.21.15.171:0/635048835 conn(0x7f44b80b9b80 msgr2=0x7f44b80b9f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T14:07:35.509 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:35.507+0000 7f44bfdab640 1 -- 172.21.15.171:0/635048835 shutdown_connections 2024-03-28T14:07:35.509 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:35.507+0000 7f44bfdab640 1 -- 172.21.15.171:0/635048835 wait complete. 2024-03-28T14:07:35.509 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:35.507+0000 7f44bfdab640 1 Processor -- start 2024-03-28T14:07:35.509 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:35.507+0000 7f44bfdab640 1 -- start start 2024-03-28T14:07:35.511 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:35.507+0000 7f44bfdab640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f44b809b2d0 0x7f44b80b2f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:07:35.511 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:35.507+0000 7f44bfdab640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f44b80b9350 0x7f44b81420c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:07:35.512 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:35.507+0000 7f44bfdab640 1 --2- >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f44b8141ca0 0x7f44b80b2300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:07:35.512 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:35.507+0000 7f44bfdab640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f44b8155fe0 con 0x7f44b809b2d0 2024-03-28T14:07:35.512 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:35.507+0000 7f44bfdab640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f44b8155e60 con 0x7f44b80b9350 2024-03-28T14:07:35.512 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:35.507+0000 7f44bfdab640 1 -- --> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] -- mon_getmap magic: 0 -- 0x7f44b8156160 con 0x7f44b8141ca0 2024-03-28T14:07:35.512 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:35.507+0000 7f44bd346640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f44b80b9350 0x7f44b81420c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:07:35.512 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:35.507+0000 7f44be348640 1 --2- >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f44b8141ca0 0x7f44b80b2300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:07:35.512 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:35.507+0000 7f44bd346640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f44b80b9350 0x7f44b81420c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.171:3300/0 says I am v2:172.21.15.171:42976/0 (socket says 172.21.15.171:42976) 2024-03-28T14:07:35.512 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:35.507+0000 7f44be348640 1 --2- >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f44b8141ca0 0x7f44b80b2300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.171:3301/0 says I am v2:172.21.15.171:52446/0 (socket says 172.21.15.171:52446) 2024-03-28T14:07:35.512 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:35.507+0000 7f44bd346640 1 -- 172.21.15.171:0/3869183316 learned_addr learned my addr 172.21.15.171:0/3869183316 (peer_addr_for_me v2:172.21.15.171:0/0) 2024-03-28T14:07:35.512 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:35.507+0000 7f44bdb47640 1 --2- 172.21.15.171:0/3869183316 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f44b809b2d0 0x7f44b80b2f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:07:35.512 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:35.507+0000 7f44be348640 1 -- 172.21.15.171:0/3869183316 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f44b80b9350 msgr2=0x7f44b81420c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:07:35.512 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:35.507+0000 7f44be348640 1 --2- 172.21.15.171:0/3869183316 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f44b80b9350 0x7f44b81420c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:07:35.512 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:35.507+0000 7f44be348640 1 -- 172.21.15.171:0/3869183316 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f44b809b2d0 msgr2=0x7f44b80b2f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:07:35.513 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:35.507+0000 7f44be348640 1 --2- 172.21.15.171:0/3869183316 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f44b809b2d0 0x7f44b80b2f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:07:35.513 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:35.507+0000 7f44be348640 1 -- 172.21.15.171:0/3869183316 --> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f44b80b3570 con 0x7f44b8141ca0 2024-03-28T14:07:35.513 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:35.507+0000 7f44bd346640 1 --2- 172.21.15.171:0/3869183316 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f44b80b9350 0x7f44b81420c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T14:07:35.513 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:35.507+0000 7f44be348640 1 --2- 172.21.15.171:0/3869183316 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f44b8141ca0 0x7f44b80b2300 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f44a80ab000 tx=0x7f44a8018ef0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:07:35.513 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:35.507+0000 7f44b67fc640 1 -- 172.21.15.171:0/3869183316 <== mon.2 v2:172.21.15.171:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44a80b0050 con 0x7f44b8141ca0 2024-03-28T14:07:35.513 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:35.507+0000 7f44b67fc640 1 -- 172.21.15.171:0/3869183316 <== mon.2 v2:172.21.15.171:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f44a80bc070 con 0x7f44b8141ca0 2024-03-28T14:07:35.513 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:35.507+0000 7f44bfdab640 1 -- 172.21.15.171:0/3869183316 --> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f44b80b3850 con 0x7f44b8141ca0 2024-03-28T14:07:35.513 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:35.507+0000 7f44b67fc640 1 -- 172.21.15.171:0/3869183316 <== mon.2 v2:172.21.15.171:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44a80b5a50 con 0x7f44b8141ca0 2024-03-28T14:07:35.513 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:35.507+0000 7f44bfdab640 1 -- 172.21.15.171:0/3869183316 --> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f44b81e0980 con 0x7f44b8141ca0 2024-03-28T14:07:35.513 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:35.507+0000 7f44b67fc640 1 -- 172.21.15.171:0/3869183316 <== mon.2 v2:172.21.15.171:3301/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f44a80b5bf0 con 0x7f44b8141ca0 2024-03-28T14:07:35.513 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:35.507+0000 7f44b67fc640 1 -- 172.21.15.171:0/3869183316 <== mon.2 v2:172.21.15.171:3301/0 5 ==== osd_map(2..2 src has 1..2) ==== 725+0+0 (secure 0 0 0) 0x7f44a80c1bc0 con 0x7f44b8141ca0 2024-03-28T14:07:35.514 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:35.511+0000 7f44bfdab640 1 -- 172.21.15.171:0/3869183316 --> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f44b80687c0 con 0x7f44b8141ca0 2024-03-28T14:07:35.523 INFO:tasks.ceph.mgr.y.smithi104.stderr:2024-03-28T14:07:35.524+0000 7f57c9cb1180 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-28T14:07:35.524 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:35.523+0000 7f44b67fc640 1 -- 172.21.15.171:0/3869183316 <== mon.2 v2:172.21.15.171:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7f44b80b3570 con 0x7f44b8141ca0 2024-03-28T14:07:35.592 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:35.591+0000 7f44bfdab640 1 -- 172.21.15.171:0/3869183316 --> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] -- mon_command({"prefix": "osd new", "uuid": "1cb70540-4e42-402f-a446-aaec4ca83007", "id": 0} v 0) -- 0x7f44b80b8a30 con 0x7f44b8141ca0 2024-03-28T14:07:35.593 INFO:tasks.ceph.mgr.x.smithi171.stderr:2024-03-28T14:07:35.591+0000 7fa461f1b180 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-28T14:07:35.602 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:35.599+0000 7f44b67fc640 1 -- 172.21.15.171:0/3869183316 <== mon.2 v2:172.21.15.171:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "1cb70540-4e42-402f-a446-aaec4ca83007", "id": 0}]=0 v3) ==== 112+0+2 (secure 0 0 0) 0x7f44b8147770 con 0x7f44b8141ca0 2024-03-28T14:07:35.602 INFO:teuthology.orchestra.run.smithi171.stdout:0 2024-03-28T14:07:35.603 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:35.599+0000 7f44bfdab640 1 -- 172.21.15.171:0/3869183316 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f44b8141ca0 msgr2=0x7f44b80b2300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:07:35.604 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:35.599+0000 7f44bfdab640 1 --2- 172.21.15.171:0/3869183316 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f44b8141ca0 0x7f44b80b2300 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f44a80ab000 tx=0x7f44a8018ef0 comp rx=0 tx=0).stop 2024-03-28T14:07:35.604 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:35.599+0000 7f44bfdab640 1 -- 172.21.15.171:0/3869183316 shutdown_connections 2024-03-28T14:07:35.604 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:35.599+0000 7f44bfdab640 1 --2- 172.21.15.171:0/3869183316 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f44b8141ca0 0x7f44b80b2300 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:07:35.604 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:35.599+0000 7f44bfdab640 1 --2- 172.21.15.171:0/3869183316 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f44b80b9350 0x7f44b81420c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:07:35.604 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:35.599+0000 7f44bfdab640 1 --2- 172.21.15.171:0/3869183316 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f44b809b2d0 0x7f44b80b2f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:07:35.604 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:35.599+0000 7f44bfdab640 1 -- 172.21.15.171:0/3869183316 >> 172.21.15.171:0/3869183316 conn(0x7f44b80b9b80 msgr2=0x7f44b806b810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T14:07:35.604 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:35.599+0000 7f44bfdab640 1 -- 172.21.15.171:0/3869183316 shutdown_connections 2024-03-28T14:07:35.604 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:35.599+0000 7f44bfdab640 1 -- 172.21.15.171:0/3869183316 wait complete. 2024-03-28T14:07:35.620 DEBUG:teuthology.orchestra.run.smithi171:> sudo ceph --cluster ceph osd new 3ab7afdd-b980-418c-81f2-004e16de3dca 1 2024-03-28T14:07:35.738 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:35.735+0000 7f48798d6640 1 Processor -- start 2024-03-28T14:07:35.738 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:35.735+0000 7f48798d6640 1 -- start start 2024-03-28T14:07:35.739 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:35.735+0000 7f48798d6640 1 --2- >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f48740b5600 0x7f48741417b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:07:35.739 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:35.735+0000 7f48798d6640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f487406e0f0 con 0x7f48740b5120 2024-03-28T14:07:35.739 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:35.735+0000 7f48798d6640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f48740b0ee0 con 0x7f487406dd10 2024-03-28T14:07:35.739 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:35.735+0000 7f48798d6640 1 -- --> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] -- mon_getmap magic: 0 -- 0x7f48740b0b70 con 0x7f48740b5600 2024-03-28T14:07:35.739 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:35.735+0000 7f48727fc640 1 --2- >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f48740b5600 0x7f48741417b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:07:35.739 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:35.735+0000 7f48727fc640 1 --2- >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f48740b5600 0x7f48741417b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.171:3301/0 says I am v2:172.21.15.171:52454/0 (socket says 172.21.15.171:52454) 2024-03-28T14:07:35.739 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:35.735+0000 7f48727fc640 1 -- 172.21.15.171:0/2288026655 learned_addr learned my addr 172.21.15.171:0/2288026655 (peer_addr_for_me v2:172.21.15.171:0/0) 2024-03-28T14:07:35.740 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:35.735+0000 7f48727fc640 1 -- 172.21.15.171:0/2288026655 >> v1:172.21.15.171:6789/0 conn(0x7f487406dd10 legacy=0x7f4874072cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:07:35.740 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:35.735+0000 7f48727fc640 1 -- 172.21.15.171:0/2288026655 >> v1:172.21.15.104:6789/0 conn(0x7f48740b5120 legacy=0x7f4874152280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:07:35.740 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:35.735+0000 7f48727fc640 1 -- 172.21.15.171:0/2288026655 --> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f48740b6130 con 0x7f48740b5600 2024-03-28T14:07:35.741 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:35.739+0000 7f48727fc640 1 --2- 172.21.15.171:0/2288026655 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f48740b5600 0x7f48741417b0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f48640098a0 tx=0x7f486401c580 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8cf2569a406032a2 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:07:35.741 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:35.739+0000 7f4871ffb640 1 -- 172.21.15.171:0/2288026655 <== mon.2 v2:172.21.15.171:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f48640b0070 con 0x7f48740b5600 2024-03-28T14:07:35.741 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:35.739+0000 7f4871ffb640 1 -- 172.21.15.171:0/2288026655 <== mon.2 v2:172.21.15.171:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f48640ab600 con 0x7f48740b5600 2024-03-28T14:07:35.741 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:35.739+0000 7f4871ffb640 1 -- 172.21.15.171:0/2288026655 <== mon.2 v2:172.21.15.171:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f48640ab9d0 con 0x7f48740b5600 2024-03-28T14:07:35.742 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:35.739+0000 7f48798d6640 1 -- 172.21.15.171:0/2288026655 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f48740b5600 msgr2=0x7f48741417b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:07:35.742 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:35.739+0000 7f48798d6640 1 --2- 172.21.15.171:0/2288026655 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f48740b5600 0x7f48741417b0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f48640098a0 tx=0x7f486401c580 comp rx=0 tx=0).stop 2024-03-28T14:07:35.742 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:35.739+0000 7f48798d6640 1 -- 172.21.15.171:0/2288026655 shutdown_connections 2024-03-28T14:07:35.742 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:35.739+0000 7f48798d6640 1 --2- 172.21.15.171:0/2288026655 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f48740b5600 0x7f48741417b0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:07:35.743 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:35.739+0000 7f48798d6640 1 -- 172.21.15.171:0/2288026655 >> 172.21.15.171:0/2288026655 conn(0x7f48740ba390 msgr2=0x7f48740ba7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T14:07:35.743 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:35.739+0000 7f48798d6640 1 -- 172.21.15.171:0/2288026655 shutdown_connections 2024-03-28T14:07:35.743 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:35.739+0000 7f48798d6640 1 -- 172.21.15.171:0/2288026655 wait complete. 2024-03-28T14:07:35.745 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:35.739+0000 7f48798d6640 1 Processor -- start 2024-03-28T14:07:35.745 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:35.739+0000 7f48798d6640 1 -- start start 2024-03-28T14:07:35.745 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:35.743+0000 7f48798d6640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f487406dd10 0x7f487413e790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:07:35.745 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:35.743+0000 7f48798d6640 1 --2- >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f48740b5120 0x7f4874139270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:07:35.745 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:35.743+0000 7f48798d6640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f48740b5600 0x7f4874146330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:07:35.745 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:35.743+0000 7f48798d6640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f4874156550 con 0x7f487406dd10 2024-03-28T14:07:35.745 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:35.743+0000 7f48798d6640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f48741563d0 con 0x7f48740b5600 2024-03-28T14:07:35.745 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:35.743+0000 7f48798d6640 1 -- --> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] -- mon_getmap magic: 0 -- 0x7f48741566d0 con 0x7f48740b5120 2024-03-28T14:07:35.746 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:35.743+0000 7f48727fc640 1 --2- >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f48740b5120 0x7f4874139270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:07:35.746 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:35.743+0000 7f48727fc640 1 --2- >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f48740b5120 0x7f4874139270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.171:3301/0 says I am v2:172.21.15.171:52464/0 (socket says 172.21.15.171:52464) 2024-03-28T14:07:35.746 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:35.743+0000 7f48727fc640 1 -- 172.21.15.171:0/614278929 learned_addr learned my addr 172.21.15.171:0/614278929 (peer_addr_for_me v2:172.21.15.171:0/0) 2024-03-28T14:07:35.746 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:35.743+0000 7f48737fe640 1 --2- 172.21.15.171:0/614278929 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f48740b5600 0x7f4874146330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:07:35.746 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:35.743+0000 7f4872ffd640 1 --2- 172.21.15.171:0/614278929 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f487406dd10 0x7f487413e790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:07:35.746 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:35.743+0000 7f48727fc640 1 -- 172.21.15.171:0/614278929 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f48740b5600 msgr2=0x7f4874146330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:07:35.746 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:35.743+0000 7f48727fc640 1 --2- 172.21.15.171:0/614278929 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f48740b5600 0x7f4874146330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:07:35.746 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:35.743+0000 7f48727fc640 1 -- 172.21.15.171:0/614278929 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f487406dd10 msgr2=0x7f487413e790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:07:35.746 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:35.743+0000 7f48727fc640 1 --2- 172.21.15.171:0/614278929 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f487406dd10 0x7f487413e790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:07:35.746 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:35.743+0000 7f48727fc640 1 -- 172.21.15.171:0/614278929 --> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f487413f9e0 con 0x7f48740b5120 2024-03-28T14:07:35.746 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:35.743+0000 7f48727fc640 1 --2- 172.21.15.171:0/614278929 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f48740b5120 0x7f4874139270 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f4864019190 tx=0x7f4864019320 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:07:35.746 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:35.743+0000 7f486bfff640 1 -- 172.21.15.171:0/614278929 <== mon.2 v2:172.21.15.171:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f48640b0040 con 0x7f48740b5120 2024-03-28T14:07:35.746 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:35.743+0000 7f486bfff640 1 -- 172.21.15.171:0/614278929 <== mon.2 v2:172.21.15.171:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f48640b9070 con 0x7f48740b5120 2024-03-28T14:07:35.747 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:35.743+0000 7f48798d6640 1 -- 172.21.15.171:0/614278929 --> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4874145850 con 0x7f48740b5120 2024-03-28T14:07:35.747 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:35.743+0000 7f486bfff640 1 -- 172.21.15.171:0/614278929 <== mon.2 v2:172.21.15.171:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f48640aab90 con 0x7f48740b5120 2024-03-28T14:07:35.747 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:35.743+0000 7f48798d6640 1 -- 172.21.15.171:0/614278929 --> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f48741469f0 con 0x7f48740b5120 2024-03-28T14:07:35.747 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:35.743+0000 7f486bfff640 1 -- 172.21.15.171:0/614278929 <== mon.2 v2:172.21.15.171:3301/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f48640be460 con 0x7f48740b5120 2024-03-28T14:07:35.747 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:35.743+0000 7f486bfff640 1 -- 172.21.15.171:0/614278929 <== mon.2 v2:172.21.15.171:3301/0 5 ==== osd_map(3..3 src has 1..3) ==== 840+0+0 (secure 0 0 0) 0x7f48640b47f0 con 0x7f48740b5120 2024-03-28T14:07:35.747 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:35.743+0000 7f48798d6640 1 -- 172.21.15.171:0/614278929 --> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f48740687c0 con 0x7f48740b5120 2024-03-28T14:07:35.747 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:35.743+0000 7f486bfff640 1 -- 172.21.15.171:0/614278929 <== mon.2 v2:172.21.15.171:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7f4864016170 con 0x7f48740b5120 2024-03-28T14:07:35.807 INFO:tasks.ceph.mgr.x.smithi171.stderr:2024-03-28T14:07:35.803+0000 7fa461f1b180 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-28T14:07:35.815 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:35.811+0000 7f48798d6640 1 -- 172.21.15.171:0/614278929 --> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] -- mon_command({"prefix": "osd new", "uuid": "3ab7afdd-b980-418c-81f2-004e16de3dca", "id": 1} v 0) -- 0x7f487402d400 con 0x7f48740b5120 2024-03-28T14:07:35.829 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:35.827+0000 7f486bfff640 1 -- 172.21.15.171:0/614278929 <== mon.2 v2:172.21.15.171:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "3ab7afdd-b980-418c-81f2-004e16de3dca", "id": 1}]=0 v4) ==== 112+0+2 (secure 0 0 0) 0x7f48640aad30 con 0x7f48740b5120 2024-03-28T14:07:35.829 INFO:teuthology.orchestra.run.smithi171.stdout:1 2024-03-28T14:07:35.830 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:35.827+0000 7f48798d6640 1 -- 172.21.15.171:0/614278929 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f48740b5120 msgr2=0x7f4874139270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:07:35.830 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:35.827+0000 7f48798d6640 1 --2- 172.21.15.171:0/614278929 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f48740b5120 0x7f4874139270 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f4864019190 tx=0x7f4864019320 comp rx=0 tx=0).stop 2024-03-28T14:07:35.830 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:35.827+0000 7f48798d6640 1 -- 172.21.15.171:0/614278929 shutdown_connections 2024-03-28T14:07:35.830 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:35.827+0000 7f48798d6640 1 --2- 172.21.15.171:0/614278929 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f48740b5600 0x7f4874146330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:07:35.831 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:35.827+0000 7f48798d6640 1 --2- 172.21.15.171:0/614278929 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f48740b5120 0x7f4874139270 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:07:35.831 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:35.827+0000 7f48798d6640 1 --2- 172.21.15.171:0/614278929 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f487406dd10 0x7f487413e790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:07:35.831 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:35.827+0000 7f48798d6640 1 -- 172.21.15.171:0/614278929 >> 172.21.15.171:0/614278929 conn(0x7f48740ba390 msgr2=0x7f487406bd20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T14:07:35.831 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:35.827+0000 7f48798d6640 1 -- 172.21.15.171:0/614278929 shutdown_connections 2024-03-28T14:07:35.831 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:35.827+0000 7f48798d6640 1 -- 172.21.15.171:0/614278929 wait complete. 2024-03-28T14:07:35.845 DEBUG:teuthology.orchestra.run.smithi171:> sudo ceph --cluster ceph osd new 3e2651c8-ecae-4a9e-a053-27240a79eac2 2 2024-03-28T14:07:35.963 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:35.959+0000 7fe80f6dc640 1 Processor -- start 2024-03-28T14:07:35.963 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:35.959+0000 7fe80f6dc640 1 -- start start 2024-03-28T14:07:35.966 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:35.959+0000 7fe80f6dc640 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fe80813fc30 0x7fe808148d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:07:35.966 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:35.959+0000 7fe80f6dc640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe80802c840 con 0x7fe80802c460 2024-03-28T14:07:35.966 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:35.959+0000 7fe80f6dc640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe8080b7460 con 0x7fe80806d510 2024-03-28T14:07:35.966 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:35.959+0000 7fe80f6dc640 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 -- 0x7fe8080b7ab0 con 0x7fe80813fc30 2024-03-28T14:07:35.966 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:35.959+0000 7fe80dc79640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7fe80806d510 0x7fe80814ffd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:47272/0 (socket says 172.21.15.171:47272) 2024-03-28T14:07:35.966 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:35.959+0000 7fe80dc79640 1 -- 172.21.15.171:0/3495570450 learned_addr learned my addr 172.21.15.171:0/3495570450 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-03-28T14:07:35.966 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:35.963+0000 7fe80cc77640 1 --2- 172.21.15.171:0/3495570450 >> v2:172.21.15.104:3300/0 conn(0x7fe80813fc30 0x7fe808148d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:07:35.966 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:35.963+0000 7fe807fff640 1 -- 172.21.15.171:0/3495570450 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 278485640 0 0) 0x7fe8080b7460 con 0x7fe80806d510 2024-03-28T14:07:35.966 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:35.963+0000 7fe807fff640 1 -- 172.21.15.171:0/3495570450 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe7f0003530 con 0x7fe80806d510 2024-03-28T14:07:35.966 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:35.963+0000 7fe807fff640 1 -- 172.21.15.171:0/3495570450 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3755983504 0 0) 0x7fe7f0003530 con 0x7fe80806d510 2024-03-28T14:07:35.966 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:35.963+0000 7fe807fff640 1 -- 172.21.15.171:0/3495570450 >> v2:172.21.15.104:3300/0 conn(0x7fe80813fc30 msgr2=0x7fe808148d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T14:07:35.966 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:35.963+0000 7fe807fff640 1 --2- 172.21.15.171:0/3495570450 >> v2:172.21.15.104:3300/0 conn(0x7fe80813fc30 0x7fe808148d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:07:35.967 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:35.963+0000 7fe807fff640 1 -- 172.21.15.171:0/3495570450 >> v1:172.21.15.104:6789/0 conn(0x7fe80802c460 legacy=0x7fe80813f520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:07:35.967 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:35.963+0000 7fe807fff640 1 -- 172.21.15.171:0/3495570450 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe8080b7740 con 0x7fe80806d510 2024-03-28T14:07:35.967 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:35.963+0000 7fe807fff640 1 -- 172.21.15.171:0/3495570450 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3745580849 0 0) 0x7fe7f8001680 con 0x7fe80806d510 2024-03-28T14:07:35.967 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:35.963+0000 7fe807fff640 1 -- 172.21.15.171:0/3495570450 >> v1:172.21.15.171:6789/0 conn(0x7fe80806d510 legacy=0x7fe80814ffd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:07:35.967 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:35.963+0000 7fe807fff640 1 --2- 172.21.15.171:0/3495570450 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fe7f00039c0 0x7fe7f0003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:07:35.967 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:35.963+0000 7fe807fff640 1 --2- 172.21.15.171:0/3495570450 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fe7f0004810 0x7fe7f0004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:07:35.967 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:35.963+0000 7fe807fff640 1 --2- 172.21.15.171:0/3495570450 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe7f00052f0 0x7fe7f000b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:07:35.967 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:35.963+0000 7fe807fff640 1 -- 172.21.15.171:0/3495570450 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7fe8080b7ab0 con 0x7fe7f00052f0 2024-03-28T14:07:35.967 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:35.963+0000 7fe807fff640 1 -- 172.21.15.171:0/3495570450 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7fe7f000c1c0 con 0x7fe7f00039c0 2024-03-28T14:07:35.967 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:35.963+0000 7fe807fff640 1 -- 172.21.15.171:0/3495570450 --> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] -- mon_getmap magic: 0 -- 0x7fe7f000c3c0 con 0x7fe7f0004810 2024-03-28T14:07:35.968 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:35.963+0000 7fe80d478640 1 --2- 172.21.15.171:0/3495570450 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fe7f00039c0 0x7fe7f0003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:07:35.968 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:35.963+0000 7fe80cc77640 1 --2- 172.21.15.171:0/3495570450 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fe7f0004810 0x7fe7f0004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:07:35.968 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:35.963+0000 7fe80dc79640 1 --2- 172.21.15.171:0/3495570450 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe7f00052f0 0x7fe7f000b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:07:35.968 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:35.963+0000 7fe80d478640 1 -- 172.21.15.171:0/3495570450 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fe7f0004810 msgr2=0x7fe7f0004c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T14:07:35.968 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:35.963+0000 7fe80d478640 1 --2- 172.21.15.171:0/3495570450 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fe7f0004810 0x7fe7f0004c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:07:35.968 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:35.963+0000 7fe80d478640 1 -- 172.21.15.171:0/3495570450 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe7f00052f0 msgr2=0x7fe7f000b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T14:07:35.968 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:35.963+0000 7fe80d478640 1 --2- 172.21.15.171:0/3495570450 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe7f00052f0 0x7fe7f000b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:07:35.968 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:35.963+0000 7fe80dc79640 1 -- 172.21.15.171:0/3495570450 reap_dead start 2024-03-28T14:07:35.968 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:35.963+0000 7fe80d478640 1 -- 172.21.15.171:0/3495570450 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe7f0003530 con 0x7fe7f00039c0 2024-03-28T14:07:35.972 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:35.963+0000 7fe80d478640 1 --2- 172.21.15.171:0/3495570450 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fe7f00039c0 0x7fe7f0003de0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fe800001e40 tx=0x7fe80001c740 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9658456d210d3e44 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:07:35.972 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:35.963+0000 7fe807fff640 1 -- 172.21.15.171:0/3495570450 <== mon.1 v2:172.21.15.171:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe8000b0070 con 0x7fe7f00039c0 2024-03-28T14:07:35.972 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:35.963+0000 7fe807fff640 1 -- 172.21.15.171:0/3495570450 <== mon.1 v2:172.21.15.171:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe800003b80 con 0x7fe7f00039c0 2024-03-28T14:07:35.972 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:35.963+0000 7fe807fff640 1 -- 172.21.15.171:0/3495570450 <== mon.1 v2:172.21.15.171:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe800001a90 con 0x7fe7f00039c0 2024-03-28T14:07:35.972 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:35.963+0000 7fe80f6dc640 1 -- 172.21.15.171:0/3495570450 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fe7f00039c0 msgr2=0x7fe7f0003de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:07:35.972 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:35.963+0000 7fe80f6dc640 1 --2- 172.21.15.171:0/3495570450 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fe7f00039c0 0x7fe7f0003de0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fe800001e40 tx=0x7fe80001c740 comp rx=0 tx=0).stop 2024-03-28T14:07:35.972 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:35.963+0000 7fe80f6dc640 1 -- 172.21.15.171:0/3495570450 shutdown_connections 2024-03-28T14:07:35.973 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:35.963+0000 7fe80f6dc640 1 --2- 172.21.15.171:0/3495570450 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fe7f00039c0 0x7fe7f0003de0 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:07:35.973 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:35.963+0000 7fe80f6dc640 1 -- 172.21.15.171:0/3495570450 >> 172.21.15.171:0/3495570450 conn(0x7fe8080b9660 msgr2=0x7fe8080b9a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T14:07:35.973 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:35.963+0000 7fe80f6dc640 1 -- 172.21.15.171:0/3495570450 shutdown_connections 2024-03-28T14:07:35.973 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:35.967+0000 7fe80f6dc640 1 -- 172.21.15.171:0/3495570450 wait complete. 2024-03-28T14:07:35.973 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:35.967+0000 7fe80f6dc640 1 Processor -- start 2024-03-28T14:07:35.973 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:35.967+0000 7fe80f6dc640 1 -- start start 2024-03-28T14:07:35.973 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:35.967+0000 7fe80f6dc640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fe7f00039c0 0x7fe8080729a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:07:35.973 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:35.967+0000 7fe80f6dc640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe808073260 0x7fe808147d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:07:35.973 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:35.967+0000 7fe80f6dc640 1 --2- >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fe8081431d0 0x7fe8081e0da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:07:35.973 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:35.967+0000 7fe80f6dc640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7fe8080b60c0 con 0x7fe808073260 2024-03-28T14:07:35.973 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:35.967+0000 7fe80f6dc640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7fe808149730 con 0x7fe7f00039c0 2024-03-28T14:07:35.973 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:35.967+0000 7fe80f6dc640 1 -- --> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] -- mon_getmap magic: 0 -- 0x7fe8080b6240 con 0x7fe8081431d0 2024-03-28T14:07:35.974 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:35.967+0000 7fe80dc79640 1 --2- >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fe8081431d0 0x7fe8081e0da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:07:35.974 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:35.967+0000 7fe80d478640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fe7f00039c0 0x7fe8080729a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:07:35.974 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:35.967+0000 7fe80d478640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fe7f00039c0 0x7fe8080729a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.171:3300/0 says I am v2:172.21.15.171:42998/0 (socket says 172.21.15.171:42998) 2024-03-28T14:07:35.974 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:35.967+0000 7fe80d478640 1 -- 172.21.15.171:0/2795674533 learned_addr learned my addr 172.21.15.171:0/2795674533 (peer_addr_for_me v2:172.21.15.171:0/0) 2024-03-28T14:07:35.974 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:35.967+0000 7fe80dc79640 1 --2- 172.21.15.171:0/2795674533 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fe8081431d0 0x7fe8081e0da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.171:3301/0 says I am v2:172.21.15.171:52474/0 (socket says 172.21.15.171:52474) 2024-03-28T14:07:35.974 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:35.967+0000 7fe80cc77640 1 --2- 172.21.15.171:0/2795674533 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe808073260 0x7fe808147d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:07:35.974 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:35.967+0000 7fe80d478640 1 -- 172.21.15.171:0/2795674533 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fe8081431d0 msgr2=0x7fe8081e0da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:07:35.974 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:35.967+0000 7fe80d478640 1 --2- 172.21.15.171:0/2795674533 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fe8081431d0 0x7fe8081e0da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:07:35.974 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:35.967+0000 7fe80d478640 1 -- 172.21.15.171:0/2795674533 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe808073260 msgr2=0x7fe808147d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:07:35.974 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:35.967+0000 7fe80d478640 1 --2- 172.21.15.171:0/2795674533 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe808073260 0x7fe808147d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:07:35.974 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:35.967+0000 7fe80d478640 1 -- 172.21.15.171:0/2795674533 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe8080721a0 con 0x7fe7f00039c0 2024-03-28T14:07:35.974 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:35.967+0000 7fe80cc77640 1 --2- 172.21.15.171:0/2795674533 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe808073260 0x7fe808147d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T14:07:35.975 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:35.967+0000 7fe80d478640 1 --2- 172.21.15.171:0/2795674533 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fe7f00039c0 0x7fe8080729a0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fe80001c570 tx=0x7fe7f0005840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:07:35.975 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:35.967+0000 7fe805ffb640 1 -- 172.21.15.171:0/2795674533 <== mon.1 v2:172.21.15.171:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe8000b9070 con 0x7fe7f00039c0 2024-03-28T14:07:35.975 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:35.967+0000 7fe805ffb640 1 -- 172.21.15.171:0/2795674533 <== mon.1 v2:172.21.15.171:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe8000050d0 con 0x7fe7f00039c0 2024-03-28T14:07:35.975 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:35.967+0000 7fe80f6dc640 1 -- 172.21.15.171:0/2795674533 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe808148440 con 0x7fe7f00039c0 2024-03-28T14:07:35.975 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:35.967+0000 7fe805ffb640 1 -- 172.21.15.171:0/2795674533 <== mon.1 v2:172.21.15.171:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe8000b0040 con 0x7fe7f00039c0 2024-03-28T14:07:35.975 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:35.967+0000 7fe80f6dc640 1 -- 172.21.15.171:0/2795674533 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe8080b31b0 con 0x7fe7f00039c0 2024-03-28T14:07:35.975 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:35.967+0000 7fe805ffb640 1 -- 172.21.15.171:0/2795674533 <== mon.1 v2:172.21.15.171:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fe800003a40 con 0x7fe7f00039c0 2024-03-28T14:07:35.975 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:35.967+0000 7fe805ffb640 1 -- 172.21.15.171:0/2795674533 <== mon.1 v2:172.21.15.171:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 955+0+0 (secure 0 0 0) 0x7fe8000079a0 con 0x7fe7f00039c0 2024-03-28T14:07:35.975 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:35.967+0000 7fe80f6dc640 1 -- 172.21.15.171:0/2795674533 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe808073650 con 0x7fe7f00039c0 2024-03-28T14:07:35.975 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:35.971+0000 7fe805ffb640 1 -- 172.21.15.171:0/2795674533 <== mon.1 v2:172.21.15.171:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7fe808073650 con 0x7fe7f00039c0 2024-03-28T14:07:36.004 INFO:tasks.ceph.mgr.y.smithi104.stderr:2024-03-28T14:07:36.004+0000 7f57c9cb1180 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-28T14:07:36.041 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:36.039+0000 7fe80f6dc640 1 -- 172.21.15.171:0/2795674533 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_command({"prefix": "osd new", "uuid": "3e2651c8-ecae-4a9e-a053-27240a79eac2", "id": 2} v 0) -- 0x7fe8080687c0 con 0x7fe7f00039c0 2024-03-28T14:07:36.047 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:36.047+0000 7fe805ffb640 1 -- 172.21.15.171:0/2795674533 <== mon.1 v2:172.21.15.171:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "3e2651c8-ecae-4a9e-a053-27240a79eac2", "id": 2}]=0 v5) ==== 112+0+2 (secure 0 0 0) 0x7fe800016450 con 0x7fe7f00039c0 2024-03-28T14:07:36.048 INFO:teuthology.orchestra.run.smithi171.stdout:2 2024-03-28T14:07:36.049 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:36.047+0000 7fe80f6dc640 1 -- 172.21.15.171:0/2795674533 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fe7f00039c0 msgr2=0x7fe8080729a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:07:36.049 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:36.047+0000 7fe80f6dc640 1 --2- 172.21.15.171:0/2795674533 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fe7f00039c0 0x7fe8080729a0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fe80001c570 tx=0x7fe7f0005840 comp rx=0 tx=0).stop 2024-03-28T14:07:36.049 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:36.047+0000 7fe80f6dc640 1 -- 172.21.15.171:0/2795674533 shutdown_connections 2024-03-28T14:07:36.049 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:36.047+0000 7fe80f6dc640 1 --2- 172.21.15.171:0/2795674533 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fe8081431d0 0x7fe8081e0da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:07:36.049 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:36.047+0000 7fe80f6dc640 1 --2- 172.21.15.171:0/2795674533 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe808073260 0x7fe808147d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:07:36.049 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:36.047+0000 7fe80f6dc640 1 --2- 172.21.15.171:0/2795674533 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fe7f00039c0 0x7fe8080729a0 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:07:36.049 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:36.047+0000 7fe80f6dc640 1 -- 172.21.15.171:0/2795674533 >> 172.21.15.171:0/2795674533 conn(0x7fe8080b9660 msgr2=0x7fe80809d690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T14:07:36.050 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:36.047+0000 7fe80f6dc640 1 -- 172.21.15.171:0/2795674533 shutdown_connections 2024-03-28T14:07:36.050 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:36.047+0000 7fe80f6dc640 1 -- 172.21.15.171:0/2795674533 wait complete. 2024-03-28T14:07:36.065 DEBUG:teuthology.orchestra.run.smithi171:> sudo ceph --cluster ceph osd new 925c7e0a-38ef-47cd-b288-8bec8456fb17 3 2024-03-28T14:07:36.175 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:36.171+0000 7fa0a5ec4640 1 Processor -- start 2024-03-28T14:07:36.175 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:36.171+0000 7fa0a5ec4640 1 -- start start 2024-03-28T14:07:36.176 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:36.175+0000 7fa0a5ec4640 1 --2- >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fa0a0072bf0 0x7fa0a00b1e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:07:36.176 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:36.175+0000 7fa0a5ec4640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa0a0072020 con 0x7fa0a0071be0 2024-03-28T14:07:36.176 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:36.175+0000 7fa0a5ec4640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa0a0072fd0 con 0x7fa0a0145ce0 2024-03-28T14:07:36.176 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:36.175+0000 7fa0a5ec4640 1 -- --> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] -- mon_getmap magic: 0 -- 0x7fa0a0073250 con 0x7fa0a0072bf0 2024-03-28T14:07:36.176 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:36.175+0000 7fa09f7fe640 1 --2- >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fa0a0072bf0 0x7fa0a00b1e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:07:36.176 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:36.175+0000 7fa09ffff640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7fa0a0145ce0 0x7fa0a014fe90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:47276/0 (socket says 172.21.15.171:47276) 2024-03-28T14:07:36.176 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:36.175+0000 7fa09ffff640 1 -- 172.21.15.171:0/997439242 learned_addr learned my addr 172.21.15.171:0/997439242 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-03-28T14:07:36.177 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:36.175+0000 7fa09e7fc640 1 -- 172.21.15.171:0/997439242 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3193469209 0 0) 0x7fa0a0072fd0 con 0x7fa0a0145ce0 2024-03-28T14:07:36.177 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:36.175+0000 7fa09f7fe640 1 -- 172.21.15.171:0/997439242 >> v1:172.21.15.171:6789/0 conn(0x7fa0a0145ce0 legacy=0x7fa0a014fe90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:07:36.177 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:36.175+0000 7fa09f7fe640 1 -- 172.21.15.171:0/997439242 >> v1:172.21.15.104:6789/0 conn(0x7fa0a0071be0 legacy=0x7fa0a00b12f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T14:07:36.177 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:36.175+0000 7fa09f7fe640 1 -- 172.21.15.171:0/997439242 --> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa0a00734a0 con 0x7fa0a0072bf0 2024-03-28T14:07:36.179 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:36.175+0000 7fa09f7fe640 1 --2- 172.21.15.171:0/997439242 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fa0a0072bf0 0x7fa0a00b1e80 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fa0a00733d0 tx=0x7fa09001c360 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ac1eddf38c8280a1 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:07:36.179 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:36.175+0000 7fa09e7fc640 1 -- 172.21.15.171:0/997439242 <== mon.2 v2:172.21.15.171:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0900b0070 con 0x7fa0a0072bf0 2024-03-28T14:07:36.179 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:36.175+0000 7fa09e7fc640 1 -- 172.21.15.171:0/997439242 <== mon.2 v2:172.21.15.171:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa0900ab600 con 0x7fa0a0072bf0 2024-03-28T14:07:36.179 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:36.175+0000 7fa09e7fc640 1 -- 172.21.15.171:0/997439242 <== mon.2 v2:172.21.15.171:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0900ab9d0 con 0x7fa0a0072bf0 2024-03-28T14:07:36.180 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:36.175+0000 7fa0a5ec4640 1 -- 172.21.15.171:0/997439242 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fa0a0072bf0 msgr2=0x7fa0a00b1e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:07:36.180 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:36.175+0000 7fa0a5ec4640 1 --2- 172.21.15.171:0/997439242 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fa0a0072bf0 0x7fa0a00b1e80 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fa0a00733d0 tx=0x7fa09001c360 comp rx=0 tx=0).stop 2024-03-28T14:07:36.180 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:36.175+0000 7fa0a5ec4640 1 -- 172.21.15.171:0/997439242 shutdown_connections 2024-03-28T14:07:36.180 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:36.175+0000 7fa0a5ec4640 1 --2- 172.21.15.171:0/997439242 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fa0a0072bf0 0x7fa0a00b1e80 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:07:36.180 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:36.175+0000 7fa0a5ec4640 1 -- 172.21.15.171:0/997439242 >> 172.21.15.171:0/997439242 conn(0x7fa0a00b7b10 msgr2=0x7fa0a00b7f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T14:07:36.180 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:36.175+0000 7fa0a5ec4640 1 -- 172.21.15.171:0/997439242 shutdown_connections 2024-03-28T14:07:36.180 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:36.179+0000 7fa0a5ec4640 1 -- 172.21.15.171:0/997439242 wait complete. 2024-03-28T14:07:36.182 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:36.179+0000 7fa0a5ec4640 1 Processor -- start 2024-03-28T14:07:36.183 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:36.179+0000 7fa0a5ec4640 1 -- start start 2024-03-28T14:07:36.183 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:36.179+0000 7fa0a5ec4640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fa0a0071be0 0x7fa0a0138a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:07:36.183 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:36.179+0000 7fa0a5ec4640 1 --2- >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fa0a0072bf0 0x7fa0a00b2440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:07:36.183 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:36.179+0000 7fa0a5ec4640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa0a0145ce0 0x7fa0a00b3ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:07:36.183 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:36.179+0000 7fa0a5ec4640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7fa0a0153dd0 con 0x7fa0a0145ce0 2024-03-28T14:07:36.183 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:36.179+0000 7fa0a5ec4640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7fa0a01455f0 con 0x7fa0a0071be0 2024-03-28T14:07:36.183 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:36.179+0000 7fa0a5ec4640 1 -- --> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] -- mon_getmap magic: 0 -- 0x7fa0a0153f50 con 0x7fa0a0072bf0 2024-03-28T14:07:36.183 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:36.179+0000 7fa09f7fe640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fa0a0071be0 0x7fa0a0138a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:07:36.183 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:36.179+0000 7fa09effd640 1 --2- >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fa0a0072bf0 0x7fa0a00b2440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:07:36.183 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:36.179+0000 7fa09f7fe640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fa0a0071be0 0x7fa0a0138a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.171:3300/0 says I am v2:172.21.15.171:43002/0 (socket says 172.21.15.171:43002) 2024-03-28T14:07:36.183 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:36.179+0000 7fa09f7fe640 1 -- 172.21.15.171:0/141249442 learned_addr learned my addr 172.21.15.171:0/141249442 (peer_addr_for_me v2:172.21.15.171:0/0) 2024-03-28T14:07:36.183 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:36.179+0000 7fa09ffff640 1 --2- 172.21.15.171:0/141249442 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa0a0145ce0 0x7fa0a00b3ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:07:36.183 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:36.179+0000 7fa09f7fe640 1 -- 172.21.15.171:0/141249442 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fa0a0072bf0 msgr2=0x7fa0a00b2440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:07:36.183 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:36.179+0000 7fa09f7fe640 1 --2- 172.21.15.171:0/141249442 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fa0a0072bf0 0x7fa0a00b2440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:07:36.184 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:36.179+0000 7fa09f7fe640 1 -- 172.21.15.171:0/141249442 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa0a0145ce0 msgr2=0x7fa0a00b3ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:07:36.184 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:36.179+0000 7fa09f7fe640 1 --2- 172.21.15.171:0/141249442 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa0a0145ce0 0x7fa0a00b3ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:07:36.184 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:36.179+0000 7fa09f7fe640 1 -- 172.21.15.171:0/141249442 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa0a01e13a0 con 0x7fa0a0071be0 2024-03-28T14:07:36.184 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:36.179+0000 7fa09effd640 1 --2- 172.21.15.171:0/141249442 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fa0a0072bf0 0x7fa0a00b2440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T14:07:36.184 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:36.179+0000 7fa09ffff640 1 --2- 172.21.15.171:0/141249442 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa0a0145ce0 0x7fa0a00b3ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T14:07:36.184 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:36.179+0000 7fa09f7fe640 1 --2- 172.21.15.171:0/141249442 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fa0a0071be0 0x7fa0a0138a80 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fa090019f90 tx=0x7fa09001aba0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:07:36.184 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:36.179+0000 7fa097fff640 1 -- 172.21.15.171:0/141249442 <== mon.1 v2:172.21.15.171:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0900b0050 con 0x7fa0a0071be0 2024-03-28T14:07:36.184 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:36.179+0000 7fa097fff640 1 -- 172.21.15.171:0/141249442 <== mon.1 v2:172.21.15.171:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa0900bc070 con 0x7fa0a0071be0 2024-03-28T14:07:36.184 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:36.179+0000 7fa0a5ec4640 1 -- 172.21.15.171:0/141249442 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa0a01e1680 con 0x7fa0a0071be0 2024-03-28T14:07:36.184 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:36.179+0000 7fa097fff640 1 -- 172.21.15.171:0/141249442 <== mon.1 v2:172.21.15.171:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0900b5a50 con 0x7fa0a0071be0 2024-03-28T14:07:36.184 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:36.179+0000 7fa0a5ec4640 1 -- 172.21.15.171:0/141249442 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa0a00b4400 con 0x7fa0a0071be0 2024-03-28T14:07:36.184 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:36.179+0000 7fa097fff640 1 -- 172.21.15.171:0/141249442 <== mon.1 v2:172.21.15.171:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fa0900ab4a0 con 0x7fa0a0071be0 2024-03-28T14:07:36.184 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:36.179+0000 7fa097fff640 1 -- 172.21.15.171:0/141249442 <== mon.1 v2:172.21.15.171:3300/0 5 ==== osd_map(5..5 src has 1..5) ==== 1070+0+0 (secure 0 0 0) 0x7fa0900b7450 con 0x7fa0a0071be0 2024-03-28T14:07:36.184 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:36.179+0000 7fa0a5ec4640 1 -- 172.21.15.171:0/141249442 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa0a00687c0 con 0x7fa0a0071be0 2024-03-28T14:07:36.186 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:36.183+0000 7fa097fff640 1 -- 172.21.15.171:0/141249442 <== mon.1 v2:172.21.15.171:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7fa090016070 con 0x7fa0a0071be0 2024-03-28T14:07:36.223 INFO:tasks.ceph.mgr.y.smithi104.stderr:2024-03-28T14:07:36.224+0000 7f57c9cb1180 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-28T14:07:36.254 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:36.251+0000 7fa0a5ec4640 1 -- 172.21.15.171:0/141249442 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_command({"prefix": "osd new", "uuid": "925c7e0a-38ef-47cd-b288-8bec8456fb17", "id": 3} v 0) -- 0x7fa0a00b9990 con 0x7fa0a0071be0 2024-03-28T14:07:36.261 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:36.259+0000 7fa097fff640 1 -- 172.21.15.171:0/141249442 <== mon.1 v2:172.21.15.171:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "925c7e0a-38ef-47cd-b288-8bec8456fb17", "id": 3}]=0 v6) ==== 112+0+2 (secure 0 0 0) 0x7fa0900c1460 con 0x7fa0a0071be0 2024-03-28T14:07:36.261 INFO:teuthology.orchestra.run.smithi171.stdout:3 2024-03-28T14:07:36.262 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:36.259+0000 7fa0a5ec4640 1 -- 172.21.15.171:0/141249442 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fa0a0071be0 msgr2=0x7fa0a0138a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:07:36.262 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:36.259+0000 7fa0a5ec4640 1 --2- 172.21.15.171:0/141249442 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fa0a0071be0 0x7fa0a0138a80 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fa090019f90 tx=0x7fa09001aba0 comp rx=0 tx=0).stop 2024-03-28T14:07:36.262 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:36.259+0000 7fa0a5ec4640 1 -- 172.21.15.171:0/141249442 shutdown_connections 2024-03-28T14:07:36.263 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:36.259+0000 7fa0a5ec4640 1 --2- 172.21.15.171:0/141249442 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa0a0145ce0 0x7fa0a00b3ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:07:36.263 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:36.259+0000 7fa0a5ec4640 1 --2- 172.21.15.171:0/141249442 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fa0a0072bf0 0x7fa0a00b2440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:07:36.263 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:36.259+0000 7fa0a5ec4640 1 --2- 172.21.15.171:0/141249442 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fa0a0071be0 0x7fa0a0138a80 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:07:36.263 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:36.259+0000 7fa0a5ec4640 1 -- 172.21.15.171:0/141249442 >> 172.21.15.171:0/141249442 conn(0x7fa0a00b7b10 msgr2=0x7fa0a00b5250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T14:07:36.263 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:36.259+0000 7fa0a5ec4640 1 -- 172.21.15.171:0/141249442 shutdown_connections 2024-03-28T14:07:36.263 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:36.259+0000 7fa0a5ec4640 1 -- 172.21.15.171:0/141249442 wait complete. 2024-03-28T14:07:36.278 DEBUG:teuthology.orchestra.run.smithi171:> sudo ceph --cluster ceph osd new 4a8fac86-e595-4a0e-96e3-57edb3cc3f57 4 2024-03-28T14:07:36.367 INFO:tasks.ceph.mgr.x.smithi171.stderr:2024-03-28T14:07:36.363+0000 7fa461f1b180 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-28T14:07:36.391 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:36.387+0000 7f47d1cd1640 1 Processor -- start 2024-03-28T14:07:36.391 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:36.387+0000 7f47d1cd1640 1 -- start start 2024-03-28T14:07:36.392 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:36.387+0000 7f47d1cd1640 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f47cc072bf0 0x7f47cc0b1e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:07:36.392 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:36.387+0000 7f47d1cd1640 1 --2- >> v2:172.21.15.171:3300/0 conn(0x7f47cc145ce0 0x7f47cc146130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:07:36.392 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:36.387+0000 7f47d1cd1640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f47cc072fd0 con 0x7f47cc071be0 2024-03-28T14:07:36.392 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:36.387+0000 7f47d1cd1640 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 -- 0x7f47cc073250 con 0x7f47cc072bf0 2024-03-28T14:07:36.392 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:36.387+0000 7f47d1cd1640 1 -- --> v2:172.21.15.171:3300/0 -- mon_getmap magic: 0 -- 0x7f47cc072050 con 0x7f47cc145ce0 2024-03-28T14:07:36.393 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:36.391+0000 7f47cbfff640 1 --2- >> v2:172.21.15.171:3300/0 conn(0x7f47cc145ce0 0x7f47cc146130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:07:36.393 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:36.391+0000 7f47caffd640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7f47cc071be0 0x7f47cc0b12f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:47284/0 (socket says 172.21.15.171:47284) 2024-03-28T14:07:36.393 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:36.391+0000 7f47caffd640 1 -- 172.21.15.171:0/3160959101 learned_addr learned my addr 172.21.15.171:0/3160959101 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-03-28T14:07:36.393 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:36.391+0000 7f47cb7fe640 1 --2- 172.21.15.171:0/3160959101 >> v2:172.21.15.104:3300/0 conn(0x7f47cc072bf0 0x7f47cc0b1e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:07:36.393 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:36.391+0000 7f47cbfff640 1 -- 172.21.15.171:0/3160959101 >> v2:172.21.15.104:3300/0 conn(0x7f47cc072bf0 msgr2=0x7f47cc0b1e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T14:07:36.393 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:36.391+0000 7f47cbfff640 1 --2- 172.21.15.171:0/3160959101 >> v2:172.21.15.104:3300/0 conn(0x7f47cc072bf0 0x7f47cc0b1e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:07:36.393 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:36.391+0000 7f47cbfff640 1 -- 172.21.15.171:0/3160959101 >> v1:172.21.15.171:6789/0 conn(0x7f47cc071be0 legacy=0x7f47cc0b12f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:07:36.393 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:36.391+0000 7f47cbfff640 1 -- 172.21.15.171:0/3160959101 --> v2:172.21.15.171:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f47cc0734a0 con 0x7f47cc145ce0 2024-03-28T14:07:36.394 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:36.391+0000 7f47cbfff640 1 --2- 172.21.15.171:0/3160959101 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f47cc145ce0 0x7f47cc146130 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f47cc0733d0 tx=0x7f47c401c810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=daa8da979cae68b8 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:07:36.394 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:36.391+0000 7f47ca7fc640 1 -- 172.21.15.171:0/3160959101 <== mon.1 v2:172.21.15.171:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47c40b2070 con 0x7f47cc145ce0 2024-03-28T14:07:36.394 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:36.391+0000 7f47ca7fc640 1 -- 172.21.15.171:0/3160959101 <== mon.1 v2:172.21.15.171:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f47c40a5780 con 0x7f47cc145ce0 2024-03-28T14:07:36.394 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:36.391+0000 7f47ca7fc640 1 -- 172.21.15.171:0/3160959101 <== mon.1 v2:172.21.15.171:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47c40a5b50 con 0x7f47cc145ce0 2024-03-28T14:07:36.395 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:36.391+0000 7f47d1cd1640 1 -- 172.21.15.171:0/3160959101 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f47cc145ce0 msgr2=0x7f47cc146130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:07:36.395 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:36.391+0000 7f47d1cd1640 1 --2- 172.21.15.171:0/3160959101 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f47cc145ce0 0x7f47cc146130 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f47cc0733d0 tx=0x7f47c401c810 comp rx=0 tx=0).stop 2024-03-28T14:07:36.395 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:36.391+0000 7f47d1cd1640 1 -- 172.21.15.171:0/3160959101 shutdown_connections 2024-03-28T14:07:36.395 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:36.391+0000 7f47d1cd1640 1 --2- 172.21.15.171:0/3160959101 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f47cc145ce0 0x7f47cc146130 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:07:36.395 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:36.391+0000 7f47d1cd1640 1 --2- 172.21.15.171:0/3160959101 >> v2:172.21.15.104:3300/0 conn(0x7f47cc072bf0 0x7f47cc0b1e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:07:36.395 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:36.391+0000 7f47d1cd1640 1 -- 172.21.15.171:0/3160959101 >> 172.21.15.171:0/3160959101 conn(0x7f47cc0b7b10 msgr2=0x7f47cc0b7f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T14:07:36.396 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:36.391+0000 7f47d1cd1640 1 -- 172.21.15.171:0/3160959101 shutdown_connections 2024-03-28T14:07:36.396 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:36.391+0000 7f47d1cd1640 1 -- 172.21.15.171:0/3160959101 wait complete. 2024-03-28T14:07:36.398 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:36.395+0000 7f47d1cd1640 1 Processor -- start 2024-03-28T14:07:36.399 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:36.395+0000 7f47d1cd1640 1 -- start start 2024-03-28T14:07:36.399 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:36.395+0000 7f47d1cd1640 1 --2- >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f47cc071be0 0x7f47cc09ded0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:07:36.399 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:36.395+0000 7f47d1cd1640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f47cc072bf0 0x7f47cc09f9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:07:36.399 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:36.395+0000 7f47d1cd1640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f47cc145ce0 0x7f47cc09eba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:07:36.399 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:36.395+0000 7f47d1cd1640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f47cc152830 con 0x7f47cc072bf0 2024-03-28T14:07:36.399 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:36.395+0000 7f47d1cd1640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f47cc1526b0 con 0x7f47cc145ce0 2024-03-28T14:07:36.399 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:36.395+0000 7f47d1cd1640 1 -- --> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] -- mon_getmap magic: 0 -- 0x7f47cc1529b0 con 0x7f47cc071be0 2024-03-28T14:07:36.399 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:36.395+0000 7f47cb7fe640 1 --2- >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f47cc071be0 0x7f47cc09ded0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:07:36.399 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:36.395+0000 7f47cbfff640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f47cc145ce0 0x7f47cc09eba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:07:36.399 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:36.395+0000 7f47cb7fe640 1 --2- >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f47cc071be0 0x7f47cc09ded0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.171:3301/0 says I am v2:172.21.15.171:52504/0 (socket says 172.21.15.171:52504) 2024-03-28T14:07:36.399 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:36.395+0000 7f47cbfff640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f47cc145ce0 0x7f47cc09eba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.171:3300/0 says I am v2:172.21.15.171:43026/0 (socket says 172.21.15.171:43026) 2024-03-28T14:07:36.399 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:36.395+0000 7f47cb7fe640 1 -- 172.21.15.171:0/4105097532 learned_addr learned my addr 172.21.15.171:0/4105097532 (peer_addr_for_me v2:172.21.15.171:0/0) 2024-03-28T14:07:36.400 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:36.395+0000 7f47caffd640 1 --2- 172.21.15.171:0/4105097532 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f47cc072bf0 0x7f47cc09f9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:07:36.400 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:36.395+0000 7f47cbfff640 1 -- 172.21.15.171:0/4105097532 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f47cc071be0 msgr2=0x7f47cc09ded0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:07:36.400 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:36.395+0000 7f47cbfff640 1 --2- 172.21.15.171:0/4105097532 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f47cc071be0 0x7f47cc09ded0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:07:36.400 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:36.395+0000 7f47cbfff640 1 -- 172.21.15.171:0/4105097532 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f47cc072bf0 msgr2=0x7f47cc09f9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:07:36.400 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:36.395+0000 7f47cbfff640 1 --2- 172.21.15.171:0/4105097532 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f47cc072bf0 0x7f47cc09f9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:07:36.400 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:36.395+0000 7f47cbfff640 1 -- 172.21.15.171:0/4105097532 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f47cc0a0210 con 0x7f47cc145ce0 2024-03-28T14:07:36.400 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:36.395+0000 7f47cbfff640 1 --2- 172.21.15.171:0/4105097532 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f47cc145ce0 0x7f47cc09eba0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f47c40ac000 tx=0x7f47c40195b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:07:36.400 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:36.395+0000 7f47b3fff640 1 -- 172.21.15.171:0/4105097532 <== mon.1 v2:172.21.15.171:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47c40b2050 con 0x7f47cc145ce0 2024-03-28T14:07:36.400 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:36.395+0000 7f47b3fff640 1 -- 172.21.15.171:0/4105097532 <== mon.1 v2:172.21.15.171:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f47c40b3040 con 0x7f47cc145ce0 2024-03-28T14:07:36.400 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:36.395+0000 7f47d1cd1640 1 -- 172.21.15.171:0/4105097532 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f47cc14f520 con 0x7f47cc145ce0 2024-03-28T14:07:36.400 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:36.395+0000 7f47b3fff640 1 -- 172.21.15.171:0/4105097532 <== mon.1 v2:172.21.15.171:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47c40ad650 con 0x7f47cc145ce0 2024-03-28T14:07:36.400 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:36.395+0000 7f47d1cd1640 1 -- 172.21.15.171:0/4105097532 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f47cc0b4280 con 0x7f47cc145ce0 2024-03-28T14:07:36.401 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:36.395+0000 7f47b3fff640 1 -- 172.21.15.171:0/4105097532 <== mon.1 v2:172.21.15.171:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f47c40c0020 con 0x7f47cc145ce0 2024-03-28T14:07:36.401 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:36.395+0000 7f47b3fff640 1 -- 172.21.15.171:0/4105097532 <== mon.1 v2:172.21.15.171:3300/0 5 ==== osd_map(6..6 src has 1..6) ==== 1185+0+0 (secure 0 0 0) 0x7f47c40b9b40 con 0x7f47cc145ce0 2024-03-28T14:07:36.401 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:36.395+0000 7f47d1cd1640 1 -- 172.21.15.171:0/4105097532 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f47cc0687c0 con 0x7f47cc145ce0 2024-03-28T14:07:36.401 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:36.399+0000 7f47b3fff640 1 -- 172.21.15.171:0/4105097532 <== mon.1 v2:172.21.15.171:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7f47c40b9e20 con 0x7f47cc145ce0 2024-03-28T14:07:36.423 INFO:tasks.ceph.mgr.x.smithi171.stderr:2024-03-28T14:07:36.423+0000 7fa461f1b180 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-28T14:07:36.468 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:36.467+0000 7f47d1cd1640 1 -- 172.21.15.171:0/4105097532 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_command({"prefix": "osd new", "uuid": "4a8fac86-e595-4a0e-96e3-57edb3cc3f57", "id": 4} v 0) -- 0x7f47cc0b9060 con 0x7f47cc145ce0 2024-03-28T14:07:36.475 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:36.471+0000 7f47b3fff640 1 -- 172.21.15.171:0/4105097532 <== mon.1 v2:172.21.15.171:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "4a8fac86-e595-4a0e-96e3-57edb3cc3f57", "id": 4}]=0 v7) ==== 112+0+2 (secure 0 0 0) 0x7f47c4016520 con 0x7f47cc145ce0 2024-03-28T14:07:36.475 INFO:teuthology.orchestra.run.smithi171.stdout:4 2024-03-28T14:07:36.476 INFO:tasks.ceph.mgr.x.smithi171.stderr:2024-03-28T14:07:36.475+0000 7fa461f1b180 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-28T14:07:36.477 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:36.475+0000 7f47d1cd1640 1 -- 172.21.15.171:0/4105097532 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f47cc145ce0 msgr2=0x7f47cc09eba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:07:36.477 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:36.475+0000 7f47d1cd1640 1 --2- 172.21.15.171:0/4105097532 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f47cc145ce0 0x7f47cc09eba0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f47c40ac000 tx=0x7f47c40195b0 comp rx=0 tx=0).stop 2024-03-28T14:07:36.477 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:36.475+0000 7f47d1cd1640 1 -- 172.21.15.171:0/4105097532 shutdown_connections 2024-03-28T14:07:36.477 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:36.475+0000 7f47d1cd1640 1 --2- 172.21.15.171:0/4105097532 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f47cc145ce0 0x7f47cc09eba0 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:07:36.477 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:36.475+0000 7f47d1cd1640 1 --2- 172.21.15.171:0/4105097532 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f47cc072bf0 0x7f47cc09f9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:07:36.477 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:36.475+0000 7f47d1cd1640 1 --2- 172.21.15.171:0/4105097532 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f47cc071be0 0x7f47cc09ded0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:07:36.477 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:36.475+0000 7f47d1cd1640 1 -- 172.21.15.171:0/4105097532 >> 172.21.15.171:0/4105097532 conn(0x7f47cc0b7b10 msgr2=0x7f47cc0b9580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T14:07:36.477 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:36.475+0000 7f47d1cd1640 1 -- 172.21.15.171:0/4105097532 shutdown_connections 2024-03-28T14:07:36.478 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:36.475+0000 7f47d1cd1640 1 -- 172.21.15.171:0/4105097532 wait complete. 2024-03-28T14:07:36.492 DEBUG:teuthology.orchestra.run.smithi171:> sudo ceph --cluster ceph osd new e48d50b0-beae-40ae-b15a-9a84dbfe88ab 5 2024-03-28T14:07:36.558 INFO:tasks.ceph.mgr.x.smithi171.stderr:2024-03-28T14:07:36.555+0000 7fa461f1b180 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-28T14:07:36.606 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:36.603+0000 7f03c1c28640 1 Processor -- start 2024-03-28T14:07:36.606 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:36.603+0000 7f03c1c28640 1 -- start start 2024-03-28T14:07:36.608 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:36.603+0000 7f03c1c28640 1 --2- >> v2:172.21.15.171:3300/0 conn(0x7f03bc071e00 0x7f03bc0b0fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:07:36.608 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:36.603+0000 7f03c1c28640 1 --2- >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f03bc1492e0 0x7f03bc13f330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:07:36.608 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:36.603+0000 7f03c1c28640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f03bc0721e0 con 0x7f03bc0a0e30 2024-03-28T14:07:36.608 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:36.603+0000 7f03c1c28640 1 -- --> v2:172.21.15.171:3300/0 -- mon_getmap magic: 0 -- 0x7f03bc0b88a0 con 0x7f03bc071e00 2024-03-28T14:07:36.608 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:36.603+0000 7f03c1c28640 1 -- --> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] -- mon_getmap magic: 0 -- 0x7f03bc0b1650 con 0x7f03bc1492e0 2024-03-28T14:07:36.608 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:36.603+0000 7f03bb7fe640 1 --2- >> v2:172.21.15.171:3300/0 conn(0x7f03bc071e00 0x7f03bc0b0fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:07:36.608 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:36.603+0000 7f03bbfff640 1 --2- >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f03bc1492e0 0x7f03bc13f330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:07:36.608 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:36.603+0000 7f03bb7fe640 1 --2- >> v2:172.21.15.171:3300/0 conn(0x7f03bc071e00 0x7f03bc0b0fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.171:3300/0 says I am v2:172.21.15.171:43038/0 (socket says 172.21.15.171:43038) 2024-03-28T14:07:36.609 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:36.603+0000 7f03bb7fe640 1 -- 172.21.15.171:0/1921980713 learned_addr learned my addr 172.21.15.171:0/1921980713 (peer_addr_for_me v2:172.21.15.171:0/0) 2024-03-28T14:07:36.609 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:36.603+0000 7f03bbfff640 1 -- 172.21.15.171:0/1921980713 >> v2:172.21.15.171:3300/0 conn(0x7f03bc071e00 msgr2=0x7f03bc0b0fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T14:07:36.609 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:36.603+0000 7f03bbfff640 1 --2- 172.21.15.171:0/1921980713 >> v2:172.21.15.171:3300/0 conn(0x7f03bc071e00 0x7f03bc0b0fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:07:36.609 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:36.603+0000 7f03bbfff640 1 -- 172.21.15.171:0/1921980713 >> v1:172.21.15.104:6789/0 conn(0x7f03bc0a0e30 legacy=0x7f03bc148bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:07:36.609 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:36.603+0000 7f03bbfff640 1 -- 172.21.15.171:0/1921980713 --> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f03bc0b1930 con 0x7f03bc1492e0 2024-03-28T14:07:36.609 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:36.603+0000 7f03bb7fe640 1 --2- 172.21.15.171:0/1921980713 >> v2:172.21.15.171:3300/0 conn(0x7f03bc071e00 0x7f03bc0b0fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T14:07:36.611 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:36.607+0000 7f03bbfff640 1 --2- 172.21.15.171:0/1921980713 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f03bc1492e0 0x7f03bc13f330 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f03b4009d90 tx=0x7f03b401c8b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c05fb43214e30245 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:07:36.611 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:36.607+0000 7f03baffd640 1 -- 172.21.15.171:0/1921980713 <== mon.2 v2:172.21.15.171:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03b40b2070 con 0x7f03bc1492e0 2024-03-28T14:07:36.611 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:36.607+0000 7f03baffd640 1 -- 172.21.15.171:0/1921980713 <== mon.2 v2:172.21.15.171:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f03b40a5950 con 0x7f03bc1492e0 2024-03-28T14:07:36.611 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:36.607+0000 7f03baffd640 1 -- 172.21.15.171:0/1921980713 <== mon.2 v2:172.21.15.171:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03b40a5d20 con 0x7f03bc1492e0 2024-03-28T14:07:36.611 INFO:tasks.ceph.mgr.x.smithi171.stderr:2024-03-28T14:07:36.607+0000 7fa461f1b180 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-28T14:07:36.612 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:36.607+0000 7f03c1c28640 1 -- 172.21.15.171:0/1921980713 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f03bc1492e0 msgr2=0x7f03bc13f330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:07:36.613 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:36.607+0000 7f03c1c28640 1 --2- 172.21.15.171:0/1921980713 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f03bc1492e0 0x7f03bc13f330 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f03b4009d90 tx=0x7f03b401c8b0 comp rx=0 tx=0).stop 2024-03-28T14:07:36.613 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:36.611+0000 7f03c1c28640 1 -- 172.21.15.171:0/1921980713 shutdown_connections 2024-03-28T14:07:36.613 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:36.611+0000 7f03c1c28640 1 --2- 172.21.15.171:0/1921980713 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f03bc1492e0 0x7f03bc13f330 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:07:36.613 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:36.611+0000 7f03c1c28640 1 --2- 172.21.15.171:0/1921980713 >> v2:172.21.15.171:3300/0 conn(0x7f03bc071e00 0x7f03bc0b0fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:07:36.613 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:36.611+0000 7f03c1c28640 1 -- 172.21.15.171:0/1921980713 >> 172.21.15.171:0/1921980713 conn(0x7f03bc0b7a10 msgr2=0x7f03bc0b7e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T14:07:36.613 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:36.611+0000 7f03c1c28640 1 -- 172.21.15.171:0/1921980713 shutdown_connections 2024-03-28T14:07:36.613 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:36.611+0000 7f03c1c28640 1 -- 172.21.15.171:0/1921980713 wait complete. 2024-03-28T14:07:36.613 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:36.611+0000 7f03c1c28640 1 Processor -- start 2024-03-28T14:07:36.613 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:36.611+0000 7f03c1c28640 1 -- start start 2024-03-28T14:07:36.616 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:36.611+0000 7f03c1c28640 1 --2- >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f03bc071e00 0x7f03bc06d5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:07:36.616 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:36.611+0000 7f03c1c28640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f03bc0a0e30 0x7f03bc06c6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:07:36.616 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:36.611+0000 7f03c1c28640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f03bc1492e0 0x7f03bc06cc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:07:36.616 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:36.611+0000 7f03c1c28640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f03bc1529d0 con 0x7f03bc0a0e30 2024-03-28T14:07:36.616 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:36.611+0000 7f03c1c28640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f03bc152850 con 0x7f03bc1492e0 2024-03-28T14:07:36.616 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:36.611+0000 7f03c1c28640 1 -- --> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] -- mon_getmap magic: 0 -- 0x7f03bc152b50 con 0x7f03bc071e00 2024-03-28T14:07:36.616 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:36.611+0000 7f03bb7fe640 1 --2- >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f03bc071e00 0x7f03bc06d5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:07:36.616 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:36.611+0000 7f03bbfff640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f03bc1492e0 0x7f03bc06cc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:07:36.617 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:36.611+0000 7f03bb7fe640 1 --2- >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f03bc071e00 0x7f03bc06d5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.171:3301/0 says I am v2:172.21.15.171:52522/0 (socket says 172.21.15.171:52522) 2024-03-28T14:07:36.617 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:36.611+0000 7f03bb7fe640 1 -- 172.21.15.171:0/1788795292 learned_addr learned my addr 172.21.15.171:0/1788795292 (peer_addr_for_me v2:172.21.15.171:0/0) 2024-03-28T14:07:36.617 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:36.611+0000 7f03b3fff640 1 --2- 172.21.15.171:0/1788795292 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f03bc0a0e30 0x7f03bc06c6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:07:36.617 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:36.611+0000 7f03bbfff640 1 -- 172.21.15.171:0/1788795292 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f03bc071e00 msgr2=0x7f03bc06d5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:07:36.617 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:36.611+0000 7f03bbfff640 1 --2- 172.21.15.171:0/1788795292 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f03bc071e00 0x7f03bc06d5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:07:36.617 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:36.611+0000 7f03bbfff640 1 -- 172.21.15.171:0/1788795292 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f03bc0a0e30 msgr2=0x7f03bc06c6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:07:36.617 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:36.611+0000 7f03bbfff640 1 --2- 172.21.15.171:0/1788795292 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f03bc0a0e30 0x7f03bc06c6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:07:36.617 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:36.611+0000 7f03bbfff640 1 -- 172.21.15.171:0/1788795292 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f03bc140910 con 0x7f03bc1492e0 2024-03-28T14:07:36.617 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:36.611+0000 7f03bb7fe640 1 --2- 172.21.15.171:0/1788795292 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f03bc071e00 0x7f03bc06d5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T14:07:36.617 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:36.611+0000 7f03bbfff640 1 --2- 172.21.15.171:0/1788795292 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f03bc1492e0 0x7f03bc06cc20 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f03bc0b17d0 tx=0x7f03b4019650 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:07:36.617 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:36.611+0000 7f03b8ff9640 1 -- 172.21.15.171:0/1788795292 <== mon.1 v2:172.21.15.171:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03b40b2050 con 0x7f03bc1492e0 2024-03-28T14:07:36.617 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:36.611+0000 7f03b8ff9640 1 -- 172.21.15.171:0/1788795292 <== mon.1 v2:172.21.15.171:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f03b40b3040 con 0x7f03bc1492e0 2024-03-28T14:07:36.617 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:36.611+0000 7f03c1c28640 1 -- 172.21.15.171:0/1788795292 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f03bc14f9c0 con 0x7f03bc1492e0 2024-03-28T14:07:36.618 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:36.611+0000 7f03b8ff9640 1 -- 172.21.15.171:0/1788795292 <== mon.1 v2:172.21.15.171:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03b40ad850 con 0x7f03bc1492e0 2024-03-28T14:07:36.618 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:36.611+0000 7f03c1c28640 1 -- 172.21.15.171:0/1788795292 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f03bc140630 con 0x7f03bc1492e0 2024-03-28T14:07:36.618 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:36.611+0000 7f03b8ff9640 1 -- 172.21.15.171:0/1788795292 <== mon.1 v2:172.21.15.171:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f03b40c0020 con 0x7f03bc1492e0 2024-03-28T14:07:36.618 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:36.611+0000 7f03b8ff9640 1 -- 172.21.15.171:0/1788795292 <== mon.1 v2:172.21.15.171:3300/0 5 ==== osd_map(7..7 src has 1..7) ==== 1300+0+0 (secure 0 0 0) 0x7f03b40b9d10 con 0x7f03bc1492e0 2024-03-28T14:07:36.618 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:36.611+0000 7f03c1c28640 1 -- 172.21.15.171:0/1788795292 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f03bc02d560 con 0x7f03bc1492e0 2024-03-28T14:07:36.618 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:36.615+0000 7f03b8ff9640 1 -- 172.21.15.171:0/1788795292 <== mon.1 v2:172.21.15.171:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7f03b4016560 con 0x7f03bc1492e0 2024-03-28T14:07:36.685 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:36.683+0000 7f03c1c28640 1 -- 172.21.15.171:0/1788795292 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_command({"prefix": "osd new", "uuid": "e48d50b0-beae-40ae-b15a-9a84dbfe88ab", "id": 5} v 0) -- 0x7f03bc0687c0 con 0x7f03bc1492e0 2024-03-28T14:07:36.690 INFO:tasks.ceph.mgr.x.smithi171.stderr:2024-03-28T14:07:36.687+0000 7fa461f1b180 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-28T14:07:36.692 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:36.691+0000 7f03b8ff9640 1 -- 172.21.15.171:0/1788795292 <== mon.1 v2:172.21.15.171:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "e48d50b0-beae-40ae-b15a-9a84dbfe88ab", "id": 5}]=0 v8) ==== 112+0+2 (secure 0 0 0) 0x7f03b40a57d0 con 0x7f03bc1492e0 2024-03-28T14:07:36.692 INFO:teuthology.orchestra.run.smithi171.stdout:5 2024-03-28T14:07:36.693 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:36.691+0000 7f03c1c28640 1 -- 172.21.15.171:0/1788795292 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f03bc1492e0 msgr2=0x7f03bc06cc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:07:36.693 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:36.691+0000 7f03c1c28640 1 --2- 172.21.15.171:0/1788795292 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f03bc1492e0 0x7f03bc06cc20 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f03bc0b17d0 tx=0x7f03b4019650 comp rx=0 tx=0).stop 2024-03-28T14:07:36.693 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:36.691+0000 7f03c1c28640 1 -- 172.21.15.171:0/1788795292 shutdown_connections 2024-03-28T14:07:36.693 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:36.691+0000 7f03c1c28640 1 --2- 172.21.15.171:0/1788795292 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f03bc1492e0 0x7f03bc06cc20 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:07:36.693 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:36.691+0000 7f03c1c28640 1 --2- 172.21.15.171:0/1788795292 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f03bc0a0e30 0x7f03bc06c6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:07:36.694 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:36.691+0000 7f03c1c28640 1 --2- 172.21.15.171:0/1788795292 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f03bc071e00 0x7f03bc06d5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:07:36.694 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:36.691+0000 7f03c1c28640 1 -- 172.21.15.171:0/1788795292 >> 172.21.15.171:0/1788795292 conn(0x7f03bc0b7a10 msgr2=0x7f03bc1468d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T14:07:36.694 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:36.691+0000 7f03c1c28640 1 -- 172.21.15.171:0/1788795292 shutdown_connections 2024-03-28T14:07:36.694 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:36.691+0000 7f03c1c28640 1 -- 172.21.15.171:0/1788795292 wait complete. 2024-03-28T14:07:36.708 DEBUG:teuthology.orchestra.run.smithi171:> sudo ceph --cluster ceph osd new 5e7c3292-9936-4f35-b267-aed9d4e49323 6 2024-03-28T14:07:36.746 INFO:tasks.ceph.mgr.x.smithi171.stderr:2024-03-28T14:07:36.743+0000 7fa461f1b180 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-28T14:07:36.769 INFO:tasks.ceph.mgr.y.smithi104.stderr:2024-03-28T14:07:36.768+0000 7f57c9cb1180 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-28T14:07:36.822 INFO:tasks.ceph.mgr.y.smithi104.stderr:2024-03-28T14:07:36.820+0000 7f57c9cb1180 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-28T14:07:36.824 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:36.819+0000 7f2ff40ac640 1 Processor -- start 2024-03-28T14:07:36.824 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:36.819+0000 7f2ff40ac640 1 -- start start 2024-03-28T14:07:36.824 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:36.823+0000 7f2ff40ac640 1 --2- >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f2fec02c460 0x7f2fec0b54e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:07:36.824 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:36.823+0000 7f2ff40ac640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2fec02c840 con 0x7f2fec1487e0 2024-03-28T14:07:36.824 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:36.823+0000 7f2ff40ac640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2fec0aec70 con 0x7f2fec09e8e0 2024-03-28T14:07:36.824 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:36.823+0000 7f2ff40ac640 1 -- --> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] -- mon_getmap magic: 0 -- 0x7f2fec147c40 con 0x7f2fec02c460 2024-03-28T14:07:36.824 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:36.823+0000 7f2ff1e48640 1 --2- >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f2fec02c460 0x7f2fec0b54e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:07:36.824 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:36.823+0000 7f2ff1647640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7f2fec09e8e0 0x7f2fec1480d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:47292/0 (socket says 172.21.15.171:47292) 2024-03-28T14:07:36.824 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:36.823+0000 7f2ff1647640 1 -- 172.21.15.171:0/2658718433 learned_addr learned my addr 172.21.15.171:0/2658718433 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-03-28T14:07:36.825 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:36.823+0000 7f2ff1e48640 1 -- 172.21.15.171:0/2658718433 >> v1:172.21.15.171:6789/0 conn(0x7f2fec09e8e0 legacy=0x7f2fec1480d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:07:36.825 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:36.823+0000 7f2ff1e48640 1 -- 172.21.15.171:0/2658718433 >> v1:172.21.15.104:6789/0 conn(0x7f2fec1487e0 legacy=0x7f2fec14ef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T14:07:36.825 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:36.823+0000 7f2ff1e48640 1 -- 172.21.15.171:0/2658718433 --> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2fec0aef50 con 0x7f2fec02c460 2024-03-28T14:07:36.825 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:36.823+0000 7f2ff0e46640 1 -- 172.21.15.171:0/2658718433 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 802155483 0 0) 0x7f2fec0aec70 con 0x7f2fec09e8e0 2024-03-28T14:07:36.826 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:36.823+0000 7f2ff1e48640 1 --2- 172.21.15.171:0/2658718433 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f2fec02c460 0x7f2fec0b54e0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f2fe8009820 tx=0x7f2fe801c480 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1c52df8f985bae55 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:07:36.826 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:36.823+0000 7f2ff0e46640 1 -- 172.21.15.171:0/2658718433 <== mon.2 v2:172.21.15.171:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2fe80b0070 con 0x7f2fec02c460 2024-03-28T14:07:36.826 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:36.823+0000 7f2ff0e46640 1 -- 172.21.15.171:0/2658718433 <== mon.2 v2:172.21.15.171:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2fe80ab600 con 0x7f2fec02c460 2024-03-28T14:07:36.826 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:36.823+0000 7f2ff0e46640 1 -- 172.21.15.171:0/2658718433 <== mon.2 v2:172.21.15.171:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2fe80ab9d0 con 0x7f2fec02c460 2024-03-28T14:07:36.827 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:36.823+0000 7f2ff40ac640 1 -- 172.21.15.171:0/2658718433 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f2fec02c460 msgr2=0x7f2fec0b54e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:07:36.827 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:36.823+0000 7f2ff40ac640 1 --2- 172.21.15.171:0/2658718433 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f2fec02c460 0x7f2fec0b54e0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f2fe8009820 tx=0x7f2fe801c480 comp rx=0 tx=0).stop 2024-03-28T14:07:36.828 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:36.823+0000 7f2ff40ac640 1 -- 172.21.15.171:0/2658718433 shutdown_connections 2024-03-28T14:07:36.828 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:36.823+0000 7f2ff40ac640 1 --2- 172.21.15.171:0/2658718433 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f2fec02c460 0x7f2fec0b54e0 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:07:36.828 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:36.823+0000 7f2ff40ac640 1 -- 172.21.15.171:0/2658718433 >> 172.21.15.171:0/2658718433 conn(0x7f2fec0b95f0 msgr2=0x7f2fec0b99d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T14:07:36.828 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:36.823+0000 7f2ff40ac640 1 -- 172.21.15.171:0/2658718433 shutdown_connections 2024-03-28T14:07:36.828 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:36.823+0000 7f2ff40ac640 1 -- 172.21.15.171:0/2658718433 wait complete. 2024-03-28T14:07:36.830 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:36.827+0000 7f2ff40ac640 1 Processor -- start 2024-03-28T14:07:36.830 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:36.827+0000 7f2ff40ac640 1 -- start start 2024-03-28T14:07:36.830 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:36.827+0000 7f2ff40ac640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2fec02c460 0x7f2fec13e1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:07:36.830 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:36.827+0000 7f2ff40ac640 1 --2- >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f2fec09e8e0 0x7f2fec13e720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:07:36.830 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:36.827+0000 7f2ff40ac640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f2fec1487e0 0x7f2fec1e1ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:07:36.830 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:36.827+0000 7f2ff40ac640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f2fec153070 con 0x7f2fec02c460 2024-03-28T14:07:36.831 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:36.827+0000 7f2ff40ac640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f2fec0b0830 con 0x7f2fec1487e0 2024-03-28T14:07:36.831 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:36.827+0000 7f2ff40ac640 1 -- --> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] -- mon_getmap magic: 0 -- 0x7f2fec1531f0 con 0x7f2fec09e8e0 2024-03-28T14:07:36.831 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:36.827+0000 7f2ff1647640 1 --2- >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f2fec09e8e0 0x7f2fec13e720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:07:36.831 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:36.827+0000 7f2ff1647640 1 --2- >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f2fec09e8e0 0x7f2fec13e720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.171:3301/0 says I am v2:172.21.15.171:52542/0 (socket says 172.21.15.171:52542) 2024-03-28T14:07:36.831 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:36.827+0000 7f2ff1647640 1 -- 172.21.15.171:0/221520904 learned_addr learned my addr 172.21.15.171:0/221520904 (peer_addr_for_me v2:172.21.15.171:0/0) 2024-03-28T14:07:36.831 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:36.827+0000 7f2ff2649640 1 --2- 172.21.15.171:0/221520904 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f2fec1487e0 0x7f2fec1e1ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:07:36.831 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:36.827+0000 7f2ff1e48640 1 --2- 172.21.15.171:0/221520904 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2fec02c460 0x7f2fec13e1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:07:36.831 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:36.827+0000 7f2ff1647640 1 -- 172.21.15.171:0/221520904 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f2fec1487e0 msgr2=0x7f2fec1e1ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:07:36.831 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:36.827+0000 7f2ff1647640 1 --2- 172.21.15.171:0/221520904 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f2fec1487e0 0x7f2fec1e1ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:07:36.831 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:36.827+0000 7f2ff1647640 1 -- 172.21.15.171:0/221520904 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2fec02c460 msgr2=0x7f2fec13e1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:07:36.831 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:36.827+0000 7f2ff1647640 1 --2- 172.21.15.171:0/221520904 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2fec02c460 0x7f2fec13e1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:07:36.831 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:36.827+0000 7f2ff1647640 1 -- 172.21.15.171:0/221520904 --> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2fec143570 con 0x7f2fec09e8e0 2024-03-28T14:07:36.831 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:36.827+0000 7f2ff1e48640 1 --2- 172.21.15.171:0/221520904 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2fec02c460 0x7f2fec13e1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T14:07:36.831 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:36.827+0000 7f2ff1647640 1 --2- 172.21.15.171:0/221520904 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f2fec09e8e0 0x7f2fec13e720 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f2fdc0011f0 tx=0x7f2fdc00c9b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:07:36.832 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:36.827+0000 7f2fe27fc640 1 -- 172.21.15.171:0/221520904 <== mon.2 v2:172.21.15.171:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2fdc018070 con 0x7f2fec09e8e0 2024-03-28T14:07:36.832 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:36.827+0000 7f2fe27fc640 1 -- 172.21.15.171:0/221520904 <== mon.2 v2:172.21.15.171:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2fdc0074e0 con 0x7f2fec09e8e0 2024-03-28T14:07:36.832 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:36.827+0000 7f2fe27fc640 1 -- 172.21.15.171:0/221520904 <== mon.2 v2:172.21.15.171:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2fdc004a10 con 0x7f2fec09e8e0 2024-03-28T14:07:36.832 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:36.827+0000 7f2ff40ac640 1 -- 172.21.15.171:0/221520904 --> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2fec0b2460 con 0x7f2fec09e8e0 2024-03-28T14:07:36.832 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:36.827+0000 7f2ff40ac640 1 -- 172.21.15.171:0/221520904 --> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f2fec09ccc0 con 0x7f2fec09e8e0 2024-03-28T14:07:36.832 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:36.827+0000 7f2fe27fc640 1 -- 172.21.15.171:0/221520904 <== mon.2 v2:172.21.15.171:3301/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f2fdc004bb0 con 0x7f2fec09e8e0 2024-03-28T14:07:36.832 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:36.827+0000 7f2fe27fc640 1 -- 172.21.15.171:0/221520904 <== mon.2 v2:172.21.15.171:3301/0 5 ==== osd_map(8..8 src has 1..8) ==== 1415+0+0 (secure 0 0 0) 0x7f2fdc01b950 con 0x7f2fec09e8e0 2024-03-28T14:07:36.832 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:36.827+0000 7f2ff40ac640 1 -- 172.21.15.171:0/221520904 --> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2fec0687c0 con 0x7f2fec09e8e0 2024-03-28T14:07:36.833 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:36.831+0000 7f2fe27fc640 1 -- 172.21.15.171:0/221520904 <== mon.2 v2:172.21.15.171:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7f2fdc021070 con 0x7f2fec09e8e0 2024-03-28T14:07:36.875 INFO:tasks.ceph.mgr.y.smithi104.stderr:2024-03-28T14:07:36.876+0000 7f57c9cb1180 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-28T14:07:36.901 INFO:tasks.ceph.mgr.x.smithi171.stderr:2024-03-28T14:07:36.899+0000 7fa461f1b180 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-28T14:07:36.902 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:36.899+0000 7f2ff40ac640 1 -- 172.21.15.171:0/221520904 --> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] -- mon_command({"prefix": "osd new", "uuid": "5e7c3292-9936-4f35-b267-aed9d4e49323", "id": 6} v 0) -- 0x7f2fec09e150 con 0x7f2fec09e8e0 2024-03-28T14:07:36.910 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:36.907+0000 7f2fe27fc640 1 -- 172.21.15.171:0/221520904 <== mon.2 v2:172.21.15.171:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "5e7c3292-9936-4f35-b267-aed9d4e49323", "id": 6}]=0 v9) ==== 112+0+2 (secure 0 0 0) 0x7f2fdc01bbc0 con 0x7f2fec09e8e0 2024-03-28T14:07:36.910 INFO:teuthology.orchestra.run.smithi171.stdout:6 2024-03-28T14:07:36.911 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:36.907+0000 7f2ff40ac640 1 -- 172.21.15.171:0/221520904 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f2fec09e8e0 msgr2=0x7f2fec13e720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:07:36.911 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:36.907+0000 7f2ff40ac640 1 --2- 172.21.15.171:0/221520904 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f2fec09e8e0 0x7f2fec13e720 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f2fdc0011f0 tx=0x7f2fdc00c9b0 comp rx=0 tx=0).stop 2024-03-28T14:07:36.911 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:36.907+0000 7f2ff40ac640 1 -- 172.21.15.171:0/221520904 shutdown_connections 2024-03-28T14:07:36.911 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:36.907+0000 7f2ff40ac640 1 --2- 172.21.15.171:0/221520904 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f2fec1487e0 0x7f2fec1e1ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:07:36.911 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:36.907+0000 7f2ff40ac640 1 --2- 172.21.15.171:0/221520904 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f2fec09e8e0 0x7f2fec13e720 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:07:36.911 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:36.907+0000 7f2ff40ac640 1 --2- 172.21.15.171:0/221520904 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2fec02c460 0x7f2fec13e1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:07:36.911 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:36.907+0000 7f2ff40ac640 1 -- 172.21.15.171:0/221520904 >> 172.21.15.171:0/221520904 conn(0x7f2fec0b95f0 msgr2=0x7f2fec09c490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T14:07:36.911 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:36.907+0000 7f2ff40ac640 1 -- 172.21.15.171:0/221520904 shutdown_connections 2024-03-28T14:07:36.911 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:36.907+0000 7f2ff40ac640 1 -- 172.21.15.171:0/221520904 wait complete. 2024-03-28T14:07:36.926 DEBUG:teuthology.orchestra.run.smithi171:> sudo ceph --cluster ceph osd new 339b8414-d7f0-4fb0-a446-efaf635cb7ff 7 2024-03-28T14:07:36.953 INFO:tasks.ceph.mgr.y.smithi104.stderr:2024-03-28T14:07:36.952+0000 7f57c9cb1180 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-28T14:07:36.964 INFO:tasks.ceph.mgr.x.smithi171.stderr:2024-03-28T14:07:36.963+0000 7fa461f1b180 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-28T14:07:37.003 INFO:tasks.ceph.mgr.y.smithi104.stderr:2024-03-28T14:07:37.004+0000 7f57c9cb1180 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-28T14:07:37.074 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:37.071+0000 7f2a98495640 1 Processor -- start 2024-03-28T14:07:37.075 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:37.071+0000 7f2a98495640 1 -- start start 2024-03-28T14:07:37.076 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:37.071+0000 7f2a98495640 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f2a9006dd10 0x7f2a90072cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:07:37.076 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:37.071+0000 7f2a98495640 1 --2- >> v2:172.21.15.171:3300/0 conn(0x7f2a900b5600 0x7f2a901417b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:07:37.076 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:37.071+0000 7f2a98495640 1 --2- >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f2a900b5120 0x7f2a90141cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:07:37.077 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:37.071+0000 7f2a98495640 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 -- 0x7f2a90073230 con 0x7f2a9006dd10 2024-03-28T14:07:37.077 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:37.071+0000 7f2a98495640 1 -- --> v2:172.21.15.171:3300/0 -- mon_getmap magic: 0 -- 0x7f2a9006e0f0 con 0x7f2a900b5600 2024-03-28T14:07:37.077 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:37.071+0000 7f2a98495640 1 -- --> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] -- mon_getmap magic: 0 -- 0x7f2a900b0ee0 con 0x7f2a900b5120 2024-03-28T14:07:37.077 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:37.075+0000 7f2a95a30640 1 --2- >> v2:172.21.15.171:3300/0 conn(0x7f2a900b5600 0x7f2a901417b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:07:37.077 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:37.075+0000 7f2a96a32640 1 --2- >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f2a900b5120 0x7f2a90141cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:07:37.077 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:37.075+0000 7f2a95a30640 1 --2- >> v2:172.21.15.171:3300/0 conn(0x7f2a900b5600 0x7f2a901417b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.171:3300/0 says I am v2:172.21.15.171:43062/0 (socket says 172.21.15.171:43062) 2024-03-28T14:07:37.077 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:37.075+0000 7f2a95a30640 1 -- 172.21.15.171:0/3395526432 learned_addr learned my addr 172.21.15.171:0/3395526432 (peer_addr_for_me v2:172.21.15.171:0/0) 2024-03-28T14:07:37.077 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:37.075+0000 7f2a96231640 1 --2- 172.21.15.171:0/3395526432 >> v2:172.21.15.104:3300/0 conn(0x7f2a9006dd10 0x7f2a90072cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:07:37.077 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:37.075+0000 7f2a96a32640 1 -- 172.21.15.171:0/3395526432 >> v2:172.21.15.171:3300/0 conn(0x7f2a900b5600 msgr2=0x7f2a901417b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T14:07:37.077 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:37.075+0000 7f2a96a32640 1 --2- 172.21.15.171:0/3395526432 >> v2:172.21.15.171:3300/0 conn(0x7f2a900b5600 0x7f2a901417b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:07:37.077 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:37.075+0000 7f2a96a32640 1 -- 172.21.15.171:0/3395526432 >> v2:172.21.15.104:3300/0 conn(0x7f2a9006dd10 msgr2=0x7f2a90072cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T14:07:37.077 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:37.075+0000 7f2a96a32640 1 --2- 172.21.15.171:0/3395526432 >> v2:172.21.15.104:3300/0 conn(0x7f2a9006dd10 0x7f2a90072cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:07:37.077 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:37.075+0000 7f2a96a32640 1 -- 172.21.15.171:0/3395526432 --> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2a900b0b70 con 0x7f2a900b5120 2024-03-28T14:07:37.077 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:37.075+0000 7f2a95a30640 1 --2- 172.21.15.171:0/3395526432 >> v2:172.21.15.171:3300/0 conn(0x7f2a900b5600 0x7f2a901417b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T14:07:37.078 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:37.075+0000 7f2a96a32640 1 --2- 172.21.15.171:0/3395526432 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f2a900b5120 0x7f2a90141cf0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f2a90145610 tx=0x7f2a8801c930 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8b3d1b20848dd343 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:07:37.078 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:37.075+0000 7f2a9522f640 1 -- 172.21.15.171:0/3395526432 <== mon.2 v2:172.21.15.171:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a880b2070 con 0x7f2a900b5120 2024-03-28T14:07:37.078 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:37.075+0000 7f2a9522f640 1 -- 172.21.15.171:0/3395526432 <== mon.2 v2:172.21.15.171:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2a880a58c0 con 0x7f2a900b5120 2024-03-28T14:07:37.078 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:37.075+0000 7f2a9522f640 1 -- 172.21.15.171:0/3395526432 <== mon.2 v2:172.21.15.171:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a880a5c90 con 0x7f2a900b5120 2024-03-28T14:07:37.079 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:37.075+0000 7f2a98495640 1 -- 172.21.15.171:0/3395526432 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f2a900b5120 msgr2=0x7f2a90141cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:07:37.079 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:37.075+0000 7f2a98495640 1 --2- 172.21.15.171:0/3395526432 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f2a900b5120 0x7f2a90141cf0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f2a90145610 tx=0x7f2a8801c930 comp rx=0 tx=0).stop 2024-03-28T14:07:37.079 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:37.075+0000 7f2a98495640 1 -- 172.21.15.171:0/3395526432 shutdown_connections 2024-03-28T14:07:37.079 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:37.075+0000 7f2a98495640 1 --2- 172.21.15.171:0/3395526432 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f2a900b5120 0x7f2a90141cf0 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:07:37.080 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:37.075+0000 7f2a98495640 1 --2- 172.21.15.171:0/3395526432 >> v2:172.21.15.171:3300/0 conn(0x7f2a900b5600 0x7f2a901417b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:07:37.080 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:37.075+0000 7f2a98495640 1 --2- 172.21.15.171:0/3395526432 >> v2:172.21.15.104:3300/0 conn(0x7f2a9006dd10 0x7f2a90072cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:07:37.080 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:37.075+0000 7f2a98495640 1 -- 172.21.15.171:0/3395526432 >> 172.21.15.171:0/3395526432 conn(0x7f2a900ba390 msgr2=0x7f2a900ba7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T14:07:37.080 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:37.075+0000 7f2a98495640 1 -- 172.21.15.171:0/3395526432 shutdown_connections 2024-03-28T14:07:37.080 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:37.075+0000 7f2a98495640 1 -- 172.21.15.171:0/3395526432 wait complete. 2024-03-28T14:07:37.080 INFO:tasks.ceph.mgr.y.smithi104.stderr:2024-03-28T14:07:37.080+0000 7f57c9cb1180 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-28T14:07:37.083 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:37.079+0000 7f2a98495640 1 Processor -- start 2024-03-28T14:07:37.083 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:37.079+0000 7f2a98495640 1 -- start start 2024-03-28T14:07:37.083 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:37.079+0000 7f2a98495640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f2a9006dd10 0x7f2a900b8c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:07:37.083 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:37.079+0000 7f2a98495640 1 --2- >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f2a900b5120 0x7f2a901e04e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:07:37.083 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:37.079+0000 7f2a98495640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2a900b5600 0x7f2a9013e4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:07:37.083 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:37.079+0000 7f2a98495640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f2a9006c030 con 0x7f2a900b5600 2024-03-28T14:07:37.083 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:37.079+0000 7f2a98495640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f2a900b6340 con 0x7f2a9006dd10 2024-03-28T14:07:37.083 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:37.079+0000 7f2a98495640 1 -- --> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] -- mon_getmap magic: 0 -- 0x7f2a9006c1b0 con 0x7f2a900b5120 2024-03-28T14:07:37.083 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:37.079+0000 7f2a95a30640 1 --2- >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f2a900b5120 0x7f2a901e04e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:07:37.083 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:37.079+0000 7f2a96231640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f2a9006dd10 0x7f2a900b8c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:07:37.084 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:37.079+0000 7f2a95a30640 1 --2- >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f2a900b5120 0x7f2a901e04e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.171:3301/0 says I am v2:172.21.15.171:52558/0 (socket says 172.21.15.171:52558) 2024-03-28T14:07:37.084 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:37.079+0000 7f2a96231640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f2a9006dd10 0x7f2a900b8c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.171:3300/0 says I am v2:172.21.15.171:43064/0 (socket says 172.21.15.171:43064) 2024-03-28T14:07:37.084 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:37.079+0000 7f2a95a30640 1 -- 172.21.15.171:0/2971847497 learned_addr learned my addr 172.21.15.171:0/2971847497 (peer_addr_for_me v2:172.21.15.171:0/0) 2024-03-28T14:07:37.084 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:37.079+0000 7f2a96a32640 1 --2- 172.21.15.171:0/2971847497 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2a900b5600 0x7f2a9013e4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:07:37.084 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:37.079+0000 7f2a95a30640 1 -- 172.21.15.171:0/2971847497 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f2a9006dd10 msgr2=0x7f2a900b8c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:07:37.084 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:37.079+0000 7f2a95a30640 1 --2- 172.21.15.171:0/2971847497 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f2a9006dd10 0x7f2a900b8c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:07:37.084 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:37.079+0000 7f2a95a30640 1 -- 172.21.15.171:0/2971847497 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2a900b5600 msgr2=0x7f2a9013e4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:07:37.084 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:37.079+0000 7f2a95a30640 1 --2- 172.21.15.171:0/2971847497 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2a900b5600 0x7f2a9013e4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:07:37.084 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:37.079+0000 7f2a95a30640 1 -- 172.21.15.171:0/2971847497 --> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2a901450f0 con 0x7f2a900b5120 2024-03-28T14:07:37.084 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:37.079+0000 7f2a96231640 1 --2- 172.21.15.171:0/2971847497 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f2a9006dd10 0x7f2a900b8c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T14:07:37.084 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:37.079+0000 7f2a95a30640 1 --2- 172.21.15.171:0/2971847497 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f2a900b5120 0x7f2a901e04e0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f2a8c00e930 tx=0x7f2a8c00edf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:07:37.084 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:37.079+0000 7f2a86ffd640 1 -- 172.21.15.171:0/2971847497 <== mon.2 v2:172.21.15.171:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a8c017070 con 0x7f2a900b5120 2024-03-28T14:07:37.084 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:37.079+0000 7f2a86ffd640 1 -- 172.21.15.171:0/2971847497 <== mon.2 v2:172.21.15.171:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2a8c00ca00 con 0x7f2a900b5120 2024-03-28T14:07:37.084 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:37.079+0000 7f2a98495640 1 -- 172.21.15.171:0/2971847497 --> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2a90145390 con 0x7f2a900b5120 2024-03-28T14:07:37.085 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:37.079+0000 7f2a86ffd640 1 -- 172.21.15.171:0/2971847497 <== mon.2 v2:172.21.15.171:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a8c00cdd0 con 0x7f2a900b5120 2024-03-28T14:07:37.085 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:37.079+0000 7f2a98495640 1 -- 172.21.15.171:0/2971847497 --> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f2a9009c150 con 0x7f2a900b5120 2024-03-28T14:07:37.085 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:37.079+0000 7f2a86ffd640 1 -- 172.21.15.171:0/2971847497 <== mon.2 v2:172.21.15.171:3301/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f2a8c01a070 con 0x7f2a900b5120 2024-03-28T14:07:37.085 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:37.079+0000 7f2a86ffd640 1 -- 172.21.15.171:0/2971847497 <== mon.2 v2:172.21.15.171:3301/0 5 ==== osd_map(9..9 src has 1..9) ==== 1530+0+0 (secure 0 0 0) 0x7f2a8c013ad0 con 0x7f2a900b5120 2024-03-28T14:07:37.085 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:37.079+0000 7f2a98495640 1 -- 172.21.15.171:0/2971847497 --> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2a9009ab90 con 0x7f2a900b5120 2024-03-28T14:07:37.086 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:37.083+0000 7f2a86ffd640 1 -- 172.21.15.171:0/2971847497 <== mon.2 v2:172.21.15.171:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7f2a9009ab90 con 0x7f2a900b5120 2024-03-28T14:07:37.132 INFO:tasks.ceph.mgr.y.smithi104.stderr:2024-03-28T14:07:37.132+0000 7f57c9cb1180 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-28T14:07:37.153 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:37.151+0000 7f2a98495640 1 -- 172.21.15.171:0/2971847497 --> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] -- mon_command({"prefix": "osd new", "uuid": "339b8414-d7f0-4fb0-a446-efaf635cb7ff", "id": 7} v 0) -- 0x7f2a90067da0 con 0x7f2a900b5120 2024-03-28T14:07:37.162 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:37.159+0000 7f2a86ffd640 1 -- 172.21.15.171:0/2971847497 <== mon.2 v2:172.21.15.171:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "339b8414-d7f0-4fb0-a446-efaf635cb7ff", "id": 7}]=0 v10) ==== 112+0+2 (secure 0 0 0) 0x7f2a8c006000 con 0x7f2a900b5120 2024-03-28T14:07:37.163 INFO:teuthology.orchestra.run.smithi171.stdout:7 2024-03-28T14:07:37.164 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:37.159+0000 7f2a98495640 1 -- 172.21.15.171:0/2971847497 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f2a900b5120 msgr2=0x7f2a901e04e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:07:37.164 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:37.159+0000 7f2a98495640 1 --2- 172.21.15.171:0/2971847497 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f2a900b5120 0x7f2a901e04e0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f2a8c00e930 tx=0x7f2a8c00edf0 comp rx=0 tx=0).stop 2024-03-28T14:07:37.164 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:37.163+0000 7f2a98495640 1 -- 172.21.15.171:0/2971847497 shutdown_connections 2024-03-28T14:07:37.164 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:37.163+0000 7f2a98495640 1 --2- 172.21.15.171:0/2971847497 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2a900b5600 0x7f2a9013e4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:07:37.164 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:37.163+0000 7f2a98495640 1 --2- 172.21.15.171:0/2971847497 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f2a900b5120 0x7f2a901e04e0 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:07:37.164 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:37.163+0000 7f2a98495640 1 --2- 172.21.15.171:0/2971847497 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f2a9006dd10 0x7f2a900b8c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:07:37.164 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:37.163+0000 7f2a98495640 1 -- 172.21.15.171:0/2971847497 >> 172.21.15.171:0/2971847497 conn(0x7f2a900ba390 msgr2=0x7f2a900b4650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T14:07:37.164 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:37.163+0000 7f2a98495640 1 -- 172.21.15.171:0/2971847497 shutdown_connections 2024-03-28T14:07:37.164 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-28T14:07:37.163+0000 7f2a98495640 1 -- 172.21.15.171:0/2971847497 wait complete. 2024-03-28T14:07:37.178 INFO:tasks.ceph.osd.0:Restarting daemon 2024-03-28T14:07:37.179 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2024-03-28T14:07:37.181 INFO:tasks.ceph.osd.0:Started 2024-03-28T14:07:37.181 INFO:tasks.ceph.osd.1:Restarting daemon 2024-03-28T14:07:37.181 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2024-03-28T14:07:37.184 INFO:tasks.ceph.osd.1:Started 2024-03-28T14:07:37.184 INFO:tasks.ceph.osd.2:Restarting daemon 2024-03-28T14:07:37.184 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2024-03-28T14:07:37.186 INFO:tasks.ceph.osd.2:Started 2024-03-28T14:07:37.186 INFO:tasks.ceph.osd.3:Restarting daemon 2024-03-28T14:07:37.186 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3 2024-03-28T14:07:37.188 INFO:tasks.ceph.osd.3:Started 2024-03-28T14:07:37.188 INFO:tasks.ceph.osd.4:Restarting daemon 2024-03-28T14:07:37.189 DEBUG:teuthology.orchestra.run.smithi171:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 4 2024-03-28T14:07:37.190 INFO:tasks.ceph.osd.4:Started 2024-03-28T14:07:37.191 INFO:tasks.ceph.osd.5:Restarting daemon 2024-03-28T14:07:37.191 DEBUG:teuthology.orchestra.run.smithi171:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 5 2024-03-28T14:07:37.193 INFO:tasks.ceph.osd.5:Started 2024-03-28T14:07:37.193 INFO:tasks.ceph.osd.6:Restarting daemon 2024-03-28T14:07:37.193 DEBUG:teuthology.orchestra.run.smithi171:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 6 2024-03-28T14:07:37.195 INFO:tasks.ceph.osd.6:Started 2024-03-28T14:07:37.195 INFO:tasks.ceph.osd.7:Restarting daemon 2024-03-28T14:07:37.195 DEBUG:teuthology.orchestra.run.smithi171:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 7 2024-03-28T14:07:37.197 INFO:tasks.ceph.osd.7:Started 2024-03-28T14:07:37.198 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-28T14:07:37.310 INFO:tasks.ceph.mgr.y.smithi104.stderr:2024-03-28T14:07:37.308+0000 7f57c9cb1180 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-28T14:07:37.330 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:37.328+0000 7f65c8136640 1 Processor -- start 2024-03-28T14:07:37.330 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:37.328+0000 7f65c8136640 1 -- start start 2024-03-28T14:07:37.332 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:37.332+0000 7f65c8136640 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f65c00b4f70 0x7f65c0072ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:07:37.332 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:37.332+0000 7f65c8136640 1 --2- >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f65c00733e0 0x7f65c014ffe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:07:37.332 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:37.332+0000 7f65c8136640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f65c006bc10 con 0x7f65c006b830 2024-03-28T14:07:37.332 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:37.332+0000 7f65c8136640 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 -- 0x7f65c00a0120 con 0x7f65c00b4f70 2024-03-28T14:07:37.332 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:37.332+0000 7f65c8136640 1 -- --> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] -- mon_getmap magic: 0 -- 0x7f65c00b5a10 con 0x7f65c00733e0 2024-03-28T14:07:37.332 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:37.332+0000 7f65c56d1640 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f65c00b4f70 0x7f65c0072ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:07:37.332 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:37.332+0000 7f65c56d1640 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f65c00b4f70 0x7f65c0072ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:54770/0 (socket says 172.21.15.104:54770) 2024-03-28T14:07:37.332 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:37.332+0000 7f65c56d1640 1 -- 172.21.15.104:0/2917844697 learned_addr learned my addr 172.21.15.104:0/2917844697 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-03-28T14:07:37.332 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:37.332+0000 7f65c66d3640 1 --2- 172.21.15.104:0/2917844697 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f65c00733e0 0x7f65c014ffe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:07:37.332 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:37.332+0000 7f65c56d1640 1 -- 172.21.15.104:0/2917844697 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f65c00733e0 msgr2=0x7f65c014ffe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T14:07:37.332 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:37.332+0000 7f65c56d1640 1 --2- 172.21.15.104:0/2917844697 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f65c00733e0 0x7f65c014ffe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:07:37.332 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:37.332+0000 7f65c56d1640 1 -- 172.21.15.104:0/2917844697 >> v1:172.21.15.171:6789/0 conn(0x7f65c006b830 legacy=0x7f65c009fa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:07:37.333 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:37.332+0000 7f65c56d1640 1 -- 172.21.15.104:0/2917844697 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f65c00b5cf0 con 0x7f65c00b4f70 2024-03-28T14:07:37.333 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:37.332+0000 7f65c56d1640 1 --2- 172.21.15.104:0/2917844697 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f65c00b4f70 0x7f65c0072ea0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f65b0009870 tx=0x7f65b001c440 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ce21179c84a94caa server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:07:37.333 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:37.332+0000 7f65c4ed0640 1 -- 172.21.15.104:0/2917844697 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65b00b0070 con 0x7f65c00b4f70 2024-03-28T14:07:37.333 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:37.332+0000 7f65c4ed0640 1 -- 172.21.15.104:0/2917844697 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f65b0004230 con 0x7f65c00b4f70 2024-03-28T14:07:37.334 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:37.332+0000 7f65c4ed0640 1 -- 172.21.15.104:0/2917844697 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65b0003620 con 0x7f65c00b4f70 2024-03-28T14:07:37.334 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:37.332+0000 7f65c8136640 1 -- 172.21.15.104:0/2917844697 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f65c00b4f70 msgr2=0x7f65c0072ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:07:37.335 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:37.332+0000 7f65c8136640 1 --2- 172.21.15.104:0/2917844697 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f65c00b4f70 0x7f65c0072ea0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f65b0009870 tx=0x7f65b001c440 comp rx=0 tx=0).stop 2024-03-28T14:07:37.335 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:37.332+0000 7f65c8136640 1 -- 172.21.15.104:0/2917844697 shutdown_connections 2024-03-28T14:07:37.335 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:37.332+0000 7f65c8136640 1 --2- 172.21.15.104:0/2917844697 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f65c00733e0 0x7f65c014ffe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:07:37.335 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:37.332+0000 7f65c8136640 1 --2- 172.21.15.104:0/2917844697 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f65c00b4f70 0x7f65c0072ea0 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:07:37.335 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:37.332+0000 7f65c8136640 1 -- 172.21.15.104:0/2917844697 >> 172.21.15.104:0/2917844697 conn(0x7f65c00ba400 msgr2=0x7f65c00ba810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T14:07:37.335 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:37.336+0000 7f65c8136640 1 -- 172.21.15.104:0/2917844697 shutdown_connections 2024-03-28T14:07:37.335 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:37.336+0000 7f65c8136640 1 -- 172.21.15.104:0/2917844697 wait complete. 2024-03-28T14:07:37.338 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:37.336+0000 7f65c8136640 1 Processor -- start 2024-03-28T14:07:37.338 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:37.336+0000 7f65c8136640 1 -- start start 2024-03-28T14:07:37.338 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:37.336+0000 7f65c8136640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f65c006b830 0x7f65c00af2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:07:37.338 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:37.336+0000 7f65c8136640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f65c00733e0 0x7f65c0072630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:07:37.339 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:37.336+0000 7f65c8136640 1 --2- >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f65c00b4f70 0x7f65c01e3970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:07:37.339 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:37.336+0000 7f65c8136640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f65c0152ec0 con 0x7f65c006b830 2024-03-28T14:07:37.339 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:37.336+0000 7f65c8136640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f65c0152d40 con 0x7f65c00733e0 2024-03-28T14:07:37.339 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:37.336+0000 7f65c8136640 1 -- --> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] -- mon_getmap magic: 0 -- 0x7f65c0153040 con 0x7f65c00b4f70 2024-03-28T14:07:37.339 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:37.336+0000 7f65c5ed2640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f65c006b830 0x7f65c00af2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:07:37.339 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:37.336+0000 7f65c5ed2640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f65c006b830 0x7f65c00af2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:54780/0 (socket says 172.21.15.104:54780) 2024-03-28T14:07:37.339 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:37.336+0000 7f65c5ed2640 1 -- 172.21.15.104:0/2940619835 learned_addr learned my addr 172.21.15.104:0/2940619835 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-03-28T14:07:37.339 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:37.336+0000 7f65c56d1640 1 --2- 172.21.15.104:0/2940619835 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f65c00733e0 0x7f65c0072630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:07:37.339 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:37.336+0000 7f65c66d3640 1 --2- 172.21.15.104:0/2940619835 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f65c00b4f70 0x7f65c01e3970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:07:37.339 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:37.336+0000 7f65c5ed2640 1 -- 172.21.15.104:0/2940619835 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f65c00b4f70 msgr2=0x7f65c01e3970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:07:37.339 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:37.336+0000 7f65c5ed2640 1 --2- 172.21.15.104:0/2940619835 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f65c00b4f70 0x7f65c01e3970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:07:37.339 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:37.336+0000 7f65c5ed2640 1 -- 172.21.15.104:0/2940619835 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f65c00733e0 msgr2=0x7f65c0072630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:07:37.339 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:37.336+0000 7f65c5ed2640 1 --2- 172.21.15.104:0/2940619835 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f65c00733e0 0x7f65c0072630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:07:37.339 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:37.336+0000 7f65c5ed2640 1 -- 172.21.15.104:0/2940619835 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f65c013a230 con 0x7f65c006b830 2024-03-28T14:07:37.340 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:37.336+0000 7f65c66d3640 1 --2- 172.21.15.104:0/2940619835 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f65c00b4f70 0x7f65c01e3970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T14:07:37.340 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:37.336+0000 7f65c5ed2640 1 --2- 172.21.15.104:0/2940619835 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f65c006b830 0x7f65c00af2d0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f65bc00c420 tx=0x7f65bc00c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:07:37.340 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:37.336+0000 7f65b67fc640 1 -- 172.21.15.104:0/2940619835 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65bc0052d0 con 0x7f65c006b830 2024-03-28T14:07:37.340 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:37.336+0000 7f65b67fc640 1 -- 172.21.15.104:0/2940619835 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f65bc005470 con 0x7f65c006b830 2024-03-28T14:07:37.340 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:37.336+0000 7f65c8136640 1 -- 172.21.15.104:0/2940619835 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f65c00b05d0 con 0x7f65c006b830 2024-03-28T14:07:37.340 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:37.336+0000 7f65b67fc640 1 -- 172.21.15.104:0/2940619835 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65bc007260 con 0x7f65c006b830 2024-03-28T14:07:37.340 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:37.336+0000 7f65c8136640 1 -- 172.21.15.104:0/2940619835 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f65c00af020 con 0x7f65c006b830 2024-03-28T14:07:37.340 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:37.336+0000 7f65b67fc640 1 -- 172.21.15.104:0/2940619835 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f65bc007400 con 0x7f65c006b830 2024-03-28T14:07:37.340 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:37.336+0000 7f65b67fc640 1 -- 172.21.15.104:0/2940619835 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(10..10 src has 1..10) ==== 1645+0+0 (secure 0 0 0) 0x7f65bc011c60 con 0x7f65c006b830 2024-03-28T14:07:37.340 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:37.336+0000 7f65c8136640 1 -- 172.21.15.104:0/2940619835 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f65c00687c0 con 0x7f65c006b830 2024-03-28T14:07:37.356 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:37.356+0000 7f65b67fc640 1 -- 172.21.15.104:0/2940619835 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7f65bc0039c0 con 0x7f65c006b830 2024-03-28T14:07:37.374 INFO:tasks.ceph.mgr.y.smithi104.stderr:2024-03-28T14:07:37.372+0000 7f57c9cb1180 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-28T14:07:37.414 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:37.412+0000 7f65c8136640 1 -- 172.21.15.104:0/2940619835 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f65c00b7950 con 0x7f65c006b830 2024-03-28T14:07:37.414 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:37.412+0000 7f65b67fc640 1 -- 172.21.15.104:0/2940619835 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) ==== 74+0+7058 (secure 0 0 0) 0x7f65bc003ba0 con 0x7f65c006b830 2024-03-28T14:07:37.414 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-03-28T14:07:37.415 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":10,"fsid":"428a90f2-3351-435f-8fcf-f7e66c78a13e","created":"2024-03-28T14:07:34.862458+0000","modified":"2024-03-28T14:07:37.157486+0000","last_up_change":"0.000000","last_in_change":"2024-03-28T14:07:37.157486+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"1cb70540-4e42-402f-a446-aaec4ca83007","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"3ab7afdd-b980-418c-81f2-004e16de3dca","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"3e2651c8-ecae-4a9e-a053-27240a79eac2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"925c7e0a-38ef-47cd-b288-8bec8456fb17","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"4a8fac86-e595-4a0e-96e3-57edb3cc3f57","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"e48d50b0-beae-40ae-b15a-9a84dbfe88ab","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"5e7c3292-9936-4f35-b267-aed9d4e49323","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"339b8414-d7f0-4fb0-a446-efaf635cb7ff","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-28T14:07:37.416 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:37.416+0000 7f65c8136640 1 -- 172.21.15.104:0/2940619835 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f65c006b830 msgr2=0x7f65c00af2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:07:37.416 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:37.416+0000 7f65c8136640 1 --2- 172.21.15.104:0/2940619835 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f65c006b830 0x7f65c00af2d0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f65bc00c420 tx=0x7f65bc00c8e0 comp rx=0 tx=0).stop 2024-03-28T14:07:37.416 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:37.416+0000 7f65c8136640 1 -- 172.21.15.104:0/2940619835 shutdown_connections 2024-03-28T14:07:37.416 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:37.416+0000 7f65c8136640 1 --2- 172.21.15.104:0/2940619835 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f65c00b4f70 0x7f65c01e3970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:07:37.416 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:37.416+0000 7f65c8136640 1 --2- 172.21.15.104:0/2940619835 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f65c00733e0 0x7f65c0072630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:07:37.416 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:37.416+0000 7f65c8136640 1 --2- 172.21.15.104:0/2940619835 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f65c006b830 0x7f65c00af2d0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:07:37.416 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:37.416+0000 7f65c8136640 1 -- 172.21.15.104:0/2940619835 >> 172.21.15.104:0/2940619835 conn(0x7f65c00ba400 msgr2=0x7f65c013bdb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T14:07:37.416 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:37.416+0000 7f65c8136640 1 -- 172.21.15.104:0/2940619835 shutdown_connections 2024-03-28T14:07:37.416 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:37.416+0000 7f65c8136640 1 -- 172.21.15.104:0/2940619835 wait complete. 2024-03-28T14:07:37.432 INFO:tasks.ceph.ceph_manager.ceph:[] 2024-03-28T14:07:37.432 INFO:tasks.ceph:Waiting for OSDs to come up 2024-03-28T14:07:37.610 INFO:tasks.ceph.mgr.x.smithi171.stderr:/usr/lib/python3/dist-packages/scipy/__init__.py:67: UserWarning: NumPy was imported from a Python sub-interpreter but NumPy does not properly support sub-interpreters. This will likely work for most users but might cause hard to track down issues or subtle bugs. A common user of the rare sub-interpreter feature is wsgi which also allows single-interpreter mode. 2024-03-28T14:07:37.610 INFO:tasks.ceph.mgr.x.smithi171.stderr:Improvements in the case of bugs are welcome, but is not on the NumPy roadmap, and full support may require significant effort to achieve. 2024-03-28T14:07:37.610 INFO:tasks.ceph.mgr.x.smithi171.stderr: from numpy import show_config as show_numpy_config 2024-03-28T14:07:37.698 INFO:tasks.ceph.mgr.x.smithi171.stderr:2024-03-28T14:07:37.695+0000 7fa461f1b180 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-28T14:07:37.801 INFO:tasks.ceph.osd.4.smithi171.stderr:2024-03-28T14:07:37.799+0000 7f7a1ccaf7c0 -1 Falling back to public interface 2024-03-28T14:07:37.808 INFO:tasks.ceph.osd.5.smithi171.stderr:2024-03-28T14:07:37.807+0000 7f0113c387c0 -1 Falling back to public interface 2024-03-28T14:07:37.809 INFO:tasks.ceph.osd.7.smithi171.stderr:2024-03-28T14:07:37.807+0000 7ff6d53517c0 -1 Falling back to public interface 2024-03-28T14:07:37.823 INFO:tasks.ceph.osd.0.smithi104.stderr:2024-03-28T14:07:37.824+0000 7f387fe0a7c0 -1 Falling back to public interface 2024-03-28T14:07:37.828 INFO:tasks.ceph.osd.6.smithi171.stderr:2024-03-28T14:07:37.827+0000 7f380f7087c0 -1 Falling back to public interface 2024-03-28T14:07:37.831 INFO:tasks.ceph.osd.1.smithi104.stderr:2024-03-28T14:07:37.832+0000 7f5ccf2567c0 -1 Falling back to public interface 2024-03-28T14:07:37.831 INFO:tasks.ceph.osd.3.smithi104.stderr:2024-03-28T14:07:37.832+0000 7f74834017c0 -1 Falling back to public interface 2024-03-28T14:07:37.833 INFO:tasks.ceph.osd.1.smithi104.stderr:2024-03-28T14:07:37.832+0000 7f5cce216640 -1 PosixStack listen unable to listen on v1:0.0.0.0:6818/0: (98) Address already in use 2024-03-28T14:07:37.855 INFO:tasks.ceph.osd.2.smithi104.stderr:2024-03-28T14:07:37.856+0000 7fce600047c0 -1 Falling back to public interface 2024-03-28T14:07:37.882 INFO:tasks.ceph.mgr.x.smithi171.stderr:2024-03-28T14:07:37.883+0000 7fa461f1b180 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-28T14:07:38.008 INFO:tasks.ceph.mgr.y.smithi104.stderr:/usr/lib/python3/dist-packages/scipy/__init__.py:67: UserWarning: NumPy was imported from a Python sub-interpreter but NumPy does not properly support sub-interpreters. This will likely work for most users but might cause hard to track down issues or subtle bugs. A common user of the rare sub-interpreter feature is wsgi which also allows single-interpreter mode. 2024-03-28T14:07:38.008 INFO:tasks.ceph.mgr.y.smithi104.stderr:Improvements in the case of bugs are welcome, but is not on the NumPy roadmap, and full support may require significant effort to achieve. 2024-03-28T14:07:38.008 INFO:tasks.ceph.mgr.y.smithi104.stderr: from numpy import show_config as show_numpy_config 2024-03-28T14:07:38.049 INFO:tasks.ceph.mgr.x.smithi171.stderr:2024-03-28T14:07:38.050+0000 7fa461f1b180 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-28T14:07:38.087 INFO:tasks.ceph.mgr.y.smithi104.stderr:2024-03-28T14:07:38.088+0000 7f57c9cb1180 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-28T14:07:38.111 INFO:tasks.ceph.mgr.x.smithi171.stderr:2024-03-28T14:07:38.110+0000 7fa461f1b180 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-28T14:07:38.239 DEBUG:teuthology.orchestra.run.smithi104:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-28T14:07:38.245 INFO:tasks.ceph.mgr.x.smithi171.stderr:2024-03-28T14:07:38.246+0000 7fa461f1b180 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-28T14:07:38.263 INFO:tasks.ceph.mgr.y.smithi104.stderr:2024-03-28T14:07:38.264+0000 7f57c9cb1180 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-28T14:07:38.350 INFO:teuthology.misc.health.smithi104.stderr:2024-03-28T14:07:38.348+0000 7f959ddc6640 1 Processor -- start 2024-03-28T14:07:38.351 INFO:teuthology.misc.health.smithi104.stderr:2024-03-28T14:07:38.348+0000 7f959ddc6640 1 -- start start 2024-03-28T14:07:38.352 INFO:teuthology.misc.health.smithi104.stderr:2024-03-28T14:07:38.352+0000 7f959ddc6640 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f959806d960 0x7f95981416e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:07:38.352 INFO:teuthology.misc.health.smithi104.stderr:2024-03-28T14:07:38.352+0000 7f959ddc6640 1 --2- >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f95980b5920 0x7f9598141c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:07:38.352 INFO:teuthology.misc.health.smithi104.stderr:2024-03-28T14:07:38.352+0000 7f959ddc6640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f95980bbc70 con 0x7f95980bb890 2024-03-28T14:07:38.352 INFO:teuthology.misc.health.smithi104.stderr:2024-03-28T14:07:38.352+0000 7f959ddc6640 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 -- 0x7f95980b5e40 con 0x7f959806d960 2024-03-28T14:07:38.352 INFO:teuthology.misc.health.smithi104.stderr:2024-03-28T14:07:38.352+0000 7f959ddc6640 1 -- --> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] -- mon_getmap magic: 0 -- 0x7f95980b7120 con 0x7f95980b5920 2024-03-28T14:07:38.352 INFO:teuthology.misc.health.smithi104.stderr:2024-03-28T14:07:38.352+0000 7f9596ffd640 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f959806d960 0x7f95981416e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:07:38.352 INFO:teuthology.misc.health.smithi104.stderr:2024-03-28T14:07:38.352+0000 7f9596ffd640 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f959806d960 0x7f95981416e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:54782/0 (socket says 172.21.15.104:54782) 2024-03-28T14:07:38.352 INFO:teuthology.misc.health.smithi104.stderr:2024-03-28T14:07:38.352+0000 7f9596ffd640 1 -- 172.21.15.104:0/1770199867 learned_addr learned my addr 172.21.15.104:0/1770199867 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-03-28T14:07:38.353 INFO:teuthology.misc.health.smithi104.stderr:2024-03-28T14:07:38.352+0000 7f9597fff640 1 --2- 172.21.15.104:0/1770199867 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f95980b5920 0x7f9598141c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:07:38.353 INFO:teuthology.misc.health.smithi104.stderr:2024-03-28T14:07:38.352+0000 7f9596ffd640 1 -- 172.21.15.104:0/1770199867 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f95980b5920 msgr2=0x7f9598141c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T14:07:38.353 INFO:teuthology.misc.health.smithi104.stderr:2024-03-28T14:07:38.352+0000 7f9596ffd640 1 --2- 172.21.15.104:0/1770199867 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f95980b5920 0x7f9598141c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:07:38.353 INFO:teuthology.misc.health.smithi104.stderr:2024-03-28T14:07:38.352+0000 7f9596ffd640 1 -- 172.21.15.104:0/1770199867 >> v1:172.21.15.171:6789/0 conn(0x7f95980bb890 legacy=0x7f95980b1b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T14:07:38.353 INFO:teuthology.misc.health.smithi104.stderr:2024-03-28T14:07:38.352+0000 7f9596ffd640 1 -- 172.21.15.104:0/1770199867 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f95980b7400 con 0x7f959806d960 2024-03-28T14:07:38.353 INFO:teuthology.misc.health.smithi104.stderr:2024-03-28T14:07:38.352+0000 7f9597fff640 1 --2- 172.21.15.104:0/1770199867 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f95980b5920 0x7f9598141c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T14:07:38.354 INFO:teuthology.misc.health.smithi104.stderr:2024-03-28T14:07:38.352+0000 7f9596ffd640 1 --2- 172.21.15.104:0/1770199867 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f959806d960 0x7f95981416e0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f958c009870 tx=0x7f958c01c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e859e47aa9ea7b61 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:07:38.354 INFO:teuthology.misc.health.smithi104.stderr:2024-03-28T14:07:38.352+0000 7f95967fc640 1 -- 172.21.15.104:0/1770199867 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f958c0ab5f0 con 0x7f959806d960 2024-03-28T14:07:38.354 INFO:teuthology.misc.health.smithi104.stderr:2024-03-28T14:07:38.352+0000 7f95967fc640 1 -- 172.21.15.104:0/1770199867 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f958c0b0070 con 0x7f959806d960 2024-03-28T14:07:38.354 INFO:teuthology.misc.health.smithi104.stderr:2024-03-28T14:07:38.352+0000 7f95967fc640 1 -- 172.21.15.104:0/1770199867 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f958c0ab9b0 con 0x7f959806d960 2024-03-28T14:07:38.355 INFO:teuthology.misc.health.smithi104.stderr:2024-03-28T14:07:38.356+0000 7f959ddc6640 1 -- 172.21.15.104:0/1770199867 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f959806d960 msgr2=0x7f95981416e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:07:38.355 INFO:teuthology.misc.health.smithi104.stderr:2024-03-28T14:07:38.356+0000 7f959ddc6640 1 --2- 172.21.15.104:0/1770199867 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f959806d960 0x7f95981416e0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f958c009870 tx=0x7f958c01c520 comp rx=0 tx=0).stop 2024-03-28T14:07:38.355 INFO:teuthology.misc.health.smithi104.stderr:2024-03-28T14:07:38.356+0000 7f959ddc6640 1 -- 172.21.15.104:0/1770199867 shutdown_connections 2024-03-28T14:07:38.355 INFO:teuthology.misc.health.smithi104.stderr:2024-03-28T14:07:38.356+0000 7f959ddc6640 1 --2- 172.21.15.104:0/1770199867 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f95980b5920 0x7f9598141c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:07:38.356 INFO:teuthology.misc.health.smithi104.stderr:2024-03-28T14:07:38.356+0000 7f959ddc6640 1 --2- 172.21.15.104:0/1770199867 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f959806d960 0x7f95981416e0 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:07:38.356 INFO:teuthology.misc.health.smithi104.stderr:2024-03-28T14:07:38.356+0000 7f959ddc6640 1 -- 172.21.15.104:0/1770199867 >> 172.21.15.104:0/1770199867 conn(0x7f95980b8080 msgr2=0x7f95980baf20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T14:07:38.356 INFO:teuthology.misc.health.smithi104.stderr:2024-03-28T14:07:38.356+0000 7f959ddc6640 1 -- 172.21.15.104:0/1770199867 shutdown_connections 2024-03-28T14:07:38.356 INFO:teuthology.misc.health.smithi104.stderr:2024-03-28T14:07:38.356+0000 7f959ddc6640 1 -- 172.21.15.104:0/1770199867 wait complete. 2024-03-28T14:07:38.356 INFO:teuthology.misc.health.smithi104.stderr:2024-03-28T14:07:38.356+0000 7f959ddc6640 1 Processor -- start 2024-03-28T14:07:38.356 INFO:teuthology.misc.health.smithi104.stderr:2024-03-28T14:07:38.356+0000 7f959ddc6640 1 -- start start 2024-03-28T14:07:38.359 INFO:teuthology.misc.health.smithi104.stderr:2024-03-28T14:07:38.356+0000 7f959ddc6640 1 --2- >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f959806d960 0x7f95981442f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:07:38.359 INFO:teuthology.misc.health.smithi104.stderr:2024-03-28T14:07:38.356+0000 7f959ddc6640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f95980b5920 0x7f95981e08a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:07:38.359 INFO:teuthology.misc.health.smithi104.stderr:2024-03-28T14:07:38.356+0000 7f959ddc6640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f95980bb890 0x7f95981e0de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:07:38.359 INFO:teuthology.misc.health.smithi104.stderr:2024-03-28T14:07:38.356+0000 7f959ddc6640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f9598154da0 con 0x7f95980bb890 2024-03-28T14:07:38.359 INFO:teuthology.misc.health.smithi104.stderr:2024-03-28T14:07:38.356+0000 7f959ddc6640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f9598154c20 con 0x7f95980b5920 2024-03-28T14:07:38.360 INFO:teuthology.misc.health.smithi104.stderr:2024-03-28T14:07:38.356+0000 7f959ddc6640 1 -- --> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] -- mon_getmap magic: 0 -- 0x7f9598154f20 con 0x7f959806d960 2024-03-28T14:07:38.360 INFO:teuthology.misc.health.smithi104.stderr:2024-03-28T14:07:38.356+0000 7f9597fff640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f95980bb890 0x7f95981e0de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:07:38.360 INFO:teuthology.misc.health.smithi104.stderr:2024-03-28T14:07:38.356+0000 7f9597fff640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f95980bb890 0x7f95981e0de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:54792/0 (socket says 172.21.15.104:54792) 2024-03-28T14:07:38.360 INFO:teuthology.misc.health.smithi104.stderr:2024-03-28T14:07:38.356+0000 7f9597fff640 1 -- 172.21.15.104:0/2787620071 learned_addr learned my addr 172.21.15.104:0/2787620071 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-03-28T14:07:38.360 INFO:teuthology.misc.health.smithi104.stderr:2024-03-28T14:07:38.356+0000 7f95977fe640 1 --2- 172.21.15.104:0/2787620071 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f959806d960 0x7f95981442f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:07:38.360 INFO:teuthology.misc.health.smithi104.stderr:2024-03-28T14:07:38.356+0000 7f9596ffd640 1 --2- 172.21.15.104:0/2787620071 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f95980b5920 0x7f95981e08a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:07:38.360 INFO:teuthology.misc.health.smithi104.stderr:2024-03-28T14:07:38.356+0000 7f9597fff640 1 -- 172.21.15.104:0/2787620071 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f959806d960 msgr2=0x7f95981442f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:07:38.360 INFO:teuthology.misc.health.smithi104.stderr:2024-03-28T14:07:38.356+0000 7f9597fff640 1 --2- 172.21.15.104:0/2787620071 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f959806d960 0x7f95981442f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:07:38.360 INFO:teuthology.misc.health.smithi104.stderr:2024-03-28T14:07:38.356+0000 7f9597fff640 1 -- 172.21.15.104:0/2787620071 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f95980b5920 msgr2=0x7f95981e08a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:07:38.360 INFO:teuthology.misc.health.smithi104.stderr:2024-03-28T14:07:38.356+0000 7f9597fff640 1 --2- 172.21.15.104:0/2787620071 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f95980b5920 0x7f95981e08a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:07:38.360 INFO:teuthology.misc.health.smithi104.stderr:2024-03-28T14:07:38.356+0000 7f9597fff640 1 -- 172.21.15.104:0/2787620071 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9598142dd0 con 0x7f95980bb890 2024-03-28T14:07:38.360 INFO:teuthology.misc.health.smithi104.stderr:2024-03-28T14:07:38.356+0000 7f9597fff640 1 --2- 172.21.15.104:0/2787620071 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f95980bb890 0x7f95981e0de0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f959000ced0 tx=0x7f959000a710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:07:38.360 INFO:teuthology.misc.health.smithi104.stderr:2024-03-28T14:07:38.356+0000 7f957ffff640 1 -- 172.21.15.104:0/2787620071 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9590003ba0 con 0x7f95980bb890 2024-03-28T14:07:38.360 INFO:teuthology.misc.health.smithi104.stderr:2024-03-28T14:07:38.356+0000 7f957ffff640 1 -- 172.21.15.104:0/2787620071 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9590003d60 con 0x7f95980bb890 2024-03-28T14:07:38.361 INFO:teuthology.misc.health.smithi104.stderr:2024-03-28T14:07:38.356+0000 7f959ddc6640 1 -- 172.21.15.104:0/2787620071 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f95980add90 con 0x7f95980bb890 2024-03-28T14:07:38.361 INFO:teuthology.misc.health.smithi104.stderr:2024-03-28T14:07:38.356+0000 7f957ffff640 1 -- 172.21.15.104:0/2787620071 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f959000d880 con 0x7f95980bb890 2024-03-28T14:07:38.361 INFO:teuthology.misc.health.smithi104.stderr:2024-03-28T14:07:38.356+0000 7f959ddc6640 1 -- 172.21.15.104:0/2787620071 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f95980adab0 con 0x7f95980bb890 2024-03-28T14:07:38.361 INFO:teuthology.misc.health.smithi104.stderr:2024-03-28T14:07:38.360+0000 7f957ffff640 1 -- 172.21.15.104:0/2787620071 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f959000da20 con 0x7f95980bb890 2024-03-28T14:07:38.361 INFO:teuthology.misc.health.smithi104.stderr:2024-03-28T14:07:38.360+0000 7f957ffff640 1 -- 172.21.15.104:0/2787620071 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(10..10 src has 1..10) ==== 1645+0+0 (secure 0 0 0) 0x7f959000dd10 con 0x7f95980bb890 2024-03-28T14:07:38.361 INFO:teuthology.misc.health.smithi104.stderr:2024-03-28T14:07:38.360+0000 7f959ddc6640 1 -- 172.21.15.104:0/2787620071 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9598068f70 con 0x7f95980bb890 2024-03-28T14:07:38.361 INFO:teuthology.misc.health.smithi104.stderr:2024-03-28T14:07:38.360+0000 7f957ffff640 1 -- 172.21.15.104:0/2787620071 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7f9590003f10 con 0x7f95980bb890 2024-03-28T14:07:38.415 INFO:tasks.ceph.mgr.y.smithi104.stderr:2024-03-28T14:07:38.416+0000 7f57c9cb1180 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-28T14:07:38.418 INFO:teuthology.misc.health.smithi104.stderr:2024-03-28T14:07:38.420+0000 7f959ddc6640 1 -- 172.21.15.104:0/2787620071 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f9598140a30 con 0x7f95980bb890 2024-03-28T14:07:38.419 INFO:teuthology.misc.health.smithi104.stderr:2024-03-28T14:07:38.420+0000 7f957ffff640 1 -- 172.21.15.104:0/2787620071 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) ==== 74+0+7058 (secure 0 0 0) 0x7f95900040f0 con 0x7f95980bb890 2024-03-28T14:07:38.419 INFO:teuthology.misc.health.smithi104.stdout: 2024-03-28T14:07:38.419 INFO:teuthology.misc.health.smithi104.stdout:{"epoch":10,"fsid":"428a90f2-3351-435f-8fcf-f7e66c78a13e","created":"2024-03-28T14:07:34.862458+0000","modified":"2024-03-28T14:07:37.157486+0000","last_up_change":"0.000000","last_in_change":"2024-03-28T14:07:37.157486+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"1cb70540-4e42-402f-a446-aaec4ca83007","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"3ab7afdd-b980-418c-81f2-004e16de3dca","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"3e2651c8-ecae-4a9e-a053-27240a79eac2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"925c7e0a-38ef-47cd-b288-8bec8456fb17","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"4a8fac86-e595-4a0e-96e3-57edb3cc3f57","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"e48d50b0-beae-40ae-b15a-9a84dbfe88ab","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"5e7c3292-9936-4f35-b267-aed9d4e49323","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"339b8414-d7f0-4fb0-a446-efaf635cb7ff","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-28T14:07:38.421 INFO:teuthology.misc.health.smithi104.stderr:2024-03-28T14:07:38.420+0000 7f959ddc6640 1 -- 172.21.15.104:0/2787620071 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f95980bb890 msgr2=0x7f95981e0de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:07:38.421 INFO:teuthology.misc.health.smithi104.stderr:2024-03-28T14:07:38.420+0000 7f959ddc6640 1 --2- 172.21.15.104:0/2787620071 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f95980bb890 0x7f95981e0de0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f959000ced0 tx=0x7f959000a710 comp rx=0 tx=0).stop 2024-03-28T14:07:38.421 INFO:teuthology.misc.health.smithi104.stderr:2024-03-28T14:07:38.420+0000 7f959ddc6640 1 -- 172.21.15.104:0/2787620071 shutdown_connections 2024-03-28T14:07:38.421 INFO:teuthology.misc.health.smithi104.stderr:2024-03-28T14:07:38.420+0000 7f959ddc6640 1 --2- 172.21.15.104:0/2787620071 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f95980bb890 0x7f95981e0de0 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:07:38.421 INFO:teuthology.misc.health.smithi104.stderr:2024-03-28T14:07:38.420+0000 7f959ddc6640 1 --2- 172.21.15.104:0/2787620071 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f95980b5920 0x7f95981e08a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:07:38.421 INFO:teuthology.misc.health.smithi104.stderr:2024-03-28T14:07:38.420+0000 7f959ddc6640 1 --2- 172.21.15.104:0/2787620071 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f959806d960 0x7f95981442f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:07:38.421 INFO:teuthology.misc.health.smithi104.stderr:2024-03-28T14:07:38.420+0000 7f959ddc6640 1 -- 172.21.15.104:0/2787620071 >> 172.21.15.104:0/2787620071 conn(0x7f95980b8080 msgr2=0x7f95980b9dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T14:07:38.421 INFO:teuthology.misc.health.smithi104.stderr:2024-03-28T14:07:38.420+0000 7f959ddc6640 1 -- 172.21.15.104:0/2787620071 shutdown_connections 2024-03-28T14:07:38.421 INFO:teuthology.misc.health.smithi104.stderr:2024-03-28T14:07:38.420+0000 7f959ddc6640 1 -- 172.21.15.104:0/2787620071 wait complete. 2024-03-28T14:07:38.435 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-28T14:07:38.480 INFO:tasks.ceph.mgr.y.smithi104.stderr:2024-03-28T14:07:38.480+0000 7f57c9cb1180 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-28T14:07:38.613 INFO:tasks.ceph.mgr.y.smithi104.stderr:2024-03-28T14:07:38.612+0000 7f57c9cb1180 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-28T14:07:38.861 INFO:tasks.ceph.mgr.x.smithi171.stderr:2024-03-28T14:07:38.862+0000 7fa461f1b180 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-28T14:07:38.937 INFO:tasks.ceph.mgr.x.smithi171.stderr:2024-03-28T14:07:38.938+0000 7fa461f1b180 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-28T14:07:39.117 INFO:tasks.ceph.mgr.x.smithi171.stderr:2024-03-28T14:07:39.118+0000 7fa461f1b180 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-28T14:07:39.190 INFO:tasks.ceph.mgr.x.smithi171.stderr:2024-03-28T14:07:39.190+0000 7fa461f1b180 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-28T14:07:39.221 INFO:tasks.ceph.mgr.y.smithi104.stderr:2024-03-28T14:07:39.220+0000 7f57c9cb1180 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-28T14:07:39.295 INFO:tasks.ceph.mgr.y.smithi104.stderr:2024-03-28T14:07:39.296+0000 7f57c9cb1180 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-28T14:07:39.475 INFO:tasks.ceph.mgr.y.smithi104.stderr:2024-03-28T14:07:39.476+0000 7f57c9cb1180 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-28T14:07:39.477 INFO:tasks.ceph.mgr.x.smithi171.stderr:2024-03-28T14:07:39.478+0000 7fa461f1b180 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-28T14:07:39.538 INFO:tasks.ceph.mgr.x.smithi171.stderr:2024-03-28T14:07:39.538+0000 7fa461f1b180 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-28T14:07:39.547 INFO:tasks.ceph.mgr.y.smithi104.stderr:2024-03-28T14:07:39.548+0000 7f57c9cb1180 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-28T14:07:39.842 INFO:tasks.ceph.mgr.y.smithi104.stderr:2024-03-28T14:07:39.840+0000 7f57c9cb1180 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-28T14:07:39.903 INFO:tasks.ceph.mgr.y.smithi104.stderr:2024-03-28T14:07:39.904+0000 7f57c9cb1180 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-28T14:07:40.579 INFO:tasks.ceph.mgr.x.smithi171.stderr:2024-03-28T14:07:40.578+0000 7fa409261640 -1 mgr.server handle_report got status from non-daemon mon.b 2024-03-28T14:07:40.580 INFO:tasks.ceph.mgr.x.smithi171.stderr:2024-03-28T14:07:40.582+0000 7fa409261640 -1 mgr.server handle_report got status from non-daemon mon.c 2024-03-28T14:07:41.622 INFO:tasks.ceph.osd.7.smithi171.stderr:2024-03-28T14:07:41.622+0000 7ff6d53517c0 -1 osd.7 0 log_to_monitors true 2024-03-28T14:07:41.641 INFO:tasks.ceph.osd.4.smithi171.stderr:2024-03-28T14:07:41.642+0000 7f7a1ccaf7c0 -1 osd.4 0 log_to_monitors true 2024-03-28T14:07:41.668 INFO:tasks.ceph.osd.2.smithi104.stderr:2024-03-28T14:07:41.668+0000 7fce600047c0 -1 osd.2 0 log_to_monitors true 2024-03-28T14:07:41.685 INFO:tasks.ceph.osd.1.smithi104.stderr:2024-03-28T14:07:41.684+0000 7f5ccf2567c0 -1 osd.1 0 log_to_monitors true 2024-03-28T14:07:41.700 INFO:tasks.ceph.osd.0.smithi104.stderr:2024-03-28T14:07:41.700+0000 7f387fe0a7c0 -1 osd.0 0 log_to_monitors true 2024-03-28T14:07:41.722 INFO:tasks.ceph.osd.6.smithi171.stderr:2024-03-28T14:07:41.722+0000 7f380f7087c0 -1 osd.6 0 log_to_monitors true 2024-03-28T14:07:41.729 INFO:tasks.ceph.osd.3.smithi104.stderr:2024-03-28T14:07:41.728+0000 7f74834017c0 -1 osd.3 0 log_to_monitors true 2024-03-28T14:07:41.733 INFO:tasks.ceph.osd.5.smithi171.stderr:2024-03-28T14:07:41.734+0000 7f0113c387c0 -1 osd.5 0 log_to_monitors true 2024-03-28T14:07:44.979 INFO:tasks.ceph.osd.6.smithi171.stderr:2024-03-28T14:07:44.978+0000 7f380bec3640 -1 osd.6 0 waiting for initial osdmap 2024-03-28T14:07:45.011 INFO:tasks.ceph.osd.0.smithi104.stderr:2024-03-28T14:07:45.012+0000 7f387bdb3640 -1 osd.0 0 waiting for initial osdmap 2024-03-28T14:07:45.016 INFO:tasks.ceph.osd.6.smithi171.stderr:2024-03-28T14:07:45.014+0000 7f38034e5640 -1 osd.6 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-28T14:07:45.023 INFO:tasks.ceph.osd.3.smithi104.stderr:2024-03-28T14:07:45.024+0000 7f747f3aa640 -1 osd.3 0 waiting for initial osdmap 2024-03-28T14:07:45.039 INFO:tasks.ceph.osd.0.smithi104.stderr:2024-03-28T14:07:45.040+0000 7f3873be7640 -1 osd.0 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-28T14:07:45.054 INFO:tasks.ceph.osd.7.smithi171.stderr:2024-03-28T14:07:45.054+0000 7ff6d12fa640 -1 osd.7 0 waiting for initial osdmap 2024-03-28T14:07:45.063 INFO:tasks.ceph.osd.3.smithi104.stderr:2024-03-28T14:07:45.064+0000 7f74771de640 -1 osd.3 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-28T14:07:45.067 INFO:tasks.ceph.osd.2.smithi104.stderr:2024-03-28T14:07:45.068+0000 7fce5bfad640 -1 osd.2 0 waiting for initial osdmap 2024-03-28T14:07:45.076 INFO:tasks.ceph.osd.7.smithi171.stderr:2024-03-28T14:07:45.078+0000 7ff6c912e640 -1 osd.7 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-28T14:07:45.080 INFO:tasks.ceph.osd.5.smithi171.stderr:2024-03-28T14:07:45.082+0000 7f010fbe1640 -1 osd.5 0 waiting for initial osdmap 2024-03-28T14:07:45.091 INFO:tasks.ceph.osd.4.smithi171.stderr:2024-03-28T14:07:45.090+0000 7f7a18c58640 -1 osd.4 0 waiting for initial osdmap 2024-03-28T14:07:45.091 INFO:tasks.ceph.osd.1.smithi104.stderr:2024-03-28T14:07:45.092+0000 7f5ccba11640 -1 osd.1 0 waiting for initial osdmap 2024-03-28T14:07:45.095 INFO:tasks.ceph.osd.2.smithi104.stderr:2024-03-28T14:07:45.096+0000 7fce53de1640 -1 osd.2 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-28T14:07:45.113 INFO:tasks.ceph.osd.5.smithi171.stderr:2024-03-28T14:07:45.114+0000 7f0107a15640 -1 osd.5 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-28T14:07:45.116 INFO:tasks.ceph.osd.1.smithi104.stderr:2024-03-28T14:07:45.116+0000 7f5cc3033640 -1 osd.1 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-28T14:07:45.117 INFO:tasks.ceph.osd.4.smithi171.stderr:2024-03-28T14:07:45.118+0000 7f7a10a8c640 -1 osd.4 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-28T14:07:45.242 DEBUG:teuthology.orchestra.run.smithi104:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-28T14:07:45.358 INFO:teuthology.misc.health.smithi104.stderr:2024-03-28T14:07:45.356+0000 7ff2b5fe0640 1 Processor -- start 2024-03-28T14:07:45.361 INFO:teuthology.misc.health.smithi104.stderr:2024-03-28T14:07:45.356+0000 7ff2b5fe0640 1 -- start start 2024-03-28T14:07:45.363 INFO:teuthology.misc.health.smithi104.stderr:2024-03-28T14:07:45.360+0000 7ff2b5fe0640 1 --2- >> v2:172.21.15.171:3300/0 conn(0x7ff2b00b0b50 0x7ff2b00b12f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:07:45.363 INFO:teuthology.misc.health.smithi104.stderr:2024-03-28T14:07:45.360+0000 7ff2b5fe0640 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7ff2b013f9c0 0x7ff2b014fe30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:07:45.363 INFO:teuthology.misc.health.smithi104.stderr:2024-03-28T14:07:45.360+0000 7ff2b5fe0640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff2b00b2d50 con 0x7ff2b006d970 2024-03-28T14:07:45.364 INFO:teuthology.misc.health.smithi104.stderr:2024-03-28T14:07:45.360+0000 7ff2b5fe0640 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 -- 0x7ff2b006dd50 con 0x7ff2b013f9c0 2024-03-28T14:07:45.364 INFO:teuthology.misc.health.smithi104.stderr:2024-03-28T14:07:45.360+0000 7ff2b5fe0640 1 -- --> v2:172.21.15.171:3300/0 -- mon_getmap magic: 0 -- 0x7ff2b00b2ff0 con 0x7ff2b00b0b50 2024-03-28T14:07:45.364 INFO:teuthology.misc.health.smithi104.stderr:2024-03-28T14:07:45.360+0000 7ff2affff640 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7ff2b013f9c0 0x7ff2b014fe30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:07:45.364 INFO:teuthology.misc.health.smithi104.stderr:2024-03-28T14:07:45.360+0000 7ff2affff640 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7ff2b013f9c0 0x7ff2b014fe30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:54822/0 (socket says 172.21.15.104:54822) 2024-03-28T14:07:45.364 INFO:teuthology.misc.health.smithi104.stderr:2024-03-28T14:07:45.360+0000 7ff2affff640 1 -- 172.21.15.104:0/1791362532 learned_addr learned my addr 172.21.15.104:0/1791362532 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-03-28T14:07:45.364 INFO:teuthology.misc.health.smithi104.stderr:2024-03-28T14:07:45.360+0000 7ff2aeffd640 1 --2- 172.21.15.104:0/1791362532 >> v2:172.21.15.171:3300/0 conn(0x7ff2b00b0b50 0x7ff2b00b12f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:07:45.364 INFO:teuthology.misc.health.smithi104.stderr:2024-03-28T14:07:45.360+0000 7ff2affff640 1 -- 172.21.15.104:0/1791362532 >> v2:172.21.15.171:3300/0 conn(0x7ff2b00b0b50 msgr2=0x7ff2b00b12f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T14:07:45.364 INFO:teuthology.misc.health.smithi104.stderr:2024-03-28T14:07:45.360+0000 7ff2affff640 1 --2- 172.21.15.104:0/1791362532 >> v2:172.21.15.171:3300/0 conn(0x7ff2b00b0b50 0x7ff2b00b12f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:07:45.364 INFO:teuthology.misc.health.smithi104.stderr:2024-03-28T14:07:45.360+0000 7ff2affff640 1 -- 172.21.15.104:0/1791362532 >> v1:172.21.15.171:6789/0 conn(0x7ff2b006d970 legacy=0x7ff2b013f2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T14:07:45.364 INFO:teuthology.misc.health.smithi104.stderr:2024-03-28T14:07:45.360+0000 7ff2affff640 1 -- 172.21.15.104:0/1791362532 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff2b009ee10 con 0x7ff2b013f9c0 2024-03-28T14:07:45.364 INFO:teuthology.misc.health.smithi104.stderr:2024-03-28T14:07:45.360+0000 7ff2affff640 1 --2- 172.21.15.104:0/1791362532 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff2b013f9c0 0x7ff2b014fe30 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7ff2b00b2490 tx=0x7ff2a801c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e4b149f1d419b106 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:07:45.364 INFO:teuthology.misc.health.smithi104.stderr:2024-03-28T14:07:45.360+0000 7ff2ae7fc640 1 -- 172.21.15.104:0/1791362532 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff2a80a56f0 con 0x7ff2b013f9c0 2024-03-28T14:07:45.364 INFO:teuthology.misc.health.smithi104.stderr:2024-03-28T14:07:45.360+0000 7ff2ae7fc640 1 -- 172.21.15.104:0/1791362532 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff2a80a5890 con 0x7ff2b013f9c0 2024-03-28T14:07:45.364 INFO:teuthology.misc.health.smithi104.stderr:2024-03-28T14:07:45.360+0000 7ff2ae7fc640 1 -- 172.21.15.104:0/1791362532 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff2a80a5c60 con 0x7ff2b013f9c0 2024-03-28T14:07:45.364 INFO:teuthology.misc.health.smithi104.stderr:2024-03-28T14:07:45.364+0000 7ff2b5fe0640 1 -- 172.21.15.104:0/1791362532 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff2b013f9c0 msgr2=0x7ff2b014fe30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:07:45.365 INFO:teuthology.misc.health.smithi104.stderr:2024-03-28T14:07:45.364+0000 7ff2b5fe0640 1 --2- 172.21.15.104:0/1791362532 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff2b013f9c0 0x7ff2b014fe30 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7ff2b00b2490 tx=0x7ff2a801c920 comp rx=0 tx=0).stop 2024-03-28T14:07:45.365 INFO:teuthology.misc.health.smithi104.stderr:2024-03-28T14:07:45.364+0000 7ff2b5fe0640 1 -- 172.21.15.104:0/1791362532 shutdown_connections 2024-03-28T14:07:45.365 INFO:teuthology.misc.health.smithi104.stderr:2024-03-28T14:07:45.364+0000 7ff2b5fe0640 1 --2- 172.21.15.104:0/1791362532 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff2b013f9c0 0x7ff2b014fe30 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:07:45.365 INFO:teuthology.misc.health.smithi104.stderr:2024-03-28T14:07:45.364+0000 7ff2b5fe0640 1 --2- 172.21.15.104:0/1791362532 >> v2:172.21.15.171:3300/0 conn(0x7ff2b00b0b50 0x7ff2b00b12f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:07:45.365 INFO:teuthology.misc.health.smithi104.stderr:2024-03-28T14:07:45.364+0000 7ff2b5fe0640 1 -- 172.21.15.104:0/1791362532 >> 172.21.15.104:0/1791362532 conn(0x7ff2b00b58e0 msgr2=0x7ff2b00b5cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T14:07:45.365 INFO:teuthology.misc.health.smithi104.stderr:2024-03-28T14:07:45.364+0000 7ff2b5fe0640 1 -- 172.21.15.104:0/1791362532 shutdown_connections 2024-03-28T14:07:45.365 INFO:teuthology.misc.health.smithi104.stderr:2024-03-28T14:07:45.364+0000 7ff2b5fe0640 1 -- 172.21.15.104:0/1791362532 wait complete. 2024-03-28T14:07:45.367 INFO:teuthology.misc.health.smithi104.stderr:2024-03-28T14:07:45.364+0000 7ff2b5fe0640 1 Processor -- start 2024-03-28T14:07:45.367 INFO:teuthology.misc.health.smithi104.stderr:2024-03-28T14:07:45.364+0000 7ff2b5fe0640 1 -- start start 2024-03-28T14:07:45.368 INFO:teuthology.misc.health.smithi104.stderr:2024-03-28T14:07:45.364+0000 7ff2b5fe0640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff2b006d970 0x7ff2b0138440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:07:45.368 INFO:teuthology.misc.health.smithi104.stderr:2024-03-28T14:07:45.364+0000 7ff2b5fe0640 1 --2- >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7ff2b00b0b50 0x7ff2b01e0240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:07:45.368 INFO:teuthology.misc.health.smithi104.stderr:2024-03-28T14:07:45.364+0000 7ff2b5fe0640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7ff2b013f9c0 0x7ff2b01e0780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:07:45.368 INFO:teuthology.misc.health.smithi104.stderr:2024-03-28T14:07:45.364+0000 7ff2b5fe0640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7ff2b0152f20 con 0x7ff2b006d970 2024-03-28T14:07:45.368 INFO:teuthology.misc.health.smithi104.stderr:2024-03-28T14:07:45.364+0000 7ff2b5fe0640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7ff2b0152da0 con 0x7ff2b013f9c0 2024-03-28T14:07:45.368 INFO:teuthology.misc.health.smithi104.stderr:2024-03-28T14:07:45.364+0000 7ff2b5fe0640 1 -- --> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] -- mon_getmap magic: 0 -- 0x7ff2b01530a0 con 0x7ff2b00b0b50 2024-03-28T14:07:45.368 INFO:teuthology.misc.health.smithi104.stderr:2024-03-28T14:07:45.364+0000 7ff2af7fe640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff2b006d970 0x7ff2b0138440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:07:45.368 INFO:teuthology.misc.health.smithi104.stderr:2024-03-28T14:07:45.364+0000 7ff2af7fe640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff2b006d970 0x7ff2b0138440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:54830/0 (socket says 172.21.15.104:54830) 2024-03-28T14:07:45.368 INFO:teuthology.misc.health.smithi104.stderr:2024-03-28T14:07:45.364+0000 7ff2af7fe640 1 -- 172.21.15.104:0/3540684873 learned_addr learned my addr 172.21.15.104:0/3540684873 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-03-28T14:07:45.368 INFO:teuthology.misc.health.smithi104.stderr:2024-03-28T14:07:45.364+0000 7ff2affff640 1 --2- 172.21.15.104:0/3540684873 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7ff2b013f9c0 0x7ff2b01e0780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:07:45.368 INFO:teuthology.misc.health.smithi104.stderr:2024-03-28T14:07:45.364+0000 7ff2aeffd640 1 --2- 172.21.15.104:0/3540684873 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7ff2b00b0b50 0x7ff2b01e0240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:07:45.368 INFO:teuthology.misc.health.smithi104.stderr:2024-03-28T14:07:45.364+0000 7ff2af7fe640 1 -- 172.21.15.104:0/3540684873 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7ff2b00b0b50 msgr2=0x7ff2b01e0240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:07:45.368 INFO:teuthology.misc.health.smithi104.stderr:2024-03-28T14:07:45.364+0000 7ff2af7fe640 1 --2- 172.21.15.104:0/3540684873 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7ff2b00b0b50 0x7ff2b01e0240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:07:45.368 INFO:teuthology.misc.health.smithi104.stderr:2024-03-28T14:07:45.364+0000 7ff2af7fe640 1 -- 172.21.15.104:0/3540684873 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7ff2b013f9c0 msgr2=0x7ff2b01e0780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:07:45.369 INFO:teuthology.misc.health.smithi104.stderr:2024-03-28T14:07:45.364+0000 7ff2af7fe640 1 --2- 172.21.15.104:0/3540684873 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7ff2b013f9c0 0x7ff2b01e0780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:07:45.369 INFO:teuthology.misc.health.smithi104.stderr:2024-03-28T14:07:45.364+0000 7ff2af7fe640 1 -- 172.21.15.104:0/3540684873 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff2b014ea00 con 0x7ff2b006d970 2024-03-28T14:07:45.369 INFO:teuthology.misc.health.smithi104.stderr:2024-03-28T14:07:45.364+0000 7ff2af7fe640 1 --2- 172.21.15.104:0/3540684873 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff2b006d970 0x7ff2b0138440 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7ff2a000c480 tx=0x7ff2a000c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:07:45.369 INFO:teuthology.misc.health.smithi104.stderr:2024-03-28T14:07:45.364+0000 7ff297fff640 1 -- 172.21.15.104:0/3540684873 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff2a0012830 con 0x7ff2b006d970 2024-03-28T14:07:45.369 INFO:teuthology.misc.health.smithi104.stderr:2024-03-28T14:07:45.364+0000 7ff297fff640 1 -- 172.21.15.104:0/3540684873 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff2a0017070 con 0x7ff2b006d970 2024-03-28T14:07:45.369 INFO:teuthology.misc.health.smithi104.stderr:2024-03-28T14:07:45.364+0000 7ff2b5fe0640 1 -- 172.21.15.104:0/3540684873 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff2b00aed70 con 0x7ff2b006d970 2024-03-28T14:07:45.369 INFO:teuthology.misc.health.smithi104.stderr:2024-03-28T14:07:45.364+0000 7ff297fff640 1 -- 172.21.15.104:0/3540684873 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff2a0012c30 con 0x7ff2b006d970 2024-03-28T14:07:45.369 INFO:teuthology.misc.health.smithi104.stderr:2024-03-28T14:07:45.364+0000 7ff2b5fe0640 1 -- 172.21.15.104:0/3540684873 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff2b013b480 con 0x7ff2b006d970 2024-03-28T14:07:45.369 INFO:teuthology.misc.health.smithi104.stderr:2024-03-28T14:07:45.364+0000 7ff2b5fe0640 1 -- 172.21.15.104:0/3540684873 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff2b00af0a0 con 0x7ff2b006d970 2024-03-28T14:07:45.369 INFO:teuthology.misc.health.smithi104.stderr:2024-03-28T14:07:45.368+0000 7ff297fff640 1 -- 172.21.15.104:0/3540684873 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 4) ==== 87747+0+0 (secure 0 0 0) 0x7ff2a001c460 con 0x7ff2b006d970 2024-03-28T14:07:45.369 INFO:teuthology.misc.health.smithi104.stderr:2024-03-28T14:07:45.368+0000 7ff297fff640 1 --2- 172.21.15.104:0/3540684873 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7ff29c067f60 0x7ff29c06a420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:07:45.369 INFO:teuthology.misc.health.smithi104.stderr:2024-03-28T14:07:45.368+0000 7ff297fff640 1 -- 172.21.15.104:0/3540684873 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(12..12 src has 1..12) ==== 2231+0+0 (secure 0 0 0) 0x7ff2a0085430 con 0x7ff2b006d970 2024-03-28T14:07:45.369 INFO:teuthology.misc.health.smithi104.stderr:2024-03-28T14:07:45.368+0000 7ff2aeffd640 1 --2- 172.21.15.104:0/3540684873 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7ff29c067f60 0x7ff29c06a420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:07:45.370 INFO:teuthology.misc.health.smithi104.stderr:2024-03-28T14:07:45.368+0000 7ff2aeffd640 1 --2- 172.21.15.104:0/3540684873 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7ff29c067f60 0x7ff29c06a420 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7ff2b01502c0 tx=0x7ff2a4008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:07:45.371 INFO:teuthology.misc.health.smithi104.stderr:2024-03-28T14:07:45.372+0000 7ff297fff640 1 -- 172.21.15.104:0/3540684873 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179117 (secure 0 0 0) 0x7ff2a0056ae0 con 0x7ff2b006d970 2024-03-28T14:07:45.533 INFO:teuthology.misc.health.smithi104.stderr:2024-03-28T14:07:45.532+0000 7ff2b5fe0640 1 -- 172.21.15.104:0/3540684873 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7ff2b0068f70 con 0x7ff2b006d970 2024-03-28T14:07:45.533 INFO:teuthology.misc.health.smithi104.stderr:2024-03-28T14:07:45.532+0000 7ff297fff640 1 -- 172.21.15.104:0/3540684873 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v12) ==== 74+0+7058 (secure 0 0 0) 0x7ff2a00563e0 con 0x7ff2b006d970 2024-03-28T14:07:45.534 INFO:teuthology.misc.health.smithi104.stdout: 2024-03-28T14:07:45.534 INFO:teuthology.misc.health.smithi104.stdout:{"epoch":12,"fsid":"428a90f2-3351-435f-8fcf-f7e66c78a13e","created":"2024-03-28T14:07:34.862458+0000","modified":"2024-03-28T14:07:42.911200+0000","last_up_change":"0.000000","last_in_change":"2024-03-28T14:07:37.157486+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"1cb70540-4e42-402f-a446-aaec4ca83007","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"3ab7afdd-b980-418c-81f2-004e16de3dca","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"3e2651c8-ecae-4a9e-a053-27240a79eac2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"925c7e0a-38ef-47cd-b288-8bec8456fb17","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"4a8fac86-e595-4a0e-96e3-57edb3cc3f57","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"e48d50b0-beae-40ae-b15a-9a84dbfe88ab","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"5e7c3292-9936-4f35-b267-aed9d4e49323","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"339b8414-d7f0-4fb0-a446-efaf635cb7ff","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-28T14:07:45.536 INFO:teuthology.misc.health.smithi104.stderr:2024-03-28T14:07:45.536+0000 7ff2b5fe0640 1 -- 172.21.15.104:0/3540684873 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7ff29c067f60 msgr2=0x7ff29c06a420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:07:45.536 INFO:teuthology.misc.health.smithi104.stderr:2024-03-28T14:07:45.536+0000 7ff2b5fe0640 1 --2- 172.21.15.104:0/3540684873 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7ff29c067f60 0x7ff29c06a420 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7ff2b01502c0 tx=0x7ff2a4008040 comp rx=0 tx=0).stop 2024-03-28T14:07:45.536 INFO:teuthology.misc.health.smithi104.stderr:2024-03-28T14:07:45.536+0000 7ff2b5fe0640 1 -- 172.21.15.104:0/3540684873 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff2b006d970 msgr2=0x7ff2b0138440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:07:45.536 INFO:teuthology.misc.health.smithi104.stderr:2024-03-28T14:07:45.536+0000 7ff2b5fe0640 1 --2- 172.21.15.104:0/3540684873 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff2b006d970 0x7ff2b0138440 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7ff2a000c480 tx=0x7ff2a000c940 comp rx=0 tx=0).stop 2024-03-28T14:07:45.536 INFO:teuthology.misc.health.smithi104.stderr:2024-03-28T14:07:45.536+0000 7ff2b5fe0640 1 -- 172.21.15.104:0/3540684873 shutdown_connections 2024-03-28T14:07:45.536 INFO:teuthology.misc.health.smithi104.stderr:2024-03-28T14:07:45.536+0000 7ff2b5fe0640 1 --2- 172.21.15.104:0/3540684873 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7ff2b013f9c0 0x7ff2b01e0780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:07:45.536 INFO:teuthology.misc.health.smithi104.stderr:2024-03-28T14:07:45.536+0000 7ff2b5fe0640 1 --2- 172.21.15.104:0/3540684873 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7ff2b00b0b50 0x7ff2b01e0240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:07:45.536 INFO:teuthology.misc.health.smithi104.stderr:2024-03-28T14:07:45.536+0000 7ff2b5fe0640 1 --2- 172.21.15.104:0/3540684873 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7ff29c067f60 0x7ff29c06a420 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:07:45.536 INFO:teuthology.misc.health.smithi104.stderr:2024-03-28T14:07:45.536+0000 7ff2b5fe0640 1 --2- 172.21.15.104:0/3540684873 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff2b006d970 0x7ff2b0138440 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:07:45.537 INFO:teuthology.misc.health.smithi104.stderr:2024-03-28T14:07:45.536+0000 7ff2b5fe0640 1 -- 172.21.15.104:0/3540684873 >> 172.21.15.104:0/3540684873 conn(0x7ff2b00b58e0 msgr2=0x7ff2b013e480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T14:07:45.537 INFO:teuthology.misc.health.smithi104.stderr:2024-03-28T14:07:45.536+0000 7ff2b5fe0640 1 -- 172.21.15.104:0/3540684873 shutdown_connections 2024-03-28T14:07:45.537 INFO:teuthology.misc.health.smithi104.stderr:2024-03-28T14:07:45.536+0000 7ff2b5fe0640 1 -- 172.21.15.104:0/3540684873 wait complete. 2024-03-28T14:07:45.551 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-28T14:07:52.358 DEBUG:teuthology.orchestra.run.smithi104:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-28T14:07:52.482 INFO:teuthology.misc.health.smithi104.stderr:2024-03-28T14:07:52.479+0000 7fe31326c640 1 Processor -- start 2024-03-28T14:07:52.482 INFO:teuthology.misc.health.smithi104.stderr:2024-03-28T14:07:52.479+0000 7fe31326c640 1 -- start start 2024-03-28T14:07:52.484 INFO:teuthology.misc.health.smithi104.stderr:2024-03-28T14:07:52.483+0000 7fe31326c640 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fe30c06d970 0x7fe30c0b02e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:07:52.484 INFO:teuthology.misc.health.smithi104.stderr:2024-03-28T14:07:52.483+0000 7fe31326c640 1 --2- >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fe30c0ae290 0x7fe30c0ae6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:07:52.484 INFO:teuthology.misc.health.smithi104.stderr:2024-03-28T14:07:52.483+0000 7fe31326c640 1 --2- >> v2:172.21.15.171:3300/0 conn(0x7fe30c147b80 0x7fe30c147ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:07:52.484 INFO:teuthology.misc.health.smithi104.stderr:2024-03-28T14:07:52.483+0000 7fe31326c640 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 -- 0x7fe30c0ade00 con 0x7fe30c06d970 2024-03-28T14:07:52.485 INFO:teuthology.misc.health.smithi104.stderr:2024-03-28T14:07:52.483+0000 7fe31326c640 1 -- --> v2:172.21.15.171:3300/0 -- mon_getmap magic: 0 -- 0x7fe30c0b1500 con 0x7fe30c147b80 2024-03-28T14:07:52.485 INFO:teuthology.misc.health.smithi104.stderr:2024-03-28T14:07:52.483+0000 7fe31326c640 1 -- --> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] -- mon_getmap magic: 0 -- 0x7fe30c06dd50 con 0x7fe30c0ae290 2024-03-28T14:07:52.485 INFO:teuthology.misc.health.smithi104.stderr:2024-03-28T14:07:52.483+0000 7fe311008640 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fe30c06d970 0x7fe30c0b02e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:07:52.485 INFO:teuthology.misc.health.smithi104.stderr:2024-03-28T14:07:52.483+0000 7fe311008640 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fe30c06d970 0x7fe30c0b02e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:48978/0 (socket says 172.21.15.104:48978) 2024-03-28T14:07:52.485 INFO:teuthology.misc.health.smithi104.stderr:2024-03-28T14:07:52.483+0000 7fe311008640 1 -- 172.21.15.104:0/2687152264 learned_addr learned my addr 172.21.15.104:0/2687152264 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-03-28T14:07:52.485 INFO:teuthology.misc.health.smithi104.stderr:2024-03-28T14:07:52.483+0000 7fe311809640 1 --2- 172.21.15.104:0/2687152264 >> v2:172.21.15.171:3300/0 conn(0x7fe30c147b80 0x7fe30c147ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:07:52.485 INFO:teuthology.misc.health.smithi104.stderr:2024-03-28T14:07:52.483+0000 7fe310807640 1 --2- 172.21.15.104:0/2687152264 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fe30c0ae290 0x7fe30c0ae6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:07:52.485 INFO:teuthology.misc.health.smithi104.stderr:2024-03-28T14:07:52.483+0000 7fe311008640 1 -- 172.21.15.104:0/2687152264 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fe30c0ae290 msgr2=0x7fe30c0ae6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T14:07:52.485 INFO:teuthology.misc.health.smithi104.stderr:2024-03-28T14:07:52.483+0000 7fe311008640 1 --2- 172.21.15.104:0/2687152264 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fe30c0ae290 0x7fe30c0ae6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:07:52.485 INFO:teuthology.misc.health.smithi104.stderr:2024-03-28T14:07:52.483+0000 7fe311008640 1 -- 172.21.15.104:0/2687152264 >> v2:172.21.15.171:3300/0 conn(0x7fe30c147b80 msgr2=0x7fe30c147ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T14:07:52.485 INFO:teuthology.misc.health.smithi104.stderr:2024-03-28T14:07:52.483+0000 7fe311008640 1 --2- 172.21.15.104:0/2687152264 >> v2:172.21.15.171:3300/0 conn(0x7fe30c147b80 0x7fe30c147ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:07:52.485 INFO:teuthology.misc.health.smithi104.stderr:2024-03-28T14:07:52.483+0000 7fe311008640 1 -- 172.21.15.104:0/2687152264 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe30c0b5120 con 0x7fe30c06d970 2024-03-28T14:07:52.485 INFO:teuthology.misc.health.smithi104.stderr:2024-03-28T14:07:52.483+0000 7fe311809640 1 --2- 172.21.15.104:0/2687152264 >> v2:172.21.15.171:3300/0 conn(0x7fe30c147b80 0x7fe30c147ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T14:07:52.485 INFO:teuthology.misc.health.smithi104.stderr:2024-03-28T14:07:52.483+0000 7fe310807640 1 --2- 172.21.15.104:0/2687152264 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fe30c0ae290 0x7fe30c0ae6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T14:07:52.487 INFO:teuthology.misc.health.smithi104.stderr:2024-03-28T14:07:52.483+0000 7fe311008640 1 --2- 172.21.15.104:0/2687152264 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe30c06d970 0x7fe30c0b02e0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fe30c073cb0 tx=0x7fe30801c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bf6946fcb3c4cdaa server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:07:52.487 INFO:teuthology.misc.health.smithi104.stderr:2024-03-28T14:07:52.483+0000 7fe303fff640 1 -- 172.21.15.104:0/2687152264 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe3080b0070 con 0x7fe30c06d970 2024-03-28T14:07:52.487 INFO:teuthology.misc.health.smithi104.stderr:2024-03-28T14:07:52.483+0000 7fe303fff640 1 -- 172.21.15.104:0/2687152264 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe3080ab600 con 0x7fe30c06d970 2024-03-28T14:07:52.487 INFO:teuthology.misc.health.smithi104.stderr:2024-03-28T14:07:52.483+0000 7fe303fff640 1 -- 172.21.15.104:0/2687152264 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe3080ab9d0 con 0x7fe30c06d970 2024-03-28T14:07:52.488 INFO:teuthology.misc.health.smithi104.stderr:2024-03-28T14:07:52.487+0000 7fe31326c640 1 -- 172.21.15.104:0/2687152264 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe30c06d970 msgr2=0x7fe30c0b02e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:07:52.488 INFO:teuthology.misc.health.smithi104.stderr:2024-03-28T14:07:52.487+0000 7fe31326c640 1 --2- 172.21.15.104:0/2687152264 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe30c06d970 0x7fe30c0b02e0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fe30c073cb0 tx=0x7fe30801c490 comp rx=0 tx=0).stop 2024-03-28T14:07:52.488 INFO:teuthology.misc.health.smithi104.stderr:2024-03-28T14:07:52.487+0000 7fe31326c640 1 -- 172.21.15.104:0/2687152264 shutdown_connections 2024-03-28T14:07:52.488 INFO:teuthology.misc.health.smithi104.stderr:2024-03-28T14:07:52.487+0000 7fe31326c640 1 --2- 172.21.15.104:0/2687152264 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fe30c0ae290 0x7fe30c0ae6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:07:52.488 INFO:teuthology.misc.health.smithi104.stderr:2024-03-28T14:07:52.487+0000 7fe31326c640 1 --2- 172.21.15.104:0/2687152264 >> v2:172.21.15.171:3300/0 conn(0x7fe30c147b80 0x7fe30c147ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:07:52.488 INFO:teuthology.misc.health.smithi104.stderr:2024-03-28T14:07:52.487+0000 7fe31326c640 1 --2- 172.21.15.104:0/2687152264 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe30c06d970 0x7fe30c0b02e0 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:07:52.488 INFO:teuthology.misc.health.smithi104.stderr:2024-03-28T14:07:52.487+0000 7fe31326c640 1 -- 172.21.15.104:0/2687152264 >> 172.21.15.104:0/2687152264 conn(0x7fe30c0b58e0 msgr2=0x7fe30c0b5cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T14:07:52.488 INFO:teuthology.misc.health.smithi104.stderr:2024-03-28T14:07:52.487+0000 7fe31326c640 1 -- 172.21.15.104:0/2687152264 shutdown_connections 2024-03-28T14:07:52.488 INFO:teuthology.misc.health.smithi104.stderr:2024-03-28T14:07:52.487+0000 7fe31326c640 1 -- 172.21.15.104:0/2687152264 wait complete. 2024-03-28T14:07:52.490 INFO:teuthology.misc.health.smithi104.stderr:2024-03-28T14:07:52.487+0000 7fe31326c640 1 Processor -- start 2024-03-28T14:07:52.490 INFO:teuthology.misc.health.smithi104.stderr:2024-03-28T14:07:52.487+0000 7fe31326c640 1 -- start start 2024-03-28T14:07:52.490 INFO:teuthology.misc.health.smithi104.stderr:2024-03-28T14:07:52.487+0000 7fe31326c640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fe30c06d970 0x7fe30c13b4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:07:52.491 INFO:teuthology.misc.health.smithi104.stderr:2024-03-28T14:07:52.487+0000 7fe31326c640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe30c0ae290 0x7fe30c0b7da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:07:52.491 INFO:teuthology.misc.health.smithi104.stderr:2024-03-28T14:07:52.487+0000 7fe31326c640 1 --2- >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fe30c147b80 0x7fe30c071440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:07:52.491 INFO:teuthology.misc.health.smithi104.stderr:2024-03-28T14:07:52.487+0000 7fe31326c640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7fe30c148940 con 0x7fe30c0ae290 2024-03-28T14:07:52.491 INFO:teuthology.misc.health.smithi104.stderr:2024-03-28T14:07:52.487+0000 7fe31326c640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7fe30c0aebf0 con 0x7fe30c06d970 2024-03-28T14:07:52.491 INFO:teuthology.misc.health.smithi104.stderr:2024-03-28T14:07:52.487+0000 7fe31326c640 1 -- --> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] -- mon_getmap magic: 0 -- 0x7fe30c0b2480 con 0x7fe30c147b80 2024-03-28T14:07:52.491 INFO:teuthology.misc.health.smithi104.stderr:2024-03-28T14:07:52.487+0000 7fe310807640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe30c0ae290 0x7fe30c0b7da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:07:52.491 INFO:teuthology.misc.health.smithi104.stderr:2024-03-28T14:07:52.487+0000 7fe310807640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe30c0ae290 0x7fe30c0b7da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:48992/0 (socket says 172.21.15.104:48992) 2024-03-28T14:07:52.491 INFO:teuthology.misc.health.smithi104.stderr:2024-03-28T14:07:52.487+0000 7fe310807640 1 -- 172.21.15.104:0/4155376830 learned_addr learned my addr 172.21.15.104:0/4155376830 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-03-28T14:07:52.491 INFO:teuthology.misc.health.smithi104.stderr:2024-03-28T14:07:52.487+0000 7fe311008640 1 --2- 172.21.15.104:0/4155376830 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fe30c06d970 0x7fe30c13b4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:07:52.491 INFO:teuthology.misc.health.smithi104.stderr:2024-03-28T14:07:52.487+0000 7fe311809640 1 --2- 172.21.15.104:0/4155376830 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fe30c147b80 0x7fe30c071440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:07:52.491 INFO:teuthology.misc.health.smithi104.stderr:2024-03-28T14:07:52.487+0000 7fe310807640 1 -- 172.21.15.104:0/4155376830 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fe30c147b80 msgr2=0x7fe30c071440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:07:52.491 INFO:teuthology.misc.health.smithi104.stderr:2024-03-28T14:07:52.487+0000 7fe310807640 1 --2- 172.21.15.104:0/4155376830 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fe30c147b80 0x7fe30c071440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:07:52.491 INFO:teuthology.misc.health.smithi104.stderr:2024-03-28T14:07:52.487+0000 7fe310807640 1 -- 172.21.15.104:0/4155376830 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fe30c06d970 msgr2=0x7fe30c13b4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:07:52.491 INFO:teuthology.misc.health.smithi104.stderr:2024-03-28T14:07:52.487+0000 7fe310807640 1 --2- 172.21.15.104:0/4155376830 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fe30c06d970 0x7fe30c13b4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:07:52.492 INFO:teuthology.misc.health.smithi104.stderr:2024-03-28T14:07:52.487+0000 7fe310807640 1 -- 172.21.15.104:0/4155376830 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe30c06ff30 con 0x7fe30c0ae290 2024-03-28T14:07:52.492 INFO:teuthology.misc.health.smithi104.stderr:2024-03-28T14:07:52.487+0000 7fe310807640 1 --2- 172.21.15.104:0/4155376830 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe30c0ae290 0x7fe30c0b7da0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fe2fc00c7f0 tx=0x7fe2fc00ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:07:52.492 INFO:teuthology.misc.health.smithi104.stderr:2024-03-28T14:07:52.487+0000 7fe301ffb640 1 -- 172.21.15.104:0/4155376830 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2fc007a30 con 0x7fe30c0ae290 2024-03-28T14:07:52.492 INFO:teuthology.misc.health.smithi104.stderr:2024-03-28T14:07:52.487+0000 7fe301ffb640 1 -- 172.21.15.104:0/4155376830 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe2fc007bd0 con 0x7fe30c0ae290 2024-03-28T14:07:52.492 INFO:teuthology.misc.health.smithi104.stderr:2024-03-28T14:07:52.487+0000 7fe31326c640 1 -- 172.21.15.104:0/4155376830 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe30c1431f0 con 0x7fe30c0ae290 2024-03-28T14:07:52.492 INFO:teuthology.misc.health.smithi104.stderr:2024-03-28T14:07:52.487+0000 7fe301ffb640 1 -- 172.21.15.104:0/4155376830 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2fc0113f0 con 0x7fe30c0ae290 2024-03-28T14:07:52.492 INFO:teuthology.misc.health.smithi104.stderr:2024-03-28T14:07:52.487+0000 7fe31326c640 1 -- 172.21.15.104:0/4155376830 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe30c0b75a0 con 0x7fe30c0ae290 2024-03-28T14:07:52.493 INFO:teuthology.misc.health.smithi104.stderr:2024-03-28T14:07:52.491+0000 7fe31326c640 1 -- 172.21.15.104:0/4155376830 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe30c068f70 con 0x7fe30c0ae290 2024-03-28T14:07:52.493 INFO:teuthology.misc.health.smithi104.stderr:2024-03-28T14:07:52.491+0000 7fe301ffb640 1 -- 172.21.15.104:0/4155376830 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) ==== 87803+0+0 (secure 0 0 0) 0x7fe2fc011590 con 0x7fe30c0ae290 2024-03-28T14:07:52.493 INFO:teuthology.misc.health.smithi104.stderr:2024-03-28T14:07:52.491+0000 7fe301ffb640 1 --2- 172.21.15.104:0/4155376830 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7fe2f0067fc0 0x7fe2f006a480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:07:52.493 INFO:teuthology.misc.health.smithi104.stderr:2024-03-28T14:07:52.491+0000 7fe301ffb640 1 -- 172.21.15.104:0/4155376830 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(17..17 src has 1..17) ==== 4857+0+0 (secure 0 0 0) 0x7fe2fc0863d0 con 0x7fe30c0ae290 2024-03-28T14:07:52.493 INFO:teuthology.misc.health.smithi104.stderr:2024-03-28T14:07:52.491+0000 7fe311008640 1 --2- 172.21.15.104:0/4155376830 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7fe2f0067fc0 0x7fe2f006a480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:07:52.493 INFO:teuthology.misc.health.smithi104.stderr:2024-03-28T14:07:52.491+0000 7fe311008640 1 --2- 172.21.15.104:0/4155376830 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7fe2f0067fc0 0x7fe2f006a480 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fe30c06e360 tx=0x7fe308008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:07:52.497 INFO:teuthology.misc.health.smithi104.stderr:2024-03-28T14:07:52.495+0000 7fe301ffb640 1 -- 172.21.15.104:0/4155376830 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179117 (secure 0 0 0) 0x7fe2fc015020 con 0x7fe30c0ae290 2024-03-28T14:07:52.653 INFO:teuthology.misc.health.smithi104.stderr:2024-03-28T14:07:52.651+0000 7fe31326c640 1 -- 172.21.15.104:0/4155376830 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fe30c06fc50 con 0x7fe30c0ae290 2024-03-28T14:07:52.655 INFO:teuthology.misc.health.smithi104.stderr:2024-03-28T14:07:52.655+0000 7fe301ffb640 1 -- 172.21.15.104:0/4155376830 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v17) ==== 74+0+12908 (secure 0 0 0) 0x7fe2fc056ed0 con 0x7fe30c0ae290 2024-03-28T14:07:52.655 INFO:teuthology.misc.health.smithi104.stdout: 2024-03-28T14:07:52.655 INFO:teuthology.misc.health.smithi104.stdout:{"epoch":17,"fsid":"428a90f2-3351-435f-8fcf-f7e66c78a13e","created":"2024-03-28T14:07:34.862458+0000","modified":"2024-03-28T14:07:50.087852+0000","last_up_change":"2024-03-28T14:07:46.028122+0000","last_in_change":"2024-03-28T14:07:37.157486+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-28T14:07:47.622988+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"1cb70540-4e42-402f-a446-aaec4ca83007","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6800","nonce":3745690918},{"type":"v1","addr":"172.21.15.104:6801","nonce":3745690918}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":3745690918},{"type":"v1","addr":"172.21.15.104:6803","nonce":3745690918}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":3745690918},{"type":"v1","addr":"172.21.15.104:6807","nonce":3745690918}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":3745690918},{"type":"v1","addr":"172.21.15.104:6805","nonce":3745690918}]},"public_addr":"172.21.15.104:6801/3745690918","cluster_addr":"172.21.15.104:6803/3745690918","heartbeat_back_addr":"172.21.15.104:6807/3745690918","heartbeat_front_addr":"172.21.15.104:6805/3745690918","state":["exists","up"]},{"osd":1,"uuid":"3ab7afdd-b980-418c-81f2-004e16de3dca","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":19438674},{"type":"v1","addr":"172.21.15.104:6810","nonce":19438674}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":19438674},{"type":"v1","addr":"172.21.15.104:6814","nonce":19438674}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":19438674},{"type":"v1","addr":"172.21.15.104:6823","nonce":19438674}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":19438674},{"type":"v1","addr":"172.21.15.104:6819","nonce":19438674}]},"public_addr":"172.21.15.104:6810/19438674","cluster_addr":"172.21.15.104:6814/19438674","heartbeat_back_addr":"172.21.15.104:6823/19438674","heartbeat_front_addr":"172.21.15.104:6819/19438674","state":["exists","up"]},{"osd":2,"uuid":"3e2651c8-ecae-4a9e-a053-27240a79eac2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":2352012626},{"type":"v1","addr":"172.21.15.104:6825","nonce":2352012626}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6826","nonce":2352012626},{"type":"v1","addr":"172.21.15.104:6827","nonce":2352012626}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":2352012626},{"type":"v1","addr":"172.21.15.104:6831","nonce":2352012626}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6828","nonce":2352012626},{"type":"v1","addr":"172.21.15.104:6829","nonce":2352012626}]},"public_addr":"172.21.15.104:6825/2352012626","cluster_addr":"172.21.15.104:6827/2352012626","heartbeat_back_addr":"172.21.15.104:6831/2352012626","heartbeat_front_addr":"172.21.15.104:6829/2352012626","state":["exists","up"]},{"osd":3,"uuid":"925c7e0a-38ef-47cd-b288-8bec8456fb17","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6809","nonce":3568347381},{"type":"v1","addr":"172.21.15.104:6811","nonce":3568347381}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6813","nonce":3568347381},{"type":"v1","addr":"172.21.15.104:6815","nonce":3568347381}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":3568347381},{"type":"v1","addr":"172.21.15.104:6822","nonce":3568347381}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6817","nonce":3568347381},{"type":"v1","addr":"172.21.15.104:6818","nonce":3568347381}]},"public_addr":"172.21.15.104:6811/3568347381","cluster_addr":"172.21.15.104:6815/3568347381","heartbeat_back_addr":"172.21.15.104:6822/3568347381","heartbeat_front_addr":"172.21.15.104:6818/3568347381","state":["exists","up"]},{"osd":4,"uuid":"4a8fac86-e595-4a0e-96e3-57edb3cc3f57","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6800","nonce":2107001077},{"type":"v1","addr":"172.21.15.171:6801","nonce":2107001077}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6802","nonce":2107001077},{"type":"v1","addr":"172.21.15.171:6803","nonce":2107001077}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6806","nonce":2107001077},{"type":"v1","addr":"172.21.15.171:6807","nonce":2107001077}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6804","nonce":2107001077},{"type":"v1","addr":"172.21.15.171:6805","nonce":2107001077}]},"public_addr":"172.21.15.171:6801/2107001077","cluster_addr":"172.21.15.171:6803/2107001077","heartbeat_back_addr":"172.21.15.171:6807/2107001077","heartbeat_front_addr":"172.21.15.171:6805/2107001077","state":["exists","up"]},{"osd":5,"uuid":"e48d50b0-beae-40ae-b15a-9a84dbfe88ab","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6808","nonce":2934826925},{"type":"v1","addr":"172.21.15.171:6810","nonce":2934826925}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6812","nonce":2934826925},{"type":"v1","addr":"172.21.15.171:6814","nonce":2934826925}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6819","nonce":2934826925},{"type":"v1","addr":"172.21.15.171:6821","nonce":2934826925}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6816","nonce":2934826925},{"type":"v1","addr":"172.21.15.171:6817","nonce":2934826925}]},"public_addr":"172.21.15.171:6810/2934826925","cluster_addr":"172.21.15.171:6814/2934826925","heartbeat_back_addr":"172.21.15.171:6821/2934826925","heartbeat_front_addr":"172.21.15.171:6817/2934826925","state":["exists","up"]},{"osd":6,"uuid":"5e7c3292-9936-4f35-b267-aed9d4e49323","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6824","nonce":436742919},{"type":"v1","addr":"172.21.15.171:6825","nonce":436742919}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6826","nonce":436742919},{"type":"v1","addr":"172.21.15.171:6827","nonce":436742919}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6830","nonce":436742919},{"type":"v1","addr":"172.21.15.171:6831","nonce":436742919}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6828","nonce":436742919},{"type":"v1","addr":"172.21.15.171:6829","nonce":436742919}]},"public_addr":"172.21.15.171:6825/436742919","cluster_addr":"172.21.15.171:6827/436742919","heartbeat_back_addr":"172.21.15.171:6831/436742919","heartbeat_front_addr":"172.21.15.171:6829/436742919","state":["exists","up"]},{"osd":7,"uuid":"339b8414-d7f0-4fb0-a446-efaf635cb7ff","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6809","nonce":2055631175},{"type":"v1","addr":"172.21.15.171:6811","nonce":2055631175}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6813","nonce":2055631175},{"type":"v1","addr":"172.21.15.171:6815","nonce":2055631175}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6822","nonce":2055631175},{"type":"v1","addr":"172.21.15.171:6823","nonce":2055631175}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6818","nonce":2055631175},{"type":"v1","addr":"172.21.15.171:6820","nonce":2055631175}]},"public_addr":"172.21.15.171:6811/2055631175","cluster_addr":"172.21.15.171:6815/2055631175","heartbeat_back_addr":"172.21.15.171:6823/2055631175","heartbeat_front_addr":"172.21.15.171:6820/2055631175","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T14:07:42.668881+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T14:07:42.708397+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T14:07:42.634191+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T14:07:42.733662+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T14:07:42.651541+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T14:07:42.783181+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T14:07:42.766310+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T14:07:42.670797+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-28T14:07:52.657 INFO:teuthology.misc.health.smithi104.stderr:2024-03-28T14:07:52.655+0000 7fe31326c640 1 -- 172.21.15.104:0/4155376830 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7fe2f0067fc0 msgr2=0x7fe2f006a480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:07:52.657 INFO:teuthology.misc.health.smithi104.stderr:2024-03-28T14:07:52.655+0000 7fe31326c640 1 --2- 172.21.15.104:0/4155376830 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7fe2f0067fc0 0x7fe2f006a480 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fe30c06e360 tx=0x7fe308008040 comp rx=0 tx=0).stop 2024-03-28T14:07:52.658 INFO:teuthology.misc.health.smithi104.stderr:2024-03-28T14:07:52.655+0000 7fe31326c640 1 -- 172.21.15.104:0/4155376830 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe30c0ae290 msgr2=0x7fe30c0b7da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:07:52.658 INFO:teuthology.misc.health.smithi104.stderr:2024-03-28T14:07:52.655+0000 7fe31326c640 1 --2- 172.21.15.104:0/4155376830 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe30c0ae290 0x7fe30c0b7da0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fe2fc00c7f0 tx=0x7fe2fc00ccb0 comp rx=0 tx=0).stop 2024-03-28T14:07:52.658 INFO:teuthology.misc.health.smithi104.stderr:2024-03-28T14:07:52.655+0000 7fe31326c640 1 -- 172.21.15.104:0/4155376830 shutdown_connections 2024-03-28T14:07:52.658 INFO:teuthology.misc.health.smithi104.stderr:2024-03-28T14:07:52.655+0000 7fe31326c640 1 --2- 172.21.15.104:0/4155376830 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fe30c147b80 0x7fe30c071440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:07:52.658 INFO:teuthology.misc.health.smithi104.stderr:2024-03-28T14:07:52.655+0000 7fe31326c640 1 --2- 172.21.15.104:0/4155376830 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7fe2f0067fc0 0x7fe2f006a480 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:07:52.658 INFO:teuthology.misc.health.smithi104.stderr:2024-03-28T14:07:52.655+0000 7fe31326c640 1 --2- 172.21.15.104:0/4155376830 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe30c0ae290 0x7fe30c0b7da0 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:07:52.658 INFO:teuthology.misc.health.smithi104.stderr:2024-03-28T14:07:52.655+0000 7fe31326c640 1 --2- 172.21.15.104:0/4155376830 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fe30c06d970 0x7fe30c13b4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:07:52.658 INFO:teuthology.misc.health.smithi104.stderr:2024-03-28T14:07:52.655+0000 7fe31326c640 1 -- 172.21.15.104:0/4155376830 >> 172.21.15.104:0/4155376830 conn(0x7fe30c0b58e0 msgr2=0x7fe30c06ce50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T14:07:52.658 INFO:teuthology.misc.health.smithi104.stderr:2024-03-28T14:07:52.655+0000 7fe31326c640 1 -- 172.21.15.104:0/4155376830 shutdown_connections 2024-03-28T14:07:52.658 INFO:teuthology.misc.health.smithi104.stderr:2024-03-28T14:07:52.655+0000 7fe31326c640 1 -- 172.21.15.104:0/4155376830 wait complete. 2024-03-28T14:07:52.672 DEBUG:teuthology.misc:8 of 8 OSDs are up 2024-03-28T14:07:52.672 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2024-03-28T14:07:52.672 INFO:tasks.ceph.mds.a:Restarting daemon 2024-03-28T14:07:52.672 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-03-28T14:07:52.675 INFO:tasks.ceph.mds.a:Started 2024-03-28T14:07:52.675 INFO:tasks.ceph.mds.b:Restarting daemon 2024-03-28T14:07:52.675 DEBUG:teuthology.orchestra.run.smithi171:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-03-28T14:07:52.677 INFO:tasks.ceph.mds.b:Started 2024-03-28T14:07:52.678 INFO:tasks.ceph:Setting up CephFS filesystem(s)... 2024-03-28T14:07:52.678 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-28T14:07:52.742 INFO:tasks.ceph.mds.a.smithi104.stdout:starting mds.a at 2024-03-28T14:07:52.772 INFO:tasks.ceph.mds.b.smithi171.stdout:starting mds.b at 2024-03-28T14:07:52.794 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:52.791+0000 7f21225fc640 1 Processor -- start 2024-03-28T14:07:52.794 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:52.791+0000 7f21225fc640 1 -- start start 2024-03-28T14:07:52.797 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:52.795+0000 7f21225fc640 1 --2- >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f211c071e00 0x7f211c149400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:07:52.797 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:52.795+0000 7f21225fc640 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f211c0b1f60 0x7f211c14ff70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:07:52.797 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:52.795+0000 7f21225fc640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f211c148c40 con 0x7f211c0b1550 2024-03-28T14:07:52.797 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:52.795+0000 7f21225fc640 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 -- 0x7f211c0721e0 con 0x7f211c0b1f60 2024-03-28T14:07:52.797 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:52.795+0000 7f21225fc640 1 -- --> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] -- mon_getmap magic: 0 -- 0x7f211c0b1a30 con 0x7f211c071e00 2024-03-28T14:07:52.797 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:52.795+0000 7f2120b99640 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f211c0b1f60 0x7f211c14ff70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:07:52.797 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:52.795+0000 7f2120b99640 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f211c0b1f60 0x7f211c14ff70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:49016/0 (socket says 172.21.15.104:49016) 2024-03-28T14:07:52.797 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:52.795+0000 7f2120b99640 1 -- 172.21.15.104:0/3849748682 learned_addr learned my addr 172.21.15.104:0/3849748682 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-03-28T14:07:52.798 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:52.795+0000 7f211bfff640 1 --2- 172.21.15.104:0/3849748682 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f211c071e00 0x7f211c149400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:07:52.798 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:52.795+0000 7f2120b99640 1 -- 172.21.15.104:0/3849748682 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f211c071e00 msgr2=0x7f211c149400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T14:07:52.798 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:52.795+0000 7f2120b99640 1 --2- 172.21.15.104:0/3849748682 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f211c071e00 0x7f211c149400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:07:52.798 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:52.795+0000 7f2120b99640 1 -- 172.21.15.104:0/3849748682 >> v1:172.21.15.171:6789/0 conn(0x7f211c0b1550 legacy=0x7f211c145ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:07:52.798 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:52.795+0000 7f2120b99640 1 -- 172.21.15.104:0/3849748682 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f211c148ee0 con 0x7f211c0b1f60 2024-03-28T14:07:52.798 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:52.795+0000 7f211bfff640 1 --2- 172.21.15.104:0/3849748682 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f211c071e00 0x7f211c149400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T14:07:52.799 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:52.795+0000 7f2120b99640 1 --2- 172.21.15.104:0/3849748682 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f211c0b1f60 0x7f211c14ff70 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f210c009870 tx=0x7f210c01c4d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c33cd9e8931c4f76 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:07:52.799 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:52.795+0000 7f211affd640 1 -- 172.21.15.104:0/3849748682 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f210c0b1070 con 0x7f211c0b1f60 2024-03-28T14:07:52.800 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:52.795+0000 7f211affd640 1 -- 172.21.15.104:0/3849748682 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f210c0a8050 con 0x7f211c0b1f60 2024-03-28T14:07:52.800 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:52.795+0000 7f211affd640 1 -- 172.21.15.104:0/3849748682 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f210c0ac860 con 0x7f211c0b1f60 2024-03-28T14:07:52.800 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:52.795+0000 7f21225fc640 1 -- 172.21.15.104:0/3849748682 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f211c0b1f60 msgr2=0x7f211c14ff70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:07:52.800 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:52.795+0000 7f21225fc640 1 --2- 172.21.15.104:0/3849748682 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f211c0b1f60 0x7f211c14ff70 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f210c009870 tx=0x7f210c01c4d0 comp rx=0 tx=0).stop 2024-03-28T14:07:52.800 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:52.795+0000 7f21225fc640 1 -- 172.21.15.104:0/3849748682 shutdown_connections 2024-03-28T14:07:52.800 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:52.795+0000 7f21225fc640 1 --2- 172.21.15.104:0/3849748682 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f211c0b1f60 0x7f211c14ff70 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:07:52.800 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:52.795+0000 7f21225fc640 1 --2- 172.21.15.104:0/3849748682 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f211c071e00 0x7f211c149400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:07:52.800 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:52.795+0000 7f21225fc640 1 -- 172.21.15.104:0/3849748682 >> 172.21.15.104:0/3849748682 conn(0x7f211c0b7b10 msgr2=0x7f211c0b7f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T14:07:52.800 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:52.795+0000 7f21225fc640 1 -- 172.21.15.104:0/3849748682 shutdown_connections 2024-03-28T14:07:52.800 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:52.795+0000 7f21225fc640 1 -- 172.21.15.104:0/3849748682 wait complete. 2024-03-28T14:07:52.800 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:52.799+0000 7f21225fc640 1 Processor -- start 2024-03-28T14:07:52.800 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:52.799+0000 7f21225fc640 1 -- start start 2024-03-28T14:07:52.803 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:52.799+0000 7f21225fc640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f211c071e00 0x7f211c09bb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:07:52.803 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:52.799+0000 7f21225fc640 1 --2- >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f211c0b1550 0x7f211c1d7ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:07:52.803 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:52.799+0000 7f21225fc640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f211c0b1f60 0x7f211c1d8410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:07:52.803 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:52.799+0000 7f21225fc640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f211c072540 con 0x7f211c0b1f60 2024-03-28T14:07:52.803 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:52.799+0000 7f21225fc640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f211c0726c0 con 0x7f211c071e00 2024-03-28T14:07:52.803 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:52.799+0000 7f21225fc640 1 -- --> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] -- mon_getmap magic: 0 -- 0x7f211c152cd0 con 0x7f211c0b1550 2024-03-28T14:07:52.803 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:52.799+0000 7f2120b99640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f211c0b1f60 0x7f211c1d8410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:07:52.803 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:52.799+0000 7f2120b99640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f211c0b1f60 0x7f211c1d8410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:49030/0 (socket says 172.21.15.104:49030) 2024-03-28T14:07:52.804 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:52.799+0000 7f2120b99640 1 -- 172.21.15.104:0/451171774 learned_addr learned my addr 172.21.15.104:0/451171774 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-03-28T14:07:52.804 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:52.799+0000 7f211bfff640 1 --2- 172.21.15.104:0/451171774 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f211c071e00 0x7f211c09bb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:07:52.804 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:52.799+0000 7f211b7fe640 1 --2- 172.21.15.104:0/451171774 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f211c0b1550 0x7f211c1d7ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:07:52.804 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:52.799+0000 7f2120b99640 1 -- 172.21.15.104:0/451171774 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f211c0b1550 msgr2=0x7f211c1d7ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:07:52.804 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:52.799+0000 7f2120b99640 1 --2- 172.21.15.104:0/451171774 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f211c0b1550 0x7f211c1d7ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:07:52.804 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:52.799+0000 7f2120b99640 1 -- 172.21.15.104:0/451171774 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f211c071e00 msgr2=0x7f211c09bb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:07:52.804 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:52.799+0000 7f2120b99640 1 --2- 172.21.15.104:0/451171774 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f211c071e00 0x7f211c09bb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:07:52.804 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:52.799+0000 7f2120b99640 1 -- 172.21.15.104:0/451171774 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f211c1dee70 con 0x7f211c0b1f60 2024-03-28T14:07:52.804 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:52.799+0000 7f2120b99640 1 --2- 172.21.15.104:0/451171774 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f211c0b1f60 0x7f211c1d8410 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f210c01ab80 tx=0x7f210c01ad10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:07:52.804 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:52.799+0000 7f2118ff9640 1 -- 172.21.15.104:0/451171774 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f210c0b1040 con 0x7f211c0b1f60 2024-03-28T14:07:52.804 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:52.799+0000 7f2118ff9640 1 -- 172.21.15.104:0/451171774 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f210c0bc070 con 0x7f211c0b1f60 2024-03-28T14:07:52.804 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:52.799+0000 7f21225fc640 1 -- 172.21.15.104:0/451171774 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f211c1df150 con 0x7f211c0b1f60 2024-03-28T14:07:52.804 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:52.799+0000 7f2118ff9640 1 -- 172.21.15.104:0/451171774 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f210c0b7690 con 0x7f211c0b1f60 2024-03-28T14:07:52.804 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:52.799+0000 7f21225fc640 1 -- 172.21.15.104:0/451171774 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f211c143330 con 0x7f211c0b1f60 2024-03-28T14:07:52.805 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:52.799+0000 7f21225fc640 1 -- 172.21.15.104:0/451171774 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f211c1df480 con 0x7f211c0b1f60 2024-03-28T14:07:52.805 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:52.799+0000 7f2118ff9640 1 -- 172.21.15.104:0/451171774 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) ==== 87803+0+0 (secure 0 0 0) 0x7f210c0b7830 con 0x7f211c0b1f60 2024-03-28T14:07:52.805 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:52.799+0000 7f2118ff9640 1 --2- 172.21.15.104:0/451171774 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f20fc067fc0 0x7f20fc06a480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:07:52.805 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:52.799+0000 7f2118ff9640 1 -- 172.21.15.104:0/451171774 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(17..17 src has 1..17) ==== 4857+0+0 (secure 0 0 0) 0x7f210c120170 con 0x7f211c0b1f60 2024-03-28T14:07:52.805 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:52.799+0000 7f211bfff640 1 --2- 172.21.15.104:0/451171774 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f20fc067fc0 0x7f20fc06a480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:07:52.805 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:52.803+0000 7f211bfff640 1 --2- 172.21.15.104:0/451171774 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f20fc067fc0 0x7f20fc06a480 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f2114003b90 tx=0x7f2114009910 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:07:52.807 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:52.803+0000 7f2118ff9640 1 -- 172.21.15.104:0/451171774 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179117 (secure 0 0 0) 0x7f210c016150 con 0x7f211c0b1f60 2024-03-28T14:07:52.968 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:52.967+0000 7f21225fc640 1 -- 172.21.15.104:0/451171774 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f211c0687c0 con 0x7f211c0b1f60 2024-03-28T14:07:52.970 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:52.967+0000 7f2118ff9640 1 -- 172.21.15.104:0/451171774 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v17) ==== 74+0+12908 (secure 0 0 0) 0x7f210c0f07d0 con 0x7f211c0b1f60 2024-03-28T14:07:52.970 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-03-28T14:07:52.970 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":17,"fsid":"428a90f2-3351-435f-8fcf-f7e66c78a13e","created":"2024-03-28T14:07:34.862458+0000","modified":"2024-03-28T14:07:50.087852+0000","last_up_change":"2024-03-28T14:07:46.028122+0000","last_in_change":"2024-03-28T14:07:37.157486+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-28T14:07:47.622988+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"1cb70540-4e42-402f-a446-aaec4ca83007","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6800","nonce":3745690918},{"type":"v1","addr":"172.21.15.104:6801","nonce":3745690918}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":3745690918},{"type":"v1","addr":"172.21.15.104:6803","nonce":3745690918}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":3745690918},{"type":"v1","addr":"172.21.15.104:6807","nonce":3745690918}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":3745690918},{"type":"v1","addr":"172.21.15.104:6805","nonce":3745690918}]},"public_addr":"172.21.15.104:6801/3745690918","cluster_addr":"172.21.15.104:6803/3745690918","heartbeat_back_addr":"172.21.15.104:6807/3745690918","heartbeat_front_addr":"172.21.15.104:6805/3745690918","state":["exists","up"]},{"osd":1,"uuid":"3ab7afdd-b980-418c-81f2-004e16de3dca","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":19438674},{"type":"v1","addr":"172.21.15.104:6810","nonce":19438674}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":19438674},{"type":"v1","addr":"172.21.15.104:6814","nonce":19438674}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":19438674},{"type":"v1","addr":"172.21.15.104:6823","nonce":19438674}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":19438674},{"type":"v1","addr":"172.21.15.104:6819","nonce":19438674}]},"public_addr":"172.21.15.104:6810/19438674","cluster_addr":"172.21.15.104:6814/19438674","heartbeat_back_addr":"172.21.15.104:6823/19438674","heartbeat_front_addr":"172.21.15.104:6819/19438674","state":["exists","up"]},{"osd":2,"uuid":"3e2651c8-ecae-4a9e-a053-27240a79eac2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":2352012626},{"type":"v1","addr":"172.21.15.104:6825","nonce":2352012626}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6826","nonce":2352012626},{"type":"v1","addr":"172.21.15.104:6827","nonce":2352012626}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":2352012626},{"type":"v1","addr":"172.21.15.104:6831","nonce":2352012626}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6828","nonce":2352012626},{"type":"v1","addr":"172.21.15.104:6829","nonce":2352012626}]},"public_addr":"172.21.15.104:6825/2352012626","cluster_addr":"172.21.15.104:6827/2352012626","heartbeat_back_addr":"172.21.15.104:6831/2352012626","heartbeat_front_addr":"172.21.15.104:6829/2352012626","state":["exists","up"]},{"osd":3,"uuid":"925c7e0a-38ef-47cd-b288-8bec8456fb17","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6809","nonce":3568347381},{"type":"v1","addr":"172.21.15.104:6811","nonce":3568347381}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6813","nonce":3568347381},{"type":"v1","addr":"172.21.15.104:6815","nonce":3568347381}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":3568347381},{"type":"v1","addr":"172.21.15.104:6822","nonce":3568347381}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6817","nonce":3568347381},{"type":"v1","addr":"172.21.15.104:6818","nonce":3568347381}]},"public_addr":"172.21.15.104:6811/3568347381","cluster_addr":"172.21.15.104:6815/3568347381","heartbeat_back_addr":"172.21.15.104:6822/3568347381","heartbeat_front_addr":"172.21.15.104:6818/3568347381","state":["exists","up"]},{"osd":4,"uuid":"4a8fac86-e595-4a0e-96e3-57edb3cc3f57","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6800","nonce":2107001077},{"type":"v1","addr":"172.21.15.171:6801","nonce":2107001077}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6802","nonce":2107001077},{"type":"v1","addr":"172.21.15.171:6803","nonce":2107001077}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6806","nonce":2107001077},{"type":"v1","addr":"172.21.15.171:6807","nonce":2107001077}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6804","nonce":2107001077},{"type":"v1","addr":"172.21.15.171:6805","nonce":2107001077}]},"public_addr":"172.21.15.171:6801/2107001077","cluster_addr":"172.21.15.171:6803/2107001077","heartbeat_back_addr":"172.21.15.171:6807/2107001077","heartbeat_front_addr":"172.21.15.171:6805/2107001077","state":["exists","up"]},{"osd":5,"uuid":"e48d50b0-beae-40ae-b15a-9a84dbfe88ab","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6808","nonce":2934826925},{"type":"v1","addr":"172.21.15.171:6810","nonce":2934826925}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6812","nonce":2934826925},{"type":"v1","addr":"172.21.15.171:6814","nonce":2934826925}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6819","nonce":2934826925},{"type":"v1","addr":"172.21.15.171:6821","nonce":2934826925}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6816","nonce":2934826925},{"type":"v1","addr":"172.21.15.171:6817","nonce":2934826925}]},"public_addr":"172.21.15.171:6810/2934826925","cluster_addr":"172.21.15.171:6814/2934826925","heartbeat_back_addr":"172.21.15.171:6821/2934826925","heartbeat_front_addr":"172.21.15.171:6817/2934826925","state":["exists","up"]},{"osd":6,"uuid":"5e7c3292-9936-4f35-b267-aed9d4e49323","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6824","nonce":436742919},{"type":"v1","addr":"172.21.15.171:6825","nonce":436742919}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6826","nonce":436742919},{"type":"v1","addr":"172.21.15.171:6827","nonce":436742919}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6830","nonce":436742919},{"type":"v1","addr":"172.21.15.171:6831","nonce":436742919}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6828","nonce":436742919},{"type":"v1","addr":"172.21.15.171:6829","nonce":436742919}]},"public_addr":"172.21.15.171:6825/436742919","cluster_addr":"172.21.15.171:6827/436742919","heartbeat_back_addr":"172.21.15.171:6831/436742919","heartbeat_front_addr":"172.21.15.171:6829/436742919","state":["exists","up"]},{"osd":7,"uuid":"339b8414-d7f0-4fb0-a446-efaf635cb7ff","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6809","nonce":2055631175},{"type":"v1","addr":"172.21.15.171:6811","nonce":2055631175}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6813","nonce":2055631175},{"type":"v1","addr":"172.21.15.171:6815","nonce":2055631175}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6822","nonce":2055631175},{"type":"v1","addr":"172.21.15.171:6823","nonce":2055631175}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6818","nonce":2055631175},{"type":"v1","addr":"172.21.15.171:6820","nonce":2055631175}]},"public_addr":"172.21.15.171:6811/2055631175","cluster_addr":"172.21.15.171:6815/2055631175","heartbeat_back_addr":"172.21.15.171:6823/2055631175","heartbeat_front_addr":"172.21.15.171:6820/2055631175","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T14:07:42.668881+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T14:07:42.708397+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T14:07:42.634191+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T14:07:42.733662+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T14:07:42.651541+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T14:07:42.783181+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T14:07:42.766310+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T14:07:42.670797+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-28T14:07:52.973 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:52.971+0000 7f21225fc640 1 -- 172.21.15.104:0/451171774 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f20fc067fc0 msgr2=0x7f20fc06a480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:07:52.973 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:52.971+0000 7f21225fc640 1 --2- 172.21.15.104:0/451171774 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f20fc067fc0 0x7f20fc06a480 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f2114003b90 tx=0x7f2114009910 comp rx=0 tx=0).stop 2024-03-28T14:07:52.973 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:52.971+0000 7f21225fc640 1 -- 172.21.15.104:0/451171774 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f211c0b1f60 msgr2=0x7f211c1d8410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:07:52.973 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:52.971+0000 7f21225fc640 1 --2- 172.21.15.104:0/451171774 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f211c0b1f60 0x7f211c1d8410 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f210c01ab80 tx=0x7f210c01ad10 comp rx=0 tx=0).stop 2024-03-28T14:07:52.973 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:52.971+0000 7f21225fc640 1 -- 172.21.15.104:0/451171774 shutdown_connections 2024-03-28T14:07:52.973 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:52.971+0000 7f21225fc640 1 --2- 172.21.15.104:0/451171774 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f20fc067fc0 0x7f20fc06a480 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:07:52.973 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:52.971+0000 7f21225fc640 1 --2- 172.21.15.104:0/451171774 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f211c0b1f60 0x7f211c1d8410 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:07:52.973 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:52.971+0000 7f21225fc640 1 --2- 172.21.15.104:0/451171774 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f211c0b1550 0x7f211c1d7ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:07:52.974 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:52.971+0000 7f21225fc640 1 --2- 172.21.15.104:0/451171774 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f211c071e00 0x7f211c09bb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:07:52.974 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:52.971+0000 7f21225fc640 1 -- 172.21.15.104:0/451171774 >> 172.21.15.104:0/451171774 conn(0x7f211c0b7b10 msgr2=0x7f211c072980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T14:07:52.974 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:52.971+0000 7f21225fc640 1 -- 172.21.15.104:0/451171774 shutdown_connections 2024-03-28T14:07:52.974 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:52.971+0000 7f21225fc640 1 -- 172.21.15.104:0/451171774 wait complete. 2024-03-28T14:07:52.992 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-28T14:07:47.622988+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-28T14:07:52.992 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-28T14:07:53.107 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:53.103+0000 7f5fae89f640 1 Processor -- start 2024-03-28T14:07:53.107 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:53.103+0000 7f5fae89f640 1 -- start start 2024-03-28T14:07:53.108 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:53.107+0000 7f5fae89f640 1 --2- >> v2:172.21.15.171:3300/0 conn(0x7f5fa806c8f0 0x7f5fa80b2740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:07:53.108 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:53.107+0000 7f5fae89f640 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f5fa80b3730 0x7f5fa8148c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:07:53.108 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:53.107+0000 7f5fae89f640 1 --2- >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f5fa81491b0 0x7f5fa8149620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:07:53.109 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:53.107+0000 7f5fae89f640 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 -- 0x7f5fa80b0b30 con 0x7f5fa80b3730 2024-03-28T14:07:53.109 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:53.107+0000 7f5fae89f640 1 -- --> v2:172.21.15.171:3300/0 -- mon_getmap magic: 0 -- 0x7f5fa80b22b0 con 0x7f5fa806c8f0 2024-03-28T14:07:53.109 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:53.107+0000 7f5fae89f640 1 -- --> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] -- mon_getmap magic: 0 -- 0x7f5fa80b3b60 con 0x7f5fa81491b0 2024-03-28T14:07:53.109 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:53.107+0000 7f5f9f7fe640 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f5fa80b3730 0x7f5fa8148c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:07:53.109 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:53.107+0000 7f5f9f7fe640 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f5fa80b3730 0x7f5fa8148c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:49042/0 (socket says 172.21.15.104:49042) 2024-03-28T14:07:53.109 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:53.107+0000 7f5f9f7fe640 1 -- 172.21.15.104:0/549469761 learned_addr learned my addr 172.21.15.104:0/549469761 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-03-28T14:07:53.109 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:53.107+0000 7f5fa7fff640 1 --2- 172.21.15.104:0/549469761 >> v2:172.21.15.171:3300/0 conn(0x7f5fa806c8f0 0x7f5fa80b2740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:07:53.109 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:53.107+0000 7f5face3c640 1 --2- 172.21.15.104:0/549469761 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f5fa81491b0 0x7f5fa8149620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:07:53.109 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:53.107+0000 7f5f9f7fe640 1 -- 172.21.15.104:0/549469761 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f5fa81491b0 msgr2=0x7f5fa8149620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T14:07:53.109 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:53.107+0000 7f5f9f7fe640 1 --2- 172.21.15.104:0/549469761 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f5fa81491b0 0x7f5fa8149620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:07:53.109 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:53.107+0000 7f5f9f7fe640 1 -- 172.21.15.104:0/549469761 >> v2:172.21.15.171:3300/0 conn(0x7f5fa806c8f0 msgr2=0x7f5fa80b2740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T14:07:53.109 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:53.107+0000 7f5f9f7fe640 1 --2- 172.21.15.104:0/549469761 >> v2:172.21.15.171:3300/0 conn(0x7f5fa806c8f0 0x7f5fa80b2740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:07:53.109 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:53.107+0000 7f5f9f7fe640 1 -- 172.21.15.104:0/549469761 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5fa806ccd0 con 0x7f5fa80b3730 2024-03-28T14:07:53.109 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:53.107+0000 7f5fa7fff640 1 --2- 172.21.15.104:0/549469761 >> v2:172.21.15.171:3300/0 conn(0x7f5fa806c8f0 0x7f5fa80b2740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T14:07:53.111 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:53.107+0000 7f5f9f7fe640 1 --2- 172.21.15.104:0/549469761 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5fa80b3730 0x7f5fa8148c70 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f5fa806e1f0 tx=0x7f5f9001c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2a088f8607ebbd26 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:07:53.111 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:53.107+0000 7f5fa77fe640 1 -- 172.21.15.104:0/549469761 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f900ab650 con 0x7f5fa80b3730 2024-03-28T14:07:53.112 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:53.107+0000 7f5fa77fe640 1 -- 172.21.15.104:0/549469761 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5f900b0070 con 0x7f5fa80b3730 2024-03-28T14:07:53.112 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:53.107+0000 7f5fa77fe640 1 -- 172.21.15.104:0/549469761 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f900aba10 con 0x7f5fa80b3730 2024-03-28T14:07:53.112 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:53.107+0000 7f5fae89f640 1 -- 172.21.15.104:0/549469761 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5fa80b3730 msgr2=0x7f5fa8148c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:07:53.112 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:53.107+0000 7f5fae89f640 1 --2- 172.21.15.104:0/549469761 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5fa80b3730 0x7f5fa8148c70 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f5fa806e1f0 tx=0x7f5f9001c490 comp rx=0 tx=0).stop 2024-03-28T14:07:53.112 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:53.107+0000 7f5fae89f640 1 -- 172.21.15.104:0/549469761 shutdown_connections 2024-03-28T14:07:53.112 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:53.107+0000 7f5fae89f640 1 --2- 172.21.15.104:0/549469761 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f5fa81491b0 0x7f5fa8149620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:07:53.112 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:53.107+0000 7f5fae89f640 1 --2- 172.21.15.104:0/549469761 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5fa80b3730 0x7f5fa8148c70 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:07:53.112 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:53.107+0000 7f5fae89f640 1 --2- 172.21.15.104:0/549469761 >> v2:172.21.15.171:3300/0 conn(0x7f5fa806c8f0 0x7f5fa80b2740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:07:53.112 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:53.107+0000 7f5fae89f640 1 -- 172.21.15.104:0/549469761 >> 172.21.15.104:0/549469761 conn(0x7f5fa80b8f00 msgr2=0x7f5fa80b9310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T14:07:53.112 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:53.107+0000 7f5fae89f640 1 -- 172.21.15.104:0/549469761 shutdown_connections 2024-03-28T14:07:53.112 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:53.107+0000 7f5fae89f640 1 -- 172.21.15.104:0/549469761 wait complete. 2024-03-28T14:07:53.113 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:53.111+0000 7f5fae89f640 1 Processor -- start 2024-03-28T14:07:53.113 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:53.111+0000 7f5fae89f640 1 -- start start 2024-03-28T14:07:53.115 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:53.111+0000 7f5fae89f640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5fa806c8f0 0x7f5fa814bf10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:07:53.115 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:53.111+0000 7f5fae89f640 1 --2- >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f5fa80b3730 0x7f5fa8141d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:07:53.115 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:53.111+0000 7f5fae89f640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f5fa81491b0 0x7f5fa814b120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:07:53.115 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:53.111+0000 7f5fae89f640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f5fa80b1660 con 0x7f5fa806c8f0 2024-03-28T14:07:53.116 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:53.111+0000 7f5fae89f640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f5fa80b14e0 con 0x7f5fa81491b0 2024-03-28T14:07:53.116 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:53.111+0000 7f5fae89f640 1 -- --> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] -- mon_getmap magic: 0 -- 0x7f5fa806dc70 con 0x7f5fa80b3730 2024-03-28T14:07:53.116 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:53.111+0000 7f5fa7fff640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5fa806c8f0 0x7f5fa814bf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:07:53.116 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:53.111+0000 7f5fa7fff640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5fa806c8f0 0x7f5fa814bf10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:49048/0 (socket says 172.21.15.104:49048) 2024-03-28T14:07:53.116 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:53.111+0000 7f5fa7fff640 1 -- 172.21.15.104:0/2752692873 learned_addr learned my addr 172.21.15.104:0/2752692873 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-03-28T14:07:53.116 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:53.111+0000 7f5f9f7fe640 1 --2- 172.21.15.104:0/2752692873 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f5fa80b3730 0x7f5fa8141d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:07:53.116 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:53.111+0000 7f5fa7fff640 1 -- 172.21.15.104:0/2752692873 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f5fa80b3730 msgr2=0x7f5fa8141d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:07:53.116 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:53.111+0000 7f5fa7fff640 1 --2- 172.21.15.104:0/2752692873 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f5fa80b3730 0x7f5fa8141d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:07:53.116 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:53.111+0000 7f5fa7fff640 1 -- 172.21.15.104:0/2752692873 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f5fa81491b0 msgr2=0x7f5fa814b120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:07:53.116 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:53.111+0000 7f5fa7fff640 1 --2- 172.21.15.104:0/2752692873 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f5fa81491b0 0x7f5fa814b120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:07:53.116 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:53.111+0000 7f5fa7fff640 1 -- 172.21.15.104:0/2752692873 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5fa814a1a0 con 0x7f5fa806c8f0 2024-03-28T14:07:53.117 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:53.111+0000 7f5f9f7fe640 1 --2- 172.21.15.104:0/2752692873 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f5fa80b3730 0x7f5fa8141d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T14:07:53.117 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:53.111+0000 7f5fa7fff640 1 --2- 172.21.15.104:0/2752692873 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5fa806c8f0 0x7f5fa814bf10 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f5f9800c7f0 tx=0x7f5f9800ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:07:53.117 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:53.111+0000 7f5fa57fa640 1 -- 172.21.15.104:0/2752692873 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f98015070 con 0x7f5fa806c8f0 2024-03-28T14:07:53.117 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:53.111+0000 7f5fa57fa640 1 -- 172.21.15.104:0/2752692873 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5f98007820 con 0x7f5fa806c8f0 2024-03-28T14:07:53.117 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:53.111+0000 7f5fa57fa640 1 -- 172.21.15.104:0/2752692873 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f98007c10 con 0x7f5fa806c8f0 2024-03-28T14:07:53.117 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:53.111+0000 7f5fae89f640 1 -- 172.21.15.104:0/2752692873 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5fa814c590 con 0x7f5fa806c8f0 2024-03-28T14:07:53.117 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:53.111+0000 7f5fae89f640 1 -- 172.21.15.104:0/2752692873 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5fa814aca0 con 0x7f5fa806c8f0 2024-03-28T14:07:53.117 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:53.111+0000 7f5fae89f640 1 -- 172.21.15.104:0/2752692873 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5fa80687c0 con 0x7f5fa806c8f0 2024-03-28T14:07:53.117 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:53.111+0000 7f5fa57fa640 1 -- 172.21.15.104:0/2752692873 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) ==== 87803+0+0 (secure 0 0 0) 0x7f5f98018070 con 0x7f5fa806c8f0 2024-03-28T14:07:53.117 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:53.115+0000 7f5fa57fa640 1 --2- 172.21.15.104:0/2752692873 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f5f88067fc0 0x7f5f8806a480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:07:53.117 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:53.115+0000 7f5fa57fa640 1 -- 172.21.15.104:0/2752692873 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(17..17 src has 1..17) ==== 4857+0+0 (secure 0 0 0) 0x7f5f98086270 con 0x7f5fa806c8f0 2024-03-28T14:07:53.117 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:53.115+0000 7f5f9f7fe640 1 --2- 172.21.15.104:0/2752692873 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f5f88067fc0 0x7f5f8806a480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:07:53.118 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:53.115+0000 7f5f9f7fe640 1 --2- 172.21.15.104:0/2752692873 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f5f88067fc0 0x7f5f8806a480 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f5f900af000 tx=0x7f5f90008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:07:53.119 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:53.119+0000 7f5fa57fa640 1 -- 172.21.15.104:0/2752692873 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179117 (secure 0 0 0) 0x7f5fa814a1a0 con 0x7f5fa806c8f0 2024-03-28T14:07:53.269 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:53.267+0000 7f5fae89f640 1 -- 172.21.15.104:0/2752692873 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f5fa8149940 con 0x7f5fa806c8f0 2024-03-28T14:07:53.270 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:53.267+0000 7f5fa57fa640 1 -- 172.21.15.104:0/2752692873 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v17) ==== 93+0+10 (secure 0 0 0) 0x7f5f98056e00 con 0x7f5fa806c8f0 2024-03-28T14:07:53.270 INFO:teuthology.orchestra.run.smithi104.stdout:pg_num: 1 2024-03-28T14:07:53.272 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:53.271+0000 7f5fae89f640 1 -- 172.21.15.104:0/2752692873 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f5f88067fc0 msgr2=0x7f5f8806a480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:07:53.272 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:53.271+0000 7f5fae89f640 1 --2- 172.21.15.104:0/2752692873 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f5f88067fc0 0x7f5f8806a480 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f5f900af000 tx=0x7f5f90008040 comp rx=0 tx=0).stop 2024-03-28T14:07:53.272 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:53.271+0000 7f5fae89f640 1 -- 172.21.15.104:0/2752692873 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5fa806c8f0 msgr2=0x7f5fa814bf10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:07:53.272 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:53.271+0000 7f5fae89f640 1 --2- 172.21.15.104:0/2752692873 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5fa806c8f0 0x7f5fa814bf10 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f5f9800c7f0 tx=0x7f5f9800ccb0 comp rx=0 tx=0).stop 2024-03-28T14:07:53.272 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:53.271+0000 7f5fae89f640 1 -- 172.21.15.104:0/2752692873 shutdown_connections 2024-03-28T14:07:53.272 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:53.271+0000 7f5fae89f640 1 --2- 172.21.15.104:0/2752692873 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f5fa81491b0 0x7f5fa814b120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:07:53.272 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:53.271+0000 7f5fae89f640 1 --2- 172.21.15.104:0/2752692873 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f5fa80b3730 0x7f5fa8141d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:07:53.273 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:53.271+0000 7f5fae89f640 1 --2- 172.21.15.104:0/2752692873 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f5f88067fc0 0x7f5f8806a480 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:07:53.273 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:53.271+0000 7f5fae89f640 1 --2- 172.21.15.104:0/2752692873 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5fa806c8f0 0x7f5fa814bf10 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:07:53.273 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:53.271+0000 7f5fae89f640 1 -- 172.21.15.104:0/2752692873 >> 172.21.15.104:0/2752692873 conn(0x7f5fa80b8f00 msgr2=0x7f5fa8067330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T14:07:53.273 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:53.271+0000 7f5fae89f640 1 -- 172.21.15.104:0/2752692873 shutdown_connections 2024-03-28T14:07:53.273 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:53.271+0000 7f5fae89f640 1 -- 172.21.15.104:0/2752692873 wait complete. 2024-03-28T14:07:53.287 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T14:07:53.399 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:53.399+0000 7f39b25cd640 1 Processor -- start 2024-03-28T14:07:53.400 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:53.399+0000 7f39b25cd640 1 -- start start 2024-03-28T14:07:53.401 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:53.399+0000 7f39b25cd640 1 --2- >> v2:172.21.15.171:3300/0 conn(0x7f39ac06c200 0x7f39ac14b000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:07:53.401 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:53.399+0000 7f39b25cd640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f39ac06bc10 con 0x7f39ac0b4d50 2024-03-28T14:07:53.401 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:53.399+0000 7f39b25cd640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f39ac0b58b0 con 0x7f39ac06b830 2024-03-28T14:07:53.401 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:53.399+0000 7f39b25cd640 1 -- --> v2:172.21.15.171:3300/0 -- mon_getmap magic: 0 -- 0x7f39ac0b5b90 con 0x7f39ac06c200 2024-03-28T14:07:53.401 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:53.399+0000 7f39b0b6a640 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f39ac0b4d50 0x7f39ac152180 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:60282/0 (socket says 172.21.15.104:60282) 2024-03-28T14:07:53.401 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:53.399+0000 7f39b0b6a640 1 -- 172.21.15.104:0/1112751199 learned_addr learned my addr 172.21.15.104:0/1112751199 (peer_addr_for_me v1:172.21.15.104:0/0) 2024-03-28T14:07:53.401 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:53.399+0000 7f39a37fe640 1 --2- 172.21.15.104:0/1112751199 >> v2:172.21.15.171:3300/0 conn(0x7f39ac06c200 0x7f39ac14b000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:07:53.401 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:53.399+0000 7f39ab7fe640 1 -- 172.21.15.104:0/1112751199 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1211976296 0 0) 0x7f39ac06bc10 con 0x7f39ac0b4d50 2024-03-28T14:07:53.401 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:53.399+0000 7f39ab7fe640 1 -- 172.21.15.104:0/1112751199 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3998003530 con 0x7f39ac0b4d50 2024-03-28T14:07:53.403 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:53.399+0000 7f39ab7fe640 1 -- 172.21.15.104:0/1112751199 <== mon.0 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2885189758 0 0) 0x7f3998003530 con 0x7f39ac0b4d50 2024-03-28T14:07:53.403 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:53.399+0000 7f39ab7fe640 1 -- 172.21.15.104:0/1112751199 >> v2:172.21.15.171:3300/0 conn(0x7f39ac06c200 msgr2=0x7f39ac14b000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T14:07:53.403 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:53.399+0000 7f39ab7fe640 1 --2- 172.21.15.104:0/1112751199 >> v2:172.21.15.171:3300/0 conn(0x7f39ac06c200 0x7f39ac14b000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:07:53.403 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:53.399+0000 7f39ab7fe640 1 -- 172.21.15.104:0/1112751199 >> v1:172.21.15.171:6789/0 conn(0x7f39ac06b830 legacy=0x7f39ac1418a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:07:53.403 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:53.399+0000 7f39ab7fe640 1 -- 172.21.15.104:0/1112751199 --> v1:172.21.15.104:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f39ac0b5e70 con 0x7f39ac0b4d50 2024-03-28T14:07:53.403 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:53.399+0000 7f39ab7fe640 1 -- 172.21.15.104:0/1112751199 <== mon.0 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3745580849 0 0) 0x7f39a4001a00 con 0x7f39ac0b4d50 2024-03-28T14:07:53.403 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:53.399+0000 7f39ab7fe640 1 -- 172.21.15.104:0/1112751199 >> v1:172.21.15.104:6789/0 conn(0x7f39ac0b4d50 legacy=0x7f39ac152180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:07:53.404 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:53.399+0000 7f39ab7fe640 1 --2- 172.21.15.104:0/1112751199 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f39980039c0 0x7f3998003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:07:53.404 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:53.399+0000 7f39ab7fe640 1 --2- 172.21.15.104:0/1112751199 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3998004810 0x7f3998004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:07:53.404 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:53.399+0000 7f39ab7fe640 1 --2- 172.21.15.104:0/1112751199 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f39980051d0 0x7f399800b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:07:53.404 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:53.399+0000 7f39ab7fe640 1 -- 172.21.15.104:0/1112751199 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f39ac0b5b90 con 0x7f3998004810 2024-03-28T14:07:53.404 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:53.399+0000 7f39ab7fe640 1 -- 172.21.15.104:0/1112751199 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f399800c170 con 0x7f39980039c0 2024-03-28T14:07:53.404 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:53.399+0000 7f39ab7fe640 1 -- 172.21.15.104:0/1112751199 --> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] -- mon_getmap magic: 0 -- 0x7f399800c370 con 0x7f39980051d0 2024-03-28T14:07:53.404 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:53.399+0000 7f39a37fe640 1 --2- 172.21.15.104:0/1112751199 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3998004810 0x7f3998004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:07:53.404 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:53.399+0000 7f39abfff640 1 --2- 172.21.15.104:0/1112751199 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f39980039c0 0x7f3998003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:07:53.404 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:53.399+0000 7f39a37fe640 1 -- 172.21.15.104:0/1112751199 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f39980051d0 msgr2=0x7f399800b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T14:07:53.404 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:53.399+0000 7f39a37fe640 1 --2- 172.21.15.104:0/1112751199 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f39980051d0 0x7f399800b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:07:53.404 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:53.399+0000 7f39a37fe640 1 -- 172.21.15.104:0/1112751199 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f39980039c0 msgr2=0x7f3998003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T14:07:53.404 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:53.399+0000 7f39a37fe640 1 --2- 172.21.15.104:0/1112751199 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f39980039c0 0x7f3998003de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:07:53.404 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:53.399+0000 7f39a37fe640 1 -- 172.21.15.104:0/1112751199 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3998003530 con 0x7f3998004810 2024-03-28T14:07:53.404 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:53.399+0000 7f39b0b6a640 1 -- 172.21.15.104:0/1112751199 reap_dead start 2024-03-28T14:07:53.406 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:53.403+0000 7f39a37fe640 1 --2- 172.21.15.104:0/1112751199 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3998004810 0x7f3998004c90 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f3994009ae0 tx=0x7f399401ca40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3a4c557e69d1312a server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:07:53.406 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:53.403+0000 7f39ab7fe640 1 -- 172.21.15.104:0/1112751199 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39940b2070 con 0x7f3998004810 2024-03-28T14:07:53.406 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:53.403+0000 7f39ab7fe640 1 -- 172.21.15.104:0/1112751199 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f39940a5630 con 0x7f3998004810 2024-03-28T14:07:53.406 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:53.403+0000 7f39ab7fe640 1 -- 172.21.15.104:0/1112751199 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39940096b0 con 0x7f3998004810 2024-03-28T14:07:53.406 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:53.403+0000 7f39b25cd640 1 -- 172.21.15.104:0/1112751199 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3998004810 msgr2=0x7f3998004c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:07:53.406 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:53.403+0000 7f39b25cd640 1 --2- 172.21.15.104:0/1112751199 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3998004810 0x7f3998004c90 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f3994009ae0 tx=0x7f399401ca40 comp rx=0 tx=0).stop 2024-03-28T14:07:53.406 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:53.403+0000 7f39b25cd640 1 -- 172.21.15.104:0/1112751199 shutdown_connections 2024-03-28T14:07:53.406 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:53.403+0000 7f39b25cd640 1 --2- 172.21.15.104:0/1112751199 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3998004810 0x7f3998004c90 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:07:53.407 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:53.403+0000 7f39b25cd640 1 -- 172.21.15.104:0/1112751199 >> 172.21.15.104:0/1112751199 conn(0x7f39ac06c6e0 msgr2=0x7f39ac06caf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T14:07:53.407 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:53.403+0000 7f39b25cd640 1 -- 172.21.15.104:0/1112751199 shutdown_connections 2024-03-28T14:07:53.407 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:53.403+0000 7f39b25cd640 1 -- 172.21.15.104:0/1112751199 wait complete. 2024-03-28T14:07:53.407 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:53.403+0000 7f39b25cd640 1 Processor -- start 2024-03-28T14:07:53.407 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:53.403+0000 7f39b25cd640 1 -- start start 2024-03-28T14:07:53.409 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:53.403+0000 7f39b25cd640 1 --2- >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f3998004810 0x7f39ac149720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:07:53.409 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:53.403+0000 7f39b25cd640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f39ac14a770 0x7f39ac13b0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:07:53.409 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:53.403+0000 7f39b25cd640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f39ac13b600 0x7f39ac1e0790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:07:53.409 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:53.407+0000 7f39b25cd640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f39ac06d810 con 0x7f39ac13b600 2024-03-28T14:07:53.409 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:53.407+0000 7f39b25cd640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f39ac14ba20 con 0x7f39ac14a770 2024-03-28T14:07:53.409 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:53.407+0000 7f39b25cd640 1 -- --> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] -- mon_getmap magic: 0 -- 0x7f39ac06d990 con 0x7f3998004810 2024-03-28T14:07:53.409 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:53.407+0000 7f39b0b6a640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f39ac13b600 0x7f39ac1e0790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:07:53.409 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:53.407+0000 7f39b0b6a640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f39ac13b600 0x7f39ac1e0790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:49062/0 (socket says 172.21.15.104:49062) 2024-03-28T14:07:53.409 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:53.407+0000 7f39b0b6a640 1 -- 172.21.15.104:0/4100028833 learned_addr learned my addr 172.21.15.104:0/4100028833 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-03-28T14:07:53.410 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:53.407+0000 7f39a37fe640 1 --2- 172.21.15.104:0/4100028833 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f39ac14a770 0x7f39ac13b0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:07:53.410 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:53.407+0000 7f39abfff640 1 --2- 172.21.15.104:0/4100028833 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f3998004810 0x7f39ac149720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:07:53.410 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:53.407+0000 7f39b0b6a640 1 -- 172.21.15.104:0/4100028833 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f3998004810 msgr2=0x7f39ac149720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:07:53.410 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:53.407+0000 7f39b0b6a640 1 --2- 172.21.15.104:0/4100028833 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f3998004810 0x7f39ac149720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:07:53.410 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:53.407+0000 7f39b0b6a640 1 -- 172.21.15.104:0/4100028833 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f39ac14a770 msgr2=0x7f39ac13b0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:07:53.410 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:53.407+0000 7f39b0b6a640 1 --2- 172.21.15.104:0/4100028833 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f39ac14a770 0x7f39ac13b0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:07:53.410 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:53.407+0000 7f39b0b6a640 1 -- 172.21.15.104:0/4100028833 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f39ac1455e0 con 0x7f39ac13b600 2024-03-28T14:07:53.410 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:53.407+0000 7f39a37fe640 1 --2- 172.21.15.104:0/4100028833 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f39ac14a770 0x7f39ac13b0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T14:07:53.410 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:53.407+0000 7f39b0b6a640 1 --2- 172.21.15.104:0/4100028833 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f39ac13b600 0x7f39ac1e0790 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f39ac06d740 tx=0x7f39a400ce80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:07:53.410 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:53.407+0000 7f39a97fa640 1 -- 172.21.15.104:0/4100028833 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39a400a5e0 con 0x7f39ac13b600 2024-03-28T14:07:53.410 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:53.407+0000 7f39a97fa640 1 -- 172.21.15.104:0/4100028833 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f39a400a780 con 0x7f39ac13b600 2024-03-28T14:07:53.410 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:53.407+0000 7f39b25cd640 1 -- 172.21.15.104:0/4100028833 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f39ac1458c0 con 0x7f39ac13b600 2024-03-28T14:07:53.410 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:53.407+0000 7f39a97fa640 1 -- 172.21.15.104:0/4100028833 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39a400ab70 con 0x7f39ac13b600 2024-03-28T14:07:53.411 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:53.407+0000 7f39b25cd640 1 -- 172.21.15.104:0/4100028833 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f39ac148b60 con 0x7f39ac13b600 2024-03-28T14:07:53.411 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:53.407+0000 7f39b25cd640 1 -- 172.21.15.104:0/4100028833 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f39ac0687c0 con 0x7f39ac13b600 2024-03-28T14:07:53.411 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:53.407+0000 7f39a97fa640 1 -- 172.21.15.104:0/4100028833 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) ==== 87803+0+0 (secure 0 0 0) 0x7f39a400ad10 con 0x7f39ac13b600 2024-03-28T14:07:53.411 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:53.407+0000 7f39a97fa640 1 --2- 172.21.15.104:0/4100028833 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f398c067ff0 0x7f398c06a4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:07:53.412 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:53.407+0000 7f39a97fa640 1 -- 172.21.15.104:0/4100028833 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(17..17 src has 1..17) ==== 4857+0+0 (secure 0 0 0) 0x7f39a4085a80 con 0x7f39ac13b600 2024-03-28T14:07:53.412 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:53.407+0000 7f39abfff640 1 --2- 172.21.15.104:0/4100028833 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f398c067ff0 0x7f398c06a4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:07:53.412 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:53.407+0000 7f39abfff640 1 --2- 172.21.15.104:0/4100028833 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f398c067ff0 0x7f398c06a4b0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f399c000c10 tx=0x7f399c009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:07:53.414 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:53.411+0000 7f39a97fa640 1 -- 172.21.15.104:0/4100028833 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179117 (secure 0 0 0) 0x7f39a4056610 con 0x7f39ac13b600 2024-03-28T14:07:53.608 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:53.607+0000 7f39b25cd640 1 -- 172.21.15.104:0/4100028833 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f39ac06bc10 con 0x7f39ac13b600 2024-03-28T14:07:53.608 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:53.607+0000 7f39a97fa640 1 -- 172.21.15.104:0/4100028833 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 2 v2) ==== 93+0+1395 (secure 0 0 0) 0x7f39a4055f10 con 0x7f39ac13b600 2024-03-28T14:07:53.608 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-03-28T14:07:53.609 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":2,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4245,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6833/3184673325","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":3184673325},{"type":"v1","addr":"172.21.15.104:6833","nonce":3184673325}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[]} 2024-03-28T14:07:53.609 INFO:teuthology.orchestra.run.smithi104.stderr:dumped fsmap epoch 2 2024-03-28T14:07:53.611 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:53.607+0000 7f39b25cd640 1 -- 172.21.15.104:0/4100028833 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f398c067ff0 msgr2=0x7f398c06a4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:07:53.611 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:53.607+0000 7f39b25cd640 1 --2- 172.21.15.104:0/4100028833 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f398c067ff0 0x7f398c06a4b0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f399c000c10 tx=0x7f399c009040 comp rx=0 tx=0).stop 2024-03-28T14:07:53.611 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:53.607+0000 7f39b25cd640 1 -- 172.21.15.104:0/4100028833 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f39ac13b600 msgr2=0x7f39ac1e0790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:07:53.611 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:53.607+0000 7f39b25cd640 1 --2- 172.21.15.104:0/4100028833 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f39ac13b600 0x7f39ac1e0790 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f39ac06d740 tx=0x7f39a400ce80 comp rx=0 tx=0).stop 2024-03-28T14:07:53.611 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:53.607+0000 7f39b25cd640 1 -- 172.21.15.104:0/4100028833 shutdown_connections 2024-03-28T14:07:53.611 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:53.607+0000 7f39b25cd640 1 --2- 172.21.15.104:0/4100028833 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f398c067ff0 0x7f398c06a4b0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:07:53.611 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:53.607+0000 7f39b25cd640 1 --2- 172.21.15.104:0/4100028833 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f39ac13b600 0x7f39ac1e0790 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:07:53.611 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:53.607+0000 7f39b25cd640 1 --2- 172.21.15.104:0/4100028833 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f39ac14a770 0x7f39ac13b0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:07:53.611 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:53.607+0000 7f39b25cd640 1 --2- 172.21.15.104:0/4100028833 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f3998004810 0x7f39ac149720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:07:53.611 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:53.607+0000 7f39b25cd640 1 -- 172.21.15.104:0/4100028833 >> 172.21.15.104:0/4100028833 conn(0x7f39ac06c6e0 msgr2=0x7f39ac13d8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T14:07:53.611 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:53.607+0000 7f39b25cd640 1 -- 172.21.15.104:0/4100028833 shutdown_connections 2024-03-28T14:07:53.611 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:53.607+0000 7f39b25cd640 1 -- 172.21.15.104:0/4100028833 wait complete. 2024-03-28T14:07:55.626 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T14:07:55.755 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:55.755+0000 7f83a88e5640 1 Processor -- start 2024-03-28T14:07:55.756 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:55.755+0000 7f83a88e5640 1 -- start start 2024-03-28T14:07:55.757 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:55.755+0000 7f83a88e5640 1 --2- >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f83a406ef50 0x7f83a4148b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:07:55.757 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:55.755+0000 7f83a88e5640 1 --2- >> v2:172.21.15.171:3300/0 conn(0x7f83a413fad0 0x7f83a4145640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:07:55.757 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:55.755+0000 7f83a88e5640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f83a4071de0 con 0x7f83a406f330 2024-03-28T14:07:55.757 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:55.755+0000 7f83a88e5640 1 -- --> v2:172.21.15.171:3300/0 -- mon_getmap magic: 0 -- 0x7f83a4149120 con 0x7f83a413fad0 2024-03-28T14:07:55.757 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:55.755+0000 7f83a88e5640 1 -- --> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] -- mon_getmap magic: 0 -- 0x7f83a4149400 con 0x7f83a406ef50 2024-03-28T14:07:55.757 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:55.755+0000 7f83a2d9d640 1 --2- >> v2:172.21.15.171:3300/0 conn(0x7f83a413fad0 0x7f83a4145640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:07:55.757 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:55.755+0000 7f83a2d9d640 1 --2- >> v2:172.21.15.171:3300/0 conn(0x7f83a413fad0 0x7f83a4145640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.171:3300/0 says I am v2:172.21.15.104:51266/0 (socket says 172.21.15.104:51266) 2024-03-28T14:07:55.757 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:55.755+0000 7f83a2d9d640 1 -- 172.21.15.104:0/3622992901 learned_addr learned my addr 172.21.15.104:0/3622992901 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-03-28T14:07:55.757 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:55.755+0000 7f83a259c640 1 --2- 172.21.15.104:0/3622992901 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f83a406ef50 0x7f83a4148b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:07:55.758 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:55.755+0000 7f83a2d9d640 1 -- 172.21.15.104:0/3622992901 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f83a406ef50 msgr2=0x7f83a4148b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T14:07:55.758 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:55.755+0000 7f83a2d9d640 1 --2- 172.21.15.104:0/3622992901 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f83a406ef50 0x7f83a4148b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:07:55.758 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:55.755+0000 7f83a2d9d640 1 -- 172.21.15.104:0/3622992901 >> v1:172.21.15.171:6789/0 conn(0x7f83a406f330 legacy=0x7f83a413f3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:07:55.758 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:55.755+0000 7f83a2d9d640 1 -- 172.21.15.104:0/3622992901 --> v2:172.21.15.171:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f83a41496e0 con 0x7f83a413fad0 2024-03-28T14:07:55.759 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:55.759+0000 7f83a2d9d640 1 --2- 172.21.15.104:0/3622992901 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f83a413fad0 0x7f83a4145640 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f839c009e50 tx=0x7f839c01ca30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=21f571b9ea257240 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:07:55.760 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:55.759+0000 7f83a159a640 1 -- 172.21.15.104:0/3622992901 <== mon.1 v2:172.21.15.171:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f839c0a57d0 con 0x7f83a413fad0 2024-03-28T14:07:55.760 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:55.759+0000 7f83a159a640 1 -- 172.21.15.104:0/3622992901 <== mon.1 v2:172.21.15.171:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f839c0a5970 con 0x7f83a413fad0 2024-03-28T14:07:55.760 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:55.759+0000 7f83a159a640 1 -- 172.21.15.104:0/3622992901 <== mon.1 v2:172.21.15.171:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f839c0a5d40 con 0x7f83a413fad0 2024-03-28T14:07:55.761 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:55.759+0000 7f83a88e5640 1 -- 172.21.15.104:0/3622992901 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f83a413fad0 msgr2=0x7f83a4145640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:07:55.761 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:55.759+0000 7f83a88e5640 1 --2- 172.21.15.104:0/3622992901 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f83a413fad0 0x7f83a4145640 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f839c009e50 tx=0x7f839c01ca30 comp rx=0 tx=0).stop 2024-03-28T14:07:55.761 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:55.759+0000 7f83a88e5640 1 -- 172.21.15.104:0/3622992901 shutdown_connections 2024-03-28T14:07:55.761 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:55.759+0000 7f83a88e5640 1 --2- 172.21.15.104:0/3622992901 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f83a413fad0 0x7f83a4145640 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:07:55.761 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:55.759+0000 7f83a88e5640 1 --2- 172.21.15.104:0/3622992901 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f83a406ef50 0x7f83a4148b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:07:55.761 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:55.759+0000 7f83a88e5640 1 -- 172.21.15.104:0/3622992901 >> 172.21.15.104:0/3622992901 conn(0x7f83a40b7d90 msgr2=0x7f83a40b81a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T14:07:55.761 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:55.759+0000 7f83a88e5640 1 -- 172.21.15.104:0/3622992901 shutdown_connections 2024-03-28T14:07:55.761 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:55.759+0000 7f83a88e5640 1 -- 172.21.15.104:0/3622992901 wait complete. 2024-03-28T14:07:55.762 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:55.759+0000 7f83a88e5640 1 Processor -- start 2024-03-28T14:07:55.762 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:55.759+0000 7f83a88e5640 1 -- start start 2024-03-28T14:07:55.764 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:55.759+0000 7f83a88e5640 1 --2- >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f83a406ef50 0x7f83a413c450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:07:55.764 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:55.759+0000 7f83a88e5640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f83a406f330 0x7f83a4146730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:07:55.764 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:55.759+0000 7f83a88e5640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f83a413fad0 0x7f83a40b2970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:07:55.764 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:55.759+0000 7f83a88e5640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f83a4152a20 con 0x7f83a406f330 2024-03-28T14:07:55.764 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:55.759+0000 7f83a88e5640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f83a41528a0 con 0x7f83a413fad0 2024-03-28T14:07:55.764 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:55.759+0000 7f83a88e5640 1 -- --> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] -- mon_getmap magic: 0 -- 0x7f83a4152ba0 con 0x7f83a406ef50 2024-03-28T14:07:55.764 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:55.759+0000 7f83a1d9b640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f83a406f330 0x7f83a4146730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:07:55.764 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:55.759+0000 7f83a1d9b640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f83a406f330 0x7f83a4146730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:49070/0 (socket says 172.21.15.104:49070) 2024-03-28T14:07:55.764 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:55.759+0000 7f83a1d9b640 1 -- 172.21.15.104:0/3514279357 learned_addr learned my addr 172.21.15.104:0/3514279357 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-03-28T14:07:55.764 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:55.759+0000 7f83a259c640 1 --2- 172.21.15.104:0/3514279357 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f83a406ef50 0x7f83a413c450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:07:55.764 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:55.759+0000 7f83a2d9d640 1 --2- 172.21.15.104:0/3514279357 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f83a413fad0 0x7f83a40b2970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:07:55.764 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:55.759+0000 7f83a1d9b640 1 -- 172.21.15.104:0/3514279357 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f83a406ef50 msgr2=0x7f83a413c450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:07:55.765 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:55.759+0000 7f83a1d9b640 1 --2- 172.21.15.104:0/3514279357 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f83a406ef50 0x7f83a413c450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:07:55.765 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:55.763+0000 7f83a1d9b640 1 -- 172.21.15.104:0/3514279357 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f83a413fad0 msgr2=0x7f83a40b2970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:07:55.765 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:55.763+0000 7f83a1d9b640 1 --2- 172.21.15.104:0/3514279357 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f83a413fad0 0x7f83a40b2970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:07:55.765 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:55.763+0000 7f83a1d9b640 1 -- 172.21.15.104:0/3514279357 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f83a41de0d0 con 0x7f83a406f330 2024-03-28T14:07:55.765 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:55.763+0000 7f83a2d9d640 1 --2- 172.21.15.104:0/3514279357 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f83a413fad0 0x7f83a40b2970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T14:07:55.765 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:55.763+0000 7f83a1d9b640 1 --2- 172.21.15.104:0/3514279357 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f83a406f330 0x7f83a4146730 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f839800c530 tx=0x7f839800c9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:07:55.765 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:55.763+0000 7f8392ffd640 1 -- 172.21.15.104:0/3514279357 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8398016020 con 0x7f83a406f330 2024-03-28T14:07:55.765 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:55.763+0000 7f8392ffd640 1 -- 172.21.15.104:0/3514279357 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8398005220 con 0x7f83a406f330 2024-03-28T14:07:55.765 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:55.763+0000 7f83a88e5640 1 -- 172.21.15.104:0/3514279357 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f83a41de3b0 con 0x7f83a406f330 2024-03-28T14:07:55.765 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:55.763+0000 7f8392ffd640 1 -- 172.21.15.104:0/3514279357 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8398007030 con 0x7f83a406f330 2024-03-28T14:07:55.765 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:55.763+0000 7f83a88e5640 1 -- 172.21.15.104:0/3514279357 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f83a4146c70 con 0x7f83a406f330 2024-03-28T14:07:55.766 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:55.763+0000 7f83a88e5640 1 -- 172.21.15.104:0/3514279357 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f83a40687c0 con 0x7f83a406f330 2024-03-28T14:07:55.766 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:55.763+0000 7f8392ffd640 1 -- 172.21.15.104:0/3514279357 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) ==== 87803+0+0 (secure 0 0 0) 0x7f83980071d0 con 0x7f83a406f330 2024-03-28T14:07:55.766 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:55.763+0000 7f8392ffd640 1 --2- 172.21.15.104:0/3514279357 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f8384067ff0 0x7f838406a4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:07:55.766 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:55.763+0000 7f8392ffd640 1 -- 172.21.15.104:0/3514279357 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(17..17 src has 1..17) ==== 4857+0+0 (secure 0 0 0) 0x7f8398085b40 con 0x7f83a406f330 2024-03-28T14:07:55.767 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:55.763+0000 7f83a259c640 1 --2- 172.21.15.104:0/3514279357 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f8384067ff0 0x7f838406a4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:07:55.767 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:55.763+0000 7f83a259c640 1 --2- 172.21.15.104:0/3514279357 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f8384067ff0 0x7f838406a4b0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f83a41480b0 tx=0x7f8394008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:07:55.770 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:55.767+0000 7f8392ffd640 1 -- 172.21.15.104:0/3514279357 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179117 (secure 0 0 0) 0x7f8398020020 con 0x7f83a406f330 2024-03-28T14:07:55.960 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:55.959+0000 7f83a88e5640 1 -- 172.21.15.104:0/3514279357 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f83a413d180 con 0x7f83a406f330 2024-03-28T14:07:55.960 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:55.959+0000 7f8392ffd640 1 -- 172.21.15.104:0/3514279357 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 3 v3) ==== 93+0+2222 (secure 0 0 0) 0x7f83980074c0 con 0x7f83a406f330 2024-03-28T14:07:55.961 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-03-28T14:07:55.961 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":3,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4229,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.171:6835/1775867467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6834","nonce":1775867467},{"type":"v1","addr":"172.21.15.171:6835","nonce":1775867467}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4245,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6833/3184673325","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":3184673325},{"type":"v1","addr":"172.21.15.104:6833","nonce":3184673325}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[]} 2024-03-28T14:07:55.961 INFO:teuthology.orchestra.run.smithi104.stderr:dumped fsmap epoch 3 2024-03-28T14:07:55.963 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:55.959+0000 7f83a88e5640 1 -- 172.21.15.104:0/3514279357 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f8384067ff0 msgr2=0x7f838406a4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:07:55.963 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:55.959+0000 7f83a88e5640 1 --2- 172.21.15.104:0/3514279357 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f8384067ff0 0x7f838406a4b0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f83a41480b0 tx=0x7f8394008040 comp rx=0 tx=0).stop 2024-03-28T14:07:55.963 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:55.959+0000 7f83a88e5640 1 -- 172.21.15.104:0/3514279357 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f83a406f330 msgr2=0x7f83a4146730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:07:55.963 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:55.959+0000 7f83a88e5640 1 --2- 172.21.15.104:0/3514279357 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f83a406f330 0x7f83a4146730 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f839800c530 tx=0x7f839800c9f0 comp rx=0 tx=0).stop 2024-03-28T14:07:55.964 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:55.959+0000 7f83a88e5640 1 -- 172.21.15.104:0/3514279357 shutdown_connections 2024-03-28T14:07:55.964 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:55.959+0000 7f83a88e5640 1 --2- 172.21.15.104:0/3514279357 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f83a413fad0 0x7f83a40b2970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:07:55.964 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:55.959+0000 7f83a88e5640 1 --2- 172.21.15.104:0/3514279357 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f8384067ff0 0x7f838406a4b0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:07:55.964 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:55.959+0000 7f83a88e5640 1 --2- 172.21.15.104:0/3514279357 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f83a406f330 0x7f83a4146730 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:07:55.964 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:55.959+0000 7f83a88e5640 1 --2- 172.21.15.104:0/3514279357 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f83a406ef50 0x7f83a413c450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:07:55.964 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:55.959+0000 7f83a88e5640 1 -- 172.21.15.104:0/3514279357 >> 172.21.15.104:0/3514279357 conn(0x7f83a40b7d90 msgr2=0x7f83a40b8b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T14:07:55.964 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:55.959+0000 7f83a88e5640 1 -- 172.21.15.104:0/3514279357 shutdown_connections 2024-03-28T14:07:55.964 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:55.959+0000 7f83a88e5640 1 -- 172.21.15.104:0/3514279357 wait complete. 2024-03-28T14:07:55.979 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-28T14:07:56.086 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:56.083+0000 7f16a4e87640 1 Processor -- start 2024-03-28T14:07:56.086 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:56.083+0000 7f16a4e87640 1 -- start start 2024-03-28T14:07:56.087 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:56.083+0000 7f16a4e87640 1 --2- >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f16a0147d90 0x7f16a006f210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:07:56.087 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:56.083+0000 7f16a4e87640 1 --2- >> v2:172.21.15.171:3300/0 conn(0x7f16a00b0310 0x7f16a0072550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:07:56.087 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:56.083+0000 7f16a4e87640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f16a0148940 con 0x7f16a0148170 2024-03-28T14:07:56.087 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:56.083+0000 7f16a4e87640 1 -- --> v2:172.21.15.171:3300/0 -- mon_getmap magic: 0 -- 0x7f16a00af400 con 0x7f16a00b0310 2024-03-28T14:07:56.087 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:56.083+0000 7f16a4e87640 1 -- --> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] -- mon_getmap magic: 0 -- 0x7f16a00af6e0 con 0x7f16a0147d90 2024-03-28T14:07:56.087 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:56.083+0000 7f169e59c640 1 --2- >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f16a0147d90 0x7f16a006f210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:07:56.087 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:56.083+0000 7f169ed9d640 1 --2- >> v2:172.21.15.171:3300/0 conn(0x7f16a00b0310 0x7f16a0072550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:07:56.087 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:56.087+0000 7f169e59c640 1 --2- >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f16a0147d90 0x7f16a006f210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.171:3301/0 says I am v2:172.21.15.104:60216/0 (socket says 172.21.15.104:60216) 2024-03-28T14:07:56.088 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:56.087+0000 7f169e59c640 1 -- 172.21.15.104:0/2220249707 learned_addr learned my addr 172.21.15.104:0/2220249707 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-03-28T14:07:56.088 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:56.087+0000 7f169e59c640 1 -- 172.21.15.104:0/2220249707 >> v2:172.21.15.171:3300/0 conn(0x7f16a00b0310 msgr2=0x7f16a0072550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T14:07:56.088 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:56.087+0000 7f169e59c640 1 --2- 172.21.15.104:0/2220249707 >> v2:172.21.15.171:3300/0 conn(0x7f16a00b0310 0x7f16a0072550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:07:56.088 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:56.087+0000 7f169e59c640 1 -- 172.21.15.104:0/2220249707 >> v1:172.21.15.171:6789/0 conn(0x7f16a0148170 legacy=0x7f16a00afc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:07:56.088 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:56.087+0000 7f169e59c640 1 -- 172.21.15.104:0/2220249707 --> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f16a01485d0 con 0x7f16a0147d90 2024-03-28T14:07:56.088 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:56.087+0000 7f169ed9d640 1 --2- 172.21.15.104:0/2220249707 >> v2:172.21.15.171:3300/0 conn(0x7f16a00b0310 0x7f16a0072550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T14:07:56.089 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:56.087+0000 7f169e59c640 1 --2- 172.21.15.104:0/2220249707 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f16a0147d90 0x7f16a006f210 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f1690009870 tx=0x7f169001c4d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=251b7bfa4a636da0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:07:56.090 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:56.087+0000 7f169dd9b640 1 -- 172.21.15.104:0/2220249707 <== mon.2 v2:172.21.15.171:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16900aa650 con 0x7f16a0147d90 2024-03-28T14:07:56.090 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:56.087+0000 7f169dd9b640 1 -- 172.21.15.104:0/2220249707 <== mon.2 v2:172.21.15.171:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f16900af070 con 0x7f16a0147d90 2024-03-28T14:07:56.090 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:56.087+0000 7f169dd9b640 1 -- 172.21.15.104:0/2220249707 <== mon.2 v2:172.21.15.171:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16900aaa10 con 0x7f16a0147d90 2024-03-28T14:07:56.091 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:56.087+0000 7f16a4e87640 1 -- 172.21.15.104:0/2220249707 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f16a0147d90 msgr2=0x7f16a006f210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:07:56.091 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:56.087+0000 7f16a4e87640 1 --2- 172.21.15.104:0/2220249707 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f16a0147d90 0x7f16a006f210 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f1690009870 tx=0x7f169001c4d0 comp rx=0 tx=0).stop 2024-03-28T14:07:56.091 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:56.087+0000 7f16a4e87640 1 -- 172.21.15.104:0/2220249707 shutdown_connections 2024-03-28T14:07:56.091 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:56.087+0000 7f16a4e87640 1 --2- 172.21.15.104:0/2220249707 >> v2:172.21.15.171:3300/0 conn(0x7f16a00b0310 0x7f16a0072550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:07:56.091 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:56.087+0000 7f16a4e87640 1 --2- 172.21.15.104:0/2220249707 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f16a0147d90 0x7f16a006f210 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:07:56.091 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:56.087+0000 7f16a4e87640 1 -- 172.21.15.104:0/2220249707 >> 172.21.15.104:0/2220249707 conn(0x7f16a00b7d90 msgr2=0x7f16a00b81a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T14:07:56.091 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:56.087+0000 7f16a4e87640 1 -- 172.21.15.104:0/2220249707 shutdown_connections 2024-03-28T14:07:56.091 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:56.087+0000 7f16a4e87640 1 -- 172.21.15.104:0/2220249707 wait complete. 2024-03-28T14:07:56.093 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:56.091+0000 7f16a4e87640 1 Processor -- start 2024-03-28T14:07:56.093 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:56.091+0000 7f16a4e87640 1 -- start start 2024-03-28T14:07:56.093 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:56.091+0000 7f16a4e87640 1 --2- >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f16a00b0310 0x7f16a009f080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:07:56.093 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:56.091+0000 7f16a4e87640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f16a0147d90 0x7f16a01444d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:07:56.093 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:56.091+0000 7f16a4e87640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f16a0148170 0x7f16a0143440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:07:56.094 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:56.091+0000 7f16a4e87640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f16a014fa80 con 0x7f16a0148170 2024-03-28T14:07:56.094 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:56.091+0000 7f16a4e87640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f16a014f900 con 0x7f16a0147d90 2024-03-28T14:07:56.094 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:56.091+0000 7f16a4e87640 1 -- --> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] -- mon_getmap magic: 0 -- 0x7f16a014fc00 con 0x7f16a00b0310 2024-03-28T14:07:56.094 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:56.091+0000 7f169ed9d640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f16a0148170 0x7f16a0143440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:07:56.094 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:56.091+0000 7f169ed9d640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f16a0148170 0x7f16a0143440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:49084/0 (socket says 172.21.15.104:49084) 2024-03-28T14:07:56.094 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:56.091+0000 7f169ed9d640 1 -- 172.21.15.104:0/4086937430 learned_addr learned my addr 172.21.15.104:0/4086937430 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-03-28T14:07:56.094 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:56.091+0000 7f169e59c640 1 --2- 172.21.15.104:0/4086937430 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f16a00b0310 0x7f16a009f080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:07:56.094 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:56.091+0000 7f169ed9d640 1 -- 172.21.15.104:0/4086937430 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f16a00b0310 msgr2=0x7f16a009f080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:07:56.094 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:56.091+0000 7f169ed9d640 1 --2- 172.21.15.104:0/4086937430 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f16a00b0310 0x7f16a009f080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:07:56.094 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:56.091+0000 7f169ed9d640 1 -- 172.21.15.104:0/4086937430 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f16a0147d90 msgr2=0x7f16a01444d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:07:56.094 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:56.091+0000 7f169ed9d640 1 --2- 172.21.15.104:0/4086937430 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f16a0147d90 0x7f16a01444d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:07:56.094 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:56.091+0000 7f169ed9d640 1 -- 172.21.15.104:0/4086937430 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f16a0142f60 con 0x7f16a0148170 2024-03-28T14:07:56.094 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:56.091+0000 7f169e59c640 1 --2- 172.21.15.104:0/4086937430 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f16a00b0310 0x7f16a009f080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T14:07:56.095 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:56.091+0000 7f169ed9d640 1 --2- 172.21.15.104:0/4086937430 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f16a0148170 0x7f16a0143440 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f1698009e60 tx=0x7f169800c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:07:56.095 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:56.091+0000 7f1696ffd640 1 -- 172.21.15.104:0/4086937430 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1698018030 con 0x7f16a0148170 2024-03-28T14:07:56.095 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:56.091+0000 7f1696ffd640 1 -- 172.21.15.104:0/4086937430 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f169800cdf0 con 0x7f16a0148170 2024-03-28T14:07:56.095 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:56.091+0000 7f16a4e87640 1 -- 172.21.15.104:0/4086937430 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f16a0144bb0 con 0x7f16a0148170 2024-03-28T14:07:56.095 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:56.091+0000 7f1696ffd640 1 -- 172.21.15.104:0/4086937430 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1698003bc0 con 0x7f16a0148170 2024-03-28T14:07:56.095 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:56.091+0000 7f16a4e87640 1 -- 172.21.15.104:0/4086937430 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f16a01425d0 con 0x7f16a0148170 2024-03-28T14:07:56.096 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:56.091+0000 7f16a4e87640 1 -- 172.21.15.104:0/4086937430 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f16a01339c0 con 0x7f16a0148170 2024-03-28T14:07:56.096 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:56.091+0000 7f1696ffd640 1 -- 172.21.15.104:0/4086937430 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) ==== 87803+0+0 (secure 0 0 0) 0x7f169801b070 con 0x7f16a0148170 2024-03-28T14:07:56.096 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:56.095+0000 7f1696ffd640 1 --2- 172.21.15.104:0/4086937430 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f1680067fc0 0x7f168006a480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:07:56.096 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:56.095+0000 7f1696ffd640 1 -- 172.21.15.104:0/4086937430 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(17..17 src has 1..17) ==== 4857+0+0 (secure 0 0 0) 0x7f1698086a90 con 0x7f16a0148170 2024-03-28T14:07:56.096 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:56.095+0000 7f169e59c640 1 --2- 172.21.15.104:0/4086937430 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f1680067fc0 0x7f168006a480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:07:56.096 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:56.095+0000 7f169e59c640 1 --2- 172.21.15.104:0/4086937430 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f1680067fc0 0x7f168006a480 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f16900ae000 tx=0x7f1690008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:07:56.099 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:56.099+0000 7f1696ffd640 1 -- 172.21.15.104:0/4086937430 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179117 (secure 0 0 0) 0x7f1698022070 con 0x7f16a0148170 2024-03-28T14:07:56.248 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:56.247+0000 7f16a4e87640 1 -- 172.21.15.104:0/4086937430 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f16a00687c0 con 0x7f16a0148170 2024-03-28T14:07:56.249 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:56.247+0000 7f1696ffd640 1 -- 172.21.15.104:0/4086937430 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v17) ==== 74+0+12908 (secure 0 0 0) 0x7f1698010040 con 0x7f16a0148170 2024-03-28T14:07:56.249 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-03-28T14:07:56.249 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":17,"fsid":"428a90f2-3351-435f-8fcf-f7e66c78a13e","created":"2024-03-28T14:07:34.862458+0000","modified":"2024-03-28T14:07:50.087852+0000","last_up_change":"2024-03-28T14:07:46.028122+0000","last_in_change":"2024-03-28T14:07:37.157486+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-28T14:07:47.622988+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"1cb70540-4e42-402f-a446-aaec4ca83007","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6800","nonce":3745690918},{"type":"v1","addr":"172.21.15.104:6801","nonce":3745690918}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":3745690918},{"type":"v1","addr":"172.21.15.104:6803","nonce":3745690918}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":3745690918},{"type":"v1","addr":"172.21.15.104:6807","nonce":3745690918}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":3745690918},{"type":"v1","addr":"172.21.15.104:6805","nonce":3745690918}]},"public_addr":"172.21.15.104:6801/3745690918","cluster_addr":"172.21.15.104:6803/3745690918","heartbeat_back_addr":"172.21.15.104:6807/3745690918","heartbeat_front_addr":"172.21.15.104:6805/3745690918","state":["exists","up"]},{"osd":1,"uuid":"3ab7afdd-b980-418c-81f2-004e16de3dca","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":19438674},{"type":"v1","addr":"172.21.15.104:6810","nonce":19438674}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":19438674},{"type":"v1","addr":"172.21.15.104:6814","nonce":19438674}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":19438674},{"type":"v1","addr":"172.21.15.104:6823","nonce":19438674}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":19438674},{"type":"v1","addr":"172.21.15.104:6819","nonce":19438674}]},"public_addr":"172.21.15.104:6810/19438674","cluster_addr":"172.21.15.104:6814/19438674","heartbeat_back_addr":"172.21.15.104:6823/19438674","heartbeat_front_addr":"172.21.15.104:6819/19438674","state":["exists","up"]},{"osd":2,"uuid":"3e2651c8-ecae-4a9e-a053-27240a79eac2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":2352012626},{"type":"v1","addr":"172.21.15.104:6825","nonce":2352012626}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6826","nonce":2352012626},{"type":"v1","addr":"172.21.15.104:6827","nonce":2352012626}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":2352012626},{"type":"v1","addr":"172.21.15.104:6831","nonce":2352012626}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6828","nonce":2352012626},{"type":"v1","addr":"172.21.15.104:6829","nonce":2352012626}]},"public_addr":"172.21.15.104:6825/2352012626","cluster_addr":"172.21.15.104:6827/2352012626","heartbeat_back_addr":"172.21.15.104:6831/2352012626","heartbeat_front_addr":"172.21.15.104:6829/2352012626","state":["exists","up"]},{"osd":3,"uuid":"925c7e0a-38ef-47cd-b288-8bec8456fb17","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6809","nonce":3568347381},{"type":"v1","addr":"172.21.15.104:6811","nonce":3568347381}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6813","nonce":3568347381},{"type":"v1","addr":"172.21.15.104:6815","nonce":3568347381}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":3568347381},{"type":"v1","addr":"172.21.15.104:6822","nonce":3568347381}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6817","nonce":3568347381},{"type":"v1","addr":"172.21.15.104:6818","nonce":3568347381}]},"public_addr":"172.21.15.104:6811/3568347381","cluster_addr":"172.21.15.104:6815/3568347381","heartbeat_back_addr":"172.21.15.104:6822/3568347381","heartbeat_front_addr":"172.21.15.104:6818/3568347381","state":["exists","up"]},{"osd":4,"uuid":"4a8fac86-e595-4a0e-96e3-57edb3cc3f57","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6800","nonce":2107001077},{"type":"v1","addr":"172.21.15.171:6801","nonce":2107001077}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6802","nonce":2107001077},{"type":"v1","addr":"172.21.15.171:6803","nonce":2107001077}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6806","nonce":2107001077},{"type":"v1","addr":"172.21.15.171:6807","nonce":2107001077}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6804","nonce":2107001077},{"type":"v1","addr":"172.21.15.171:6805","nonce":2107001077}]},"public_addr":"172.21.15.171:6801/2107001077","cluster_addr":"172.21.15.171:6803/2107001077","heartbeat_back_addr":"172.21.15.171:6807/2107001077","heartbeat_front_addr":"172.21.15.171:6805/2107001077","state":["exists","up"]},{"osd":5,"uuid":"e48d50b0-beae-40ae-b15a-9a84dbfe88ab","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6808","nonce":2934826925},{"type":"v1","addr":"172.21.15.171:6810","nonce":2934826925}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6812","nonce":2934826925},{"type":"v1","addr":"172.21.15.171:6814","nonce":2934826925}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6819","nonce":2934826925},{"type":"v1","addr":"172.21.15.171:6821","nonce":2934826925}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6816","nonce":2934826925},{"type":"v1","addr":"172.21.15.171:6817","nonce":2934826925}]},"public_addr":"172.21.15.171:6810/2934826925","cluster_addr":"172.21.15.171:6814/2934826925","heartbeat_back_addr":"172.21.15.171:6821/2934826925","heartbeat_front_addr":"172.21.15.171:6817/2934826925","state":["exists","up"]},{"osd":6,"uuid":"5e7c3292-9936-4f35-b267-aed9d4e49323","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6824","nonce":436742919},{"type":"v1","addr":"172.21.15.171:6825","nonce":436742919}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6826","nonce":436742919},{"type":"v1","addr":"172.21.15.171:6827","nonce":436742919}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6830","nonce":436742919},{"type":"v1","addr":"172.21.15.171:6831","nonce":436742919}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6828","nonce":436742919},{"type":"v1","addr":"172.21.15.171:6829","nonce":436742919}]},"public_addr":"172.21.15.171:6825/436742919","cluster_addr":"172.21.15.171:6827/436742919","heartbeat_back_addr":"172.21.15.171:6831/436742919","heartbeat_front_addr":"172.21.15.171:6829/436742919","state":["exists","up"]},{"osd":7,"uuid":"339b8414-d7f0-4fb0-a446-efaf635cb7ff","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6809","nonce":2055631175},{"type":"v1","addr":"172.21.15.171:6811","nonce":2055631175}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6813","nonce":2055631175},{"type":"v1","addr":"172.21.15.171:6815","nonce":2055631175}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6822","nonce":2055631175},{"type":"v1","addr":"172.21.15.171:6823","nonce":2055631175}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6818","nonce":2055631175},{"type":"v1","addr":"172.21.15.171:6820","nonce":2055631175}]},"public_addr":"172.21.15.171:6811/2055631175","cluster_addr":"172.21.15.171:6815/2055631175","heartbeat_back_addr":"172.21.15.171:6823/2055631175","heartbeat_front_addr":"172.21.15.171:6820/2055631175","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T14:07:42.668881+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T14:07:42.708397+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T14:07:42.634191+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T14:07:42.733662+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T14:07:42.651541+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T14:07:42.783181+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T14:07:42.766310+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T14:07:42.670797+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-28T14:07:56.251 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:56.247+0000 7f16a4e87640 1 -- 172.21.15.104:0/4086937430 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f1680067fc0 msgr2=0x7f168006a480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:07:56.251 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:56.247+0000 7f16a4e87640 1 --2- 172.21.15.104:0/4086937430 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f1680067fc0 0x7f168006a480 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f16900ae000 tx=0x7f1690008040 comp rx=0 tx=0).stop 2024-03-28T14:07:56.251 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:56.247+0000 7f16a4e87640 1 -- 172.21.15.104:0/4086937430 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f16a0148170 msgr2=0x7f16a0143440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:07:56.252 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:56.247+0000 7f16a4e87640 1 --2- 172.21.15.104:0/4086937430 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f16a0148170 0x7f16a0143440 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f1698009e60 tx=0x7f169800c490 comp rx=0 tx=0).stop 2024-03-28T14:07:56.252 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:56.251+0000 7f16a4e87640 1 -- 172.21.15.104:0/4086937430 shutdown_connections 2024-03-28T14:07:56.252 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:56.251+0000 7f16a4e87640 1 --2- 172.21.15.104:0/4086937430 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f1680067fc0 0x7f168006a480 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:07:56.252 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:56.251+0000 7f16a4e87640 1 --2- 172.21.15.104:0/4086937430 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f16a0148170 0x7f16a0143440 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:07:56.252 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:56.251+0000 7f16a4e87640 1 --2- 172.21.15.104:0/4086937430 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f16a0147d90 0x7f16a01444d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:07:56.253 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:56.251+0000 7f16a4e87640 1 --2- 172.21.15.104:0/4086937430 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f16a00b0310 0x7f16a009f080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:07:56.253 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:56.251+0000 7f16a4e87640 1 -- 172.21.15.104:0/4086937430 >> 172.21.15.104:0/4086937430 conn(0x7f16a00b7d90 msgr2=0x7f16a006e030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T14:07:56.253 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:56.251+0000 7f16a4e87640 1 -- 172.21.15.104:0/4086937430 shutdown_connections 2024-03-28T14:07:56.253 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:56.251+0000 7f16a4e87640 1 -- 172.21.15.104:0/4086937430 wait complete. 2024-03-28T14:07:56.267 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-28T14:07:47.622988+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-28T14:07:56.267 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-28T14:07:56.379 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:56.375+0000 7ff0fc8c1640 1 Processor -- start 2024-03-28T14:07:56.379 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:56.375+0000 7ff0fc8c1640 1 -- start start 2024-03-28T14:07:56.380 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:56.379+0000 7ff0fc8c1640 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7ff0f806c8f0 0x7ff0f80b2740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:07:56.381 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:56.379+0000 7ff0fc8c1640 1 --2- >> v2:172.21.15.171:3300/0 conn(0x7ff0f80b3730 0x7ff0f8148c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:07:56.381 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:56.379+0000 7ff0fc8c1640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff0f806ccd0 con 0x7ff0f81491b0 2024-03-28T14:07:56.381 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:56.379+0000 7ff0fc8c1640 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 -- 0x7ff0f80b0b30 con 0x7ff0f806c8f0 2024-03-28T14:07:56.381 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:56.379+0000 7ff0fc8c1640 1 -- --> v2:172.21.15.171:3300/0 -- mon_getmap magic: 0 -- 0x7ff0f80b22b0 con 0x7ff0f80b3730 2024-03-28T14:07:56.381 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:56.379+0000 7ff0f659c640 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7ff0f806c8f0 0x7ff0f80b2740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:07:56.381 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:56.379+0000 7ff0f6d9d640 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7ff0f81491b0 0x7ff0f8147b80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:60284/0 (socket says 172.21.15.104:60284) 2024-03-28T14:07:56.381 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:56.379+0000 7ff0f6d9d640 1 -- 172.21.15.104:0/2825624918 learned_addr learned my addr 172.21.15.104:0/2825624918 (peer_addr_for_me v1:172.21.15.104:0/0) 2024-03-28T14:07:56.381 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:56.379+0000 7ff0f5d9b640 1 --2- 172.21.15.104:0/2825624918 >> v2:172.21.15.171:3300/0 conn(0x7ff0f80b3730 0x7ff0f8148c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:07:56.381 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:56.379+0000 7ff0f659c640 1 -- 172.21.15.104:0/2825624918 >> v2:172.21.15.171:3300/0 conn(0x7ff0f80b3730 msgr2=0x7ff0f8148c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T14:07:56.381 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:56.379+0000 7ff0f659c640 1 --2- 172.21.15.104:0/2825624918 >> v2:172.21.15.171:3300/0 conn(0x7ff0f80b3730 0x7ff0f8148c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:07:56.381 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:56.379+0000 7ff0f659c640 1 -- 172.21.15.104:0/2825624918 >> v1:172.21.15.104:6789/0 conn(0x7ff0f81491b0 legacy=0x7ff0f8147b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:07:56.382 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:56.379+0000 7ff0f559a640 1 -- 172.21.15.104:0/2825624918 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4900084 0 0) 0x7ff0f806ccd0 con 0x7ff0f81491b0 2024-03-28T14:07:56.382 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:56.379+0000 7ff0f659c640 1 -- 172.21.15.104:0/2825624918 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff0f80b1980 con 0x7ff0f806c8f0 2024-03-28T14:07:56.382 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:56.379+0000 7ff0f659c640 1 --2- 172.21.15.104:0/2825624918 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff0f806c8f0 0x7ff0f80b2740 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7ff0f8072d50 tx=0x7ff0f001c8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5bf26ad6a0c04c10 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:07:56.382 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:56.379+0000 7ff0f559a640 1 -- 172.21.15.104:0/2825624918 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0f00b2070 con 0x7ff0f806c8f0 2024-03-28T14:07:56.383 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:56.379+0000 7ff0f559a640 1 -- 172.21.15.104:0/2825624918 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff0f00a57f0 con 0x7ff0f806c8f0 2024-03-28T14:07:56.383 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:56.379+0000 7ff0f559a640 1 -- 172.21.15.104:0/2825624918 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0f00a5bc0 con 0x7ff0f806c8f0 2024-03-28T14:07:56.383 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:56.383+0000 7ff0fc8c1640 1 -- 172.21.15.104:0/2825624918 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff0f806c8f0 msgr2=0x7ff0f80b2740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:07:56.384 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:56.383+0000 7ff0fc8c1640 1 --2- 172.21.15.104:0/2825624918 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff0f806c8f0 0x7ff0f80b2740 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7ff0f8072d50 tx=0x7ff0f001c8a0 comp rx=0 tx=0).stop 2024-03-28T14:07:56.384 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:56.383+0000 7ff0fc8c1640 1 -- 172.21.15.104:0/2825624918 shutdown_connections 2024-03-28T14:07:56.384 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:56.383+0000 7ff0fc8c1640 1 --2- 172.21.15.104:0/2825624918 >> v2:172.21.15.171:3300/0 conn(0x7ff0f80b3730 0x7ff0f8148c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:07:56.384 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:56.383+0000 7ff0fc8c1640 1 --2- 172.21.15.104:0/2825624918 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff0f806c8f0 0x7ff0f80b2740 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:07:56.384 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:56.383+0000 7ff0fc8c1640 1 -- 172.21.15.104:0/2825624918 >> 172.21.15.104:0/2825624918 conn(0x7ff0f80b8f00 msgr2=0x7ff0f80b9310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T14:07:56.384 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:56.383+0000 7ff0fc8c1640 1 -- 172.21.15.104:0/2825624918 shutdown_connections 2024-03-28T14:07:56.384 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:56.383+0000 7ff0fc8c1640 1 -- 172.21.15.104:0/2825624918 wait complete. 2024-03-28T14:07:56.386 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:56.383+0000 7ff0fc8c1640 1 Processor -- start 2024-03-28T14:07:56.386 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:56.383+0000 7ff0fc8c1640 1 -- start start 2024-03-28T14:07:56.386 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:56.383+0000 7ff0fc8c1640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7ff0f806c8f0 0x7ff0f81e38a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:07:56.387 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:56.383+0000 7ff0fc8c1640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff0f80b3730 0x7ff0f809c570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:07:56.387 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:56.383+0000 7ff0fc8c1640 1 --2- >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7ff0f81491b0 0x7ff0f809cf90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:07:56.387 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:56.383+0000 7ff0fc8c1640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7ff0f80b1660 con 0x7ff0f80b3730 2024-03-28T14:07:56.387 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:56.383+0000 7ff0fc8c1640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7ff0f80b14e0 con 0x7ff0f806c8f0 2024-03-28T14:07:56.387 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:56.383+0000 7ff0fc8c1640 1 -- --> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] -- mon_getmap magic: 0 -- 0x7ff0f806dc70 con 0x7ff0f81491b0 2024-03-28T14:07:56.387 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:56.383+0000 7ff0f5d9b640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff0f80b3730 0x7ff0f809c570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:07:56.387 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:56.383+0000 7ff0f5d9b640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff0f80b3730 0x7ff0f809c570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:49102/0 (socket says 172.21.15.104:49102) 2024-03-28T14:07:56.388 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:56.383+0000 7ff0f5d9b640 1 -- 172.21.15.104:0/4139372395 learned_addr learned my addr 172.21.15.104:0/4139372395 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-03-28T14:07:56.388 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:56.383+0000 7ff0f6d9d640 1 --2- 172.21.15.104:0/4139372395 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7ff0f81491b0 0x7ff0f809cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:07:56.388 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:56.383+0000 7ff0f659c640 1 --2- 172.21.15.104:0/4139372395 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7ff0f806c8f0 0x7ff0f81e38a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:07:56.388 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:56.383+0000 7ff0f5d9b640 1 -- 172.21.15.104:0/4139372395 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7ff0f81491b0 msgr2=0x7ff0f809cf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:07:56.388 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:56.383+0000 7ff0f5d9b640 1 --2- 172.21.15.104:0/4139372395 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7ff0f81491b0 0x7ff0f809cf90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:07:56.388 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:56.383+0000 7ff0f5d9b640 1 -- 172.21.15.104:0/4139372395 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7ff0f806c8f0 msgr2=0x7ff0f81e38a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:07:56.388 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:56.383+0000 7ff0f5d9b640 1 --2- 172.21.15.104:0/4139372395 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7ff0f806c8f0 0x7ff0f81e38a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:07:56.388 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:56.383+0000 7ff0f5d9b640 1 -- 172.21.15.104:0/4139372395 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff0f809abb0 con 0x7ff0f80b3730 2024-03-28T14:07:56.388 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:56.383+0000 7ff0f659c640 1 --2- 172.21.15.104:0/4139372395 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7ff0f806c8f0 0x7ff0f81e38a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T14:07:56.389 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:56.383+0000 7ff0f5d9b640 1 --2- 172.21.15.104:0/4139372395 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff0f80b3730 0x7ff0f809c570 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7ff0e000d800 tx=0x7ff0e000dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:07:56.389 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:56.383+0000 7ff0eeffd640 1 -- 172.21.15.104:0/4139372395 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0e000b6c0 con 0x7ff0f80b3730 2024-03-28T14:07:56.389 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:56.383+0000 7ff0eeffd640 1 -- 172.21.15.104:0/4139372395 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff0e0019070 con 0x7ff0f80b3730 2024-03-28T14:07:56.389 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:56.383+0000 7ff0eeffd640 1 -- 172.21.15.104:0/4139372395 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0e000ba60 con 0x7ff0f80b3730 2024-03-28T14:07:56.389 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:56.383+0000 7ff0fc8c1640 1 -- 172.21.15.104:0/4139372395 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff0f81e3de0 con 0x7ff0f80b3730 2024-03-28T14:07:56.389 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:56.383+0000 7ff0fc8c1640 1 -- 172.21.15.104:0/4139372395 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff0f8071cb0 con 0x7ff0f80b3730 2024-03-28T14:07:56.389 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:56.383+0000 7ff0fc8c1640 1 -- 172.21.15.104:0/4139372395 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff0f80687c0 con 0x7ff0f80b3730 2024-03-28T14:07:56.391 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:56.387+0000 7ff0eeffd640 1 -- 172.21.15.104:0/4139372395 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) ==== 87803+0+0 (secure 0 0 0) 0x7ff0e000bc00 con 0x7ff0f80b3730 2024-03-28T14:07:56.391 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:56.387+0000 7ff0eeffd640 1 --2- 172.21.15.104:0/4139372395 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7ff0d8067ff0 0x7ff0d806a4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:07:56.391 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:56.387+0000 7ff0eeffd640 1 -- 172.21.15.104:0/4139372395 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(17..17 src has 1..17) ==== 4857+0+0 (secure 0 0 0) 0x7ff0e00865c0 con 0x7ff0f80b3730 2024-03-28T14:07:56.391 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:56.387+0000 7ff0f659c640 1 --2- 172.21.15.104:0/4139372395 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7ff0d8067ff0 0x7ff0d806a4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:07:56.391 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:56.387+0000 7ff0f659c640 1 --2- 172.21.15.104:0/4139372395 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7ff0d8067ff0 0x7ff0d806a4b0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7ff0f00af000 tx=0x7ff0f00a7070 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:07:56.392 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:56.391+0000 7ff0eeffd640 1 -- 172.21.15.104:0/4139372395 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179117 (secure 0 0 0) 0x7ff0e0057150 con 0x7ff0f80b3730 2024-03-28T14:07:56.539 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:56.535+0000 7ff0fc8c1640 1 -- 172.21.15.104:0/4139372395 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7ff0f813dac0 con 0x7ff0f80b3730 2024-03-28T14:07:56.539 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:56.539+0000 7ff0eeffd640 1 -- 172.21.15.104:0/4139372395 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v17) ==== 93+0+10 (secure 0 0 0) 0x7ff0e0056a50 con 0x7ff0f80b3730 2024-03-28T14:07:56.540 INFO:teuthology.orchestra.run.smithi104.stdout:pg_num: 1 2024-03-28T14:07:56.542 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:56.539+0000 7ff0fc8c1640 1 -- 172.21.15.104:0/4139372395 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7ff0d8067ff0 msgr2=0x7ff0d806a4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:07:56.542 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:56.539+0000 7ff0fc8c1640 1 --2- 172.21.15.104:0/4139372395 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7ff0d8067ff0 0x7ff0d806a4b0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7ff0f00af000 tx=0x7ff0f00a7070 comp rx=0 tx=0).stop 2024-03-28T14:07:56.542 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:56.539+0000 7ff0fc8c1640 1 -- 172.21.15.104:0/4139372395 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff0f80b3730 msgr2=0x7ff0f809c570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:07:56.542 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:56.539+0000 7ff0fc8c1640 1 --2- 172.21.15.104:0/4139372395 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff0f80b3730 0x7ff0f809c570 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7ff0e000d800 tx=0x7ff0e000dcc0 comp rx=0 tx=0).stop 2024-03-28T14:07:56.542 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:56.539+0000 7ff0fc8c1640 1 -- 172.21.15.104:0/4139372395 shutdown_connections 2024-03-28T14:07:56.542 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:56.539+0000 7ff0fc8c1640 1 --2- 172.21.15.104:0/4139372395 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7ff0f81491b0 0x7ff0f809cf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:07:56.542 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:56.539+0000 7ff0fc8c1640 1 --2- 172.21.15.104:0/4139372395 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7ff0d8067ff0 0x7ff0d806a4b0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:07:56.543 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:56.539+0000 7ff0fc8c1640 1 --2- 172.21.15.104:0/4139372395 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff0f80b3730 0x7ff0f809c570 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:07:56.543 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:56.539+0000 7ff0fc8c1640 1 --2- 172.21.15.104:0/4139372395 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7ff0f806c8f0 0x7ff0f81e38a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:07:56.543 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:56.539+0000 7ff0fc8c1640 1 -- 172.21.15.104:0/4139372395 >> 172.21.15.104:0/4139372395 conn(0x7ff0f80b8f00 msgr2=0x7ff0f80b4b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T14:07:56.543 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:56.539+0000 7ff0fc8c1640 1 -- 172.21.15.104:0/4139372395 shutdown_connections 2024-03-28T14:07:56.543 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:56.539+0000 7ff0fc8c1640 1 -- 172.21.15.104:0/4139372395 wait complete. 2024-03-28T14:07:56.558 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-03-28T14:07:56.666 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:56.663+0000 7f336a983640 1 Processor -- start 2024-03-28T14:07:56.667 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:56.663+0000 7f336a983640 1 -- start start 2024-03-28T14:07:56.668 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:56.667+0000 7f336a983640 1 --2- >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f33640bd9e0 0x7f336414af30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:07:56.668 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:56.667+0000 7f336a983640 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f336414b470 0x7f33641413f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:07:56.668 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:56.667+0000 7f336a983640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f33640b5050 con 0x7f3364141930 2024-03-28T14:07:56.668 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:56.667+0000 7f336a983640 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 -- 0x7f33640b70b0 con 0x7f336414b470 2024-03-28T14:07:56.669 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:56.667+0000 7f336a983640 1 -- --> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] -- mon_getmap magic: 0 -- 0x7f336414b8d0 con 0x7f33640bd9e0 2024-03-28T14:07:56.669 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:56.667+0000 7f3368f20640 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f3364141930 0x7f33640b79f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:47238/0 (socket says 172.21.15.104:47238) 2024-03-28T14:07:56.669 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:56.667+0000 7f33637fe640 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f336414b470 0x7f33641413f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:07:56.669 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:56.667+0000 7f3368f20640 1 -- 172.21.15.104:0/32260534 learned_addr learned my addr 172.21.15.104:0/32260534 (peer_addr_for_me v1:172.21.15.104:0/0) 2024-03-28T14:07:56.669 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:56.667+0000 7f3363fff640 1 --2- 172.21.15.104:0/32260534 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f33640bd9e0 0x7f336414af30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:07:56.669 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:56.667+0000 7f33637fe640 1 -- 172.21.15.104:0/32260534 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f33640bd9e0 msgr2=0x7f336414af30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T14:07:56.669 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:56.667+0000 7f33637fe640 1 --2- 172.21.15.104:0/32260534 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f33640bd9e0 0x7f336414af30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:07:56.669 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:56.667+0000 7f33637fe640 1 -- 172.21.15.104:0/32260534 >> v1:172.21.15.104:6789/0 conn(0x7f3364141930 legacy=0x7f33640b79f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:07:56.669 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:56.667+0000 7f33637fe640 1 -- 172.21.15.104:0/32260534 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f33640bddc0 con 0x7f336414b470 2024-03-28T14:07:56.669 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:56.667+0000 7f3362ffd640 1 -- 172.21.15.104:0/32260534 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1881592348 0 0) 0x7f33640b5050 con 0x7f3364141930 2024-03-28T14:07:56.674 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:56.667+0000 7f33637fe640 1 --2- 172.21.15.104:0/32260534 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f336414b470 0x7f33641413f0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f33580097f0 tx=0x7f335801c360 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c2e146de1c656e8f server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:07:56.674 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:56.667+0000 7f3362ffd640 1 -- 172.21.15.104:0/32260534 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f33580b0070 con 0x7f336414b470 2024-03-28T14:07:56.674 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:56.667+0000 7f3362ffd640 1 -- 172.21.15.104:0/32260534 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f33580ab600 con 0x7f336414b470 2024-03-28T14:07:56.674 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:56.667+0000 7f3362ffd640 1 -- 172.21.15.104:0/32260534 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f33580ab9d0 con 0x7f336414b470 2024-03-28T14:07:56.674 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:56.667+0000 7f336a983640 1 -- 172.21.15.104:0/32260534 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f336414b470 msgr2=0x7f33641413f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:07:56.674 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:56.667+0000 7f336a983640 1 --2- 172.21.15.104:0/32260534 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f336414b470 0x7f33641413f0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f33580097f0 tx=0x7f335801c360 comp rx=0 tx=0).stop 2024-03-28T14:07:56.674 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:56.667+0000 7f336a983640 1 -- 172.21.15.104:0/32260534 shutdown_connections 2024-03-28T14:07:56.675 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:56.667+0000 7f336a983640 1 --2- 172.21.15.104:0/32260534 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f336414b470 0x7f33641413f0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:07:56.675 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:56.667+0000 7f336a983640 1 --2- 172.21.15.104:0/32260534 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f33640bd9e0 0x7f336414af30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:07:56.675 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:56.667+0000 7f336a983640 1 -- 172.21.15.104:0/32260534 >> 172.21.15.104:0/32260534 conn(0x7f3364073530 msgr2=0x7f33640bd070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T14:07:56.675 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:56.667+0000 7f336a983640 1 -- 172.21.15.104:0/32260534 shutdown_connections 2024-03-28T14:07:56.675 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:56.667+0000 7f336a983640 1 -- 172.21.15.104:0/32260534 wait complete. 2024-03-28T14:07:56.675 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:56.671+0000 7f336a983640 1 Processor -- start 2024-03-28T14:07:56.675 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:56.671+0000 7f336a983640 1 -- start start 2024-03-28T14:07:56.675 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:56.671+0000 7f336a983640 1 --2- >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f33640bd9e0 0x7f3364140b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:07:56.676 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:56.671+0000 7f336a983640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f3364141930 0x7f33641e0090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:07:56.676 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:56.671+0000 7f336a983640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f336414b470 0x7f33641e3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:07:56.676 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:56.671+0000 7f336a983640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f33640b6140 con 0x7f336414b470 2024-03-28T14:07:56.676 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:56.671+0000 7f336a983640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f33640b5940 con 0x7f3364141930 2024-03-28T14:07:56.676 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:56.671+0000 7f336a983640 1 -- --> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] -- mon_getmap magic: 0 -- 0x7f33640b62c0 con 0x7f33640bd9e0 2024-03-28T14:07:56.676 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:56.671+0000 7f3368f20640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f336414b470 0x7f33641e3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:07:56.677 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:56.671+0000 7f3368f20640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f336414b470 0x7f33641e3600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:42146/0 (socket says 172.21.15.104:42146) 2024-03-28T14:07:56.677 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:56.671+0000 7f3368f20640 1 -- 172.21.15.104:0/4107674881 learned_addr learned my addr 172.21.15.104:0/4107674881 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-03-28T14:07:56.677 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:56.671+0000 7f3363fff640 1 --2- 172.21.15.104:0/4107674881 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f33640bd9e0 0x7f3364140b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:07:56.677 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:56.671+0000 7f33637fe640 1 --2- 172.21.15.104:0/4107674881 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f3364141930 0x7f33641e0090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:07:56.677 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:56.671+0000 7f3368f20640 1 -- 172.21.15.104:0/4107674881 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f33640bd9e0 msgr2=0x7f3364140b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:07:56.677 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:56.671+0000 7f3368f20640 1 --2- 172.21.15.104:0/4107674881 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f33640bd9e0 0x7f3364140b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:07:56.677 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:56.671+0000 7f3368f20640 1 -- 172.21.15.104:0/4107674881 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f3364141930 msgr2=0x7f33641e0090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:07:56.677 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:56.671+0000 7f3368f20640 1 --2- 172.21.15.104:0/4107674881 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f3364141930 0x7f33641e0090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:07:56.678 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:56.671+0000 7f3368f20640 1 -- 172.21.15.104:0/4107674881 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3364142200 con 0x7f336414b470 2024-03-28T14:07:56.678 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:56.671+0000 7f3368f20640 1 --2- 172.21.15.104:0/4107674881 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f336414b470 0x7f33641e3600 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f335c00c7d0 tx=0x7f335c00cc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:07:56.678 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:56.671+0000 7f3360ff9640 1 -- 172.21.15.104:0/4107674881 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f335c018070 con 0x7f336414b470 2024-03-28T14:07:56.678 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:56.671+0000 7f3360ff9640 1 -- 172.21.15.104:0/4107674881 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f335c00a640 con 0x7f336414b470 2024-03-28T14:07:56.679 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:56.671+0000 7f336a983640 1 -- 172.21.15.104:0/4107674881 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3364139c80 con 0x7f336414b470 2024-03-28T14:07:56.679 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:56.671+0000 7f3360ff9640 1 -- 172.21.15.104:0/4107674881 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f335c00aa70 con 0x7f336414b470 2024-03-28T14:07:56.679 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:56.671+0000 7f336a983640 1 -- 172.21.15.104:0/4107674881 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3364139280 con 0x7f336414b470 2024-03-28T14:07:56.679 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:56.671+0000 7f336a983640 1 -- 172.21.15.104:0/4107674881 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f33640687c0 con 0x7f336414b470 2024-03-28T14:07:56.679 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:56.671+0000 7f3360ff9640 1 -- 172.21.15.104:0/4107674881 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) ==== 87803+0+0 (secure 0 0 0) 0x7f335c01b070 con 0x7f336414b470 2024-03-28T14:07:56.680 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:56.671+0000 7f3360ff9640 1 --2- 172.21.15.104:0/4107674881 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f3350067fc0 0x7f335006a480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:07:56.680 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:56.671+0000 7f3360ff9640 1 -- 172.21.15.104:0/4107674881 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(17..17 src has 1..17) ==== 4857+0+0 (secure 0 0 0) 0x7f335c0856e0 con 0x7f336414b470 2024-03-28T14:07:56.680 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:56.671+0000 7f3363fff640 1 --2- 172.21.15.104:0/4107674881 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f3350067fc0 0x7f335006a480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:07:56.680 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:56.675+0000 7f3363fff640 1 --2- 172.21.15.104:0/4107674881 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f3350067fc0 0x7f335006a480 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f336413d010 tx=0x7f3354008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:07:56.680 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:56.675+0000 7f3360ff9640 1 -- 172.21.15.104:0/4107674881 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179117 (secure 0 0 0) 0x7f335c056270 con 0x7f336414b470 2024-03-28T14:07:56.838 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:56.835+0000 7f336a983640 1 -- 172.21.15.104:0/4107674881 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) -- 0x7f33640b6af0 con 0x7f336414b470 2024-03-28T14:07:56.839 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:56.835+0000 7f3360ff9640 1 -- 172.21.15.104:0/4107674881 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v17) ==== 84+0+65 (secure 0 0 0) 0x7f335c055b70 con 0x7f336414b470 2024-03-28T14:07:56.839 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-03-28T14:07:56.839 INFO:teuthology.orchestra.run.smithi104.stdout:[ 2024-03-28T14:07:56.839 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-03-28T14:07:56.839 INFO:teuthology.orchestra.run.smithi104.stdout: "poolnum": 1, 2024-03-28T14:07:56.840 INFO:teuthology.orchestra.run.smithi104.stdout: "poolname": ".mgr" 2024-03-28T14:07:56.840 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-03-28T14:07:56.840 INFO:teuthology.orchestra.run.smithi104.stdout:] 2024-03-28T14:07:56.842 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:56.839+0000 7f336a983640 1 -- 172.21.15.104:0/4107674881 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f3350067fc0 msgr2=0x7f335006a480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:07:56.842 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:56.839+0000 7f336a983640 1 --2- 172.21.15.104:0/4107674881 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f3350067fc0 0x7f335006a480 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f336413d010 tx=0x7f3354008040 comp rx=0 tx=0).stop 2024-03-28T14:07:56.842 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:56.839+0000 7f336a983640 1 -- 172.21.15.104:0/4107674881 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f336414b470 msgr2=0x7f33641e3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:07:56.842 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:56.839+0000 7f336a983640 1 --2- 172.21.15.104:0/4107674881 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f336414b470 0x7f33641e3600 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f335c00c7d0 tx=0x7f335c00cc90 comp rx=0 tx=0).stop 2024-03-28T14:07:56.842 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:56.839+0000 7f336a983640 1 -- 172.21.15.104:0/4107674881 shutdown_connections 2024-03-28T14:07:56.843 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:56.839+0000 7f336a983640 1 --2- 172.21.15.104:0/4107674881 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f3350067fc0 0x7f335006a480 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:07:56.843 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:56.839+0000 7f336a983640 1 --2- 172.21.15.104:0/4107674881 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f336414b470 0x7f33641e3600 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:07:56.843 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:56.839+0000 7f336a983640 1 --2- 172.21.15.104:0/4107674881 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f3364141930 0x7f33641e0090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:07:56.843 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:56.839+0000 7f336a983640 1 --2- 172.21.15.104:0/4107674881 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f33640bd9e0 0x7f3364140b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:07:56.843 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:56.839+0000 7f336a983640 1 -- 172.21.15.104:0/4107674881 >> 172.21.15.104:0/4107674881 conn(0x7f3364073530 msgr2=0x7f33640b9af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T14:07:56.843 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:56.839+0000 7f336a983640 1 -- 172.21.15.104:0/4107674881 shutdown_connections 2024-03-28T14:07:56.843 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:56.839+0000 7f336a983640 1 -- 172.21.15.104:0/4107674881 wait complete. 2024-03-28T14:07:56.856 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-03-28T14:07:56.856 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-03-28T14:07:56.976 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:56.975+0000 7f0e86a82640 1 Processor -- start 2024-03-28T14:07:56.977 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:56.975+0000 7f0e86a82640 1 -- start start 2024-03-28T14:07:56.979 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:56.975+0000 7f0e86a82640 1 --2- >> v2:172.21.15.171:3300/0 conn(0x7f0e800bd9e0 0x7f0e8014af30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:07:56.979 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:56.975+0000 7f0e86a82640 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f0e80141b00 0x7f0e80152080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:07:56.979 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:56.975+0000 7f0e86a82640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0e8014b900 con 0x7f0e8014b470 2024-03-28T14:07:56.979 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:56.975+0000 7f0e86a82640 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 -- 0x7f0e800b70b0 con 0x7f0e80141b00 2024-03-28T14:07:56.980 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:56.975+0000 7f0e86a82640 1 -- --> v2:172.21.15.171:3300/0 -- mon_getmap magic: 0 -- 0x7f0e800b5050 con 0x7f0e800bd9e0 2024-03-28T14:07:56.980 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:56.975+0000 7f0e8501f640 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f0e80141b00 0x7f0e80152080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:07:57.076 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:56.975+0000 7f0e7ffff640 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f0e8014b470 0x7f0e801413f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:47248/0 (socket says 172.21.15.104:47248) 2024-03-28T14:07:57.076 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:56.975+0000 7f0e7ffff640 1 -- 172.21.15.104:0/1688157007 learned_addr learned my addr 172.21.15.104:0/1688157007 (peer_addr_for_me v1:172.21.15.104:0/0) 2024-03-28T14:07:57.076 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:56.975+0000 7f0e8481e640 1 --2- 172.21.15.104:0/1688157007 >> v2:172.21.15.171:3300/0 conn(0x7f0e800bd9e0 0x7f0e8014af30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:07:57.077 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:56.975+0000 7f0e8501f640 1 -- 172.21.15.104:0/1688157007 >> v2:172.21.15.171:3300/0 conn(0x7f0e800bd9e0 msgr2=0x7f0e8014af30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T14:07:57.077 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:56.975+0000 7f0e8501f640 1 --2- 172.21.15.104:0/1688157007 >> v2:172.21.15.171:3300/0 conn(0x7f0e800bd9e0 0x7f0e8014af30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:07:57.077 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:56.975+0000 7f0e8501f640 1 -- 172.21.15.104:0/1688157007 >> v1:172.21.15.104:6789/0 conn(0x7f0e8014b470 legacy=0x7f0e801413f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:07:57.077 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:56.975+0000 7f0e7f7fe640 1 -- 172.21.15.104:0/1688157007 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1106559984 0 0) 0x7f0e8014b900 con 0x7f0e8014b470 2024-03-28T14:07:57.077 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:56.975+0000 7f0e8501f640 1 -- 172.21.15.104:0/1688157007 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0e800bddc0 con 0x7f0e80141b00 2024-03-28T14:07:57.083 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:56.979+0000 7f0e8501f640 1 --2- 172.21.15.104:0/1688157007 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0e80141b00 0x7f0e80152080 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f0e800be0a0 tx=0x7f0e7001c330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f0f048cf9abb03c2 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:07:57.083 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:56.979+0000 7f0e7f7fe640 1 -- 172.21.15.104:0/1688157007 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e700b0070 con 0x7f0e80141b00 2024-03-28T14:07:57.083 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:56.979+0000 7f0e7f7fe640 1 -- 172.21.15.104:0/1688157007 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0e700ab600 con 0x7f0e80141b00 2024-03-28T14:07:57.084 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:56.979+0000 7f0e7f7fe640 1 -- 172.21.15.104:0/1688157007 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e700ab9d0 con 0x7f0e80141b00 2024-03-28T14:07:57.084 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:56.979+0000 7f0e86a82640 1 -- 172.21.15.104:0/1688157007 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0e80141b00 msgr2=0x7f0e80152080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:07:57.084 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:56.979+0000 7f0e86a82640 1 --2- 172.21.15.104:0/1688157007 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0e80141b00 0x7f0e80152080 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f0e800be0a0 tx=0x7f0e7001c330 comp rx=0 tx=0).stop 2024-03-28T14:07:57.084 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:56.979+0000 7f0e86a82640 1 -- 172.21.15.104:0/1688157007 shutdown_connections 2024-03-28T14:07:57.084 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:56.979+0000 7f0e86a82640 1 --2- 172.21.15.104:0/1688157007 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0e80141b00 0x7f0e80152080 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:07:57.084 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:56.979+0000 7f0e86a82640 1 --2- 172.21.15.104:0/1688157007 >> v2:172.21.15.171:3300/0 conn(0x7f0e800bd9e0 0x7f0e8014af30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:07:57.084 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:56.979+0000 7f0e86a82640 1 -- 172.21.15.104:0/1688157007 >> 172.21.15.104:0/1688157007 conn(0x7f0e80073530 msgr2=0x7f0e800bd070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T14:07:57.085 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:56.979+0000 7f0e86a82640 1 -- 172.21.15.104:0/1688157007 shutdown_connections 2024-03-28T14:07:57.085 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:56.979+0000 7f0e86a82640 1 -- 172.21.15.104:0/1688157007 wait complete. 2024-03-28T14:07:57.085 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:56.979+0000 7f0e86a82640 1 Processor -- start 2024-03-28T14:07:57.086 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:56.979+0000 7f0e86a82640 1 -- start start 2024-03-28T14:07:57.086 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:56.979+0000 7f0e86a82640 1 --2- >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f0e800bd9e0 0x7f0e80148ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:07:57.086 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:56.979+0000 7f0e86a82640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f0e80141b00 0x7f0e801e3a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:07:57.086 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:56.979+0000 7f0e86a82640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0e8014b470 0x7f0e801e3f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:07:57.086 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:56.979+0000 7f0e86a82640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f0e800b62c0 con 0x7f0e8014b470 2024-03-28T14:07:57.086 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:56.979+0000 7f0e86a82640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f0e800b6140 con 0x7f0e80141b00 2024-03-28T14:07:57.086 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:56.979+0000 7f0e86a82640 1 -- --> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] -- mon_getmap magic: 0 -- 0x7f0e800b4940 con 0x7f0e800bd9e0 2024-03-28T14:07:57.087 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:56.979+0000 7f0e8501f640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0e8014b470 0x7f0e801e3f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:07:57.087 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:56.979+0000 7f0e8501f640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0e8014b470 0x7f0e801e3f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:42168/0 (socket says 172.21.15.104:42168) 2024-03-28T14:07:57.087 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:56.979+0000 7f0e8501f640 1 -- 172.21.15.104:0/3605254219 learned_addr learned my addr 172.21.15.104:0/3605254219 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-03-28T14:07:57.087 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:56.979+0000 7f0e8481e640 1 --2- 172.21.15.104:0/3605254219 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f0e800bd9e0 0x7f0e80148ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:07:57.087 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:56.979+0000 7f0e8501f640 1 -- 172.21.15.104:0/3605254219 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f0e800bd9e0 msgr2=0x7f0e80148ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:07:57.087 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:56.979+0000 7f0e8501f640 1 --2- 172.21.15.104:0/3605254219 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f0e800bd9e0 0x7f0e80148ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:07:57.087 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:56.979+0000 7f0e8501f640 1 -- 172.21.15.104:0/3605254219 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f0e80141b00 msgr2=0x7f0e801e3a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:07:57.088 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:56.979+0000 7f0e8501f640 1 --2- 172.21.15.104:0/3605254219 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f0e80141b00 0x7f0e801e3a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:07:57.088 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:56.979+0000 7f0e8501f640 1 -- 172.21.15.104:0/3605254219 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0e80149f20 con 0x7f0e8014b470 2024-03-28T14:07:57.088 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:56.979+0000 7f0e8501f640 1 --2- 172.21.15.104:0/3605254219 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0e8014b470 0x7f0e801e3f70 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f0e700ae040 tx=0x7f0e700190d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:07:57.088 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:56.979+0000 7f0e7d7fa640 1 -- 172.21.15.104:0/3605254219 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e700b0040 con 0x7f0e8014b470 2024-03-28T14:07:57.088 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:56.979+0000 7f0e7d7fa640 1 -- 172.21.15.104:0/3605254219 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0e70004030 con 0x7f0e8014b470 2024-03-28T14:07:57.088 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:56.979+0000 7f0e86a82640 1 -- 172.21.15.104:0/3605254219 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0e80140550 con 0x7f0e8014b470 2024-03-28T14:07:57.088 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:56.983+0000 7f0e7d7fa640 1 -- 172.21.15.104:0/3605254219 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e700043e0 con 0x7f0e8014b470 2024-03-28T14:07:57.089 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:56.983+0000 7f0e86a82640 1 -- 172.21.15.104:0/3605254219 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0e8014a190 con 0x7f0e8014b470 2024-03-28T14:07:57.089 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:56.983+0000 7f0e86a82640 1 -- 172.21.15.104:0/3605254219 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0e800687c0 con 0x7f0e8014b470 2024-03-28T14:07:57.089 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:56.983+0000 7f0e7d7fa640 1 -- 172.21.15.104:0/3605254219 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) ==== 87803+0+0 (secure 0 0 0) 0x7f0e70003620 con 0x7f0e8014b470 2024-03-28T14:07:57.089 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:56.983+0000 7f0e7d7fa640 1 --2- 172.21.15.104:0/3605254219 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f0e60067fc0 0x7f0e6006a480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:07:57.089 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:56.983+0000 7f0e7d7fa640 1 -- 172.21.15.104:0/3605254219 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(17..17 src has 1..17) ==== 4857+0+0 (secure 0 0 0) 0x7f0e7011ffd0 con 0x7f0e8014b470 2024-03-28T14:07:57.089 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:56.983+0000 7f0e8481e640 1 --2- 172.21.15.104:0/3605254219 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f0e60067fc0 0x7f0e6006a480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:07:57.089 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:56.983+0000 7f0e8481e640 1 --2- 172.21.15.104:0/3605254219 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f0e60067fc0 0x7f0e6006a480 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f0e78009f30 tx=0x7f0e7800b040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:07:57.089 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:56.987+0000 7f0e7d7fa640 1 -- 172.21.15.104:0/3605254219 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179117 (secure 0 0 0) 0x7f0e700b8020 con 0x7f0e8014b470 2024-03-28T14:07:57.140 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:57.139+0000 7f0e86a82640 1 -- 172.21.15.104:0/3605254219 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) -- 0x7f0e801e3790 con 0x7f0e8014b470 2024-03-28T14:07:57.640 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:57.639+0000 7f0e7d7fa640 1 -- 172.21.15.104:0/3605254219 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v18) ==== 138+0+0 (secure 0 0 0) 0x7f0e70016070 con 0x7f0e8014b470 2024-03-28T14:07:57.640 INFO:teuthology.orchestra.run.smithi104.stderr:pool 'cephfs_metadata' created 2024-03-28T14:07:57.644 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:57.643+0000 7f0e86a82640 1 -- 172.21.15.104:0/3605254219 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f0e60067fc0 msgr2=0x7f0e6006a480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:07:57.645 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:57.643+0000 7f0e86a82640 1 --2- 172.21.15.104:0/3605254219 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f0e60067fc0 0x7f0e6006a480 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f0e78009f30 tx=0x7f0e7800b040 comp rx=0 tx=0).stop 2024-03-28T14:07:57.645 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:57.643+0000 7f0e86a82640 1 -- 172.21.15.104:0/3605254219 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0e8014b470 msgr2=0x7f0e801e3f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:07:57.645 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:57.643+0000 7f0e86a82640 1 --2- 172.21.15.104:0/3605254219 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0e8014b470 0x7f0e801e3f70 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f0e700ae040 tx=0x7f0e700190d0 comp rx=0 tx=0).stop 2024-03-28T14:07:57.645 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:57.643+0000 7f0e86a82640 1 -- 172.21.15.104:0/3605254219 shutdown_connections 2024-03-28T14:07:57.645 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:57.643+0000 7f0e86a82640 1 --2- 172.21.15.104:0/3605254219 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f0e60067fc0 0x7f0e6006a480 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:07:57.645 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:57.643+0000 7f0e86a82640 1 --2- 172.21.15.104:0/3605254219 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0e8014b470 0x7f0e801e3f70 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:07:57.645 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:57.643+0000 7f0e86a82640 1 --2- 172.21.15.104:0/3605254219 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f0e80141b00 0x7f0e801e3a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:07:57.645 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:57.643+0000 7f0e86a82640 1 --2- 172.21.15.104:0/3605254219 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f0e800bd9e0 0x7f0e80148ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:07:57.645 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:57.643+0000 7f0e86a82640 1 -- 172.21.15.104:0/3605254219 >> 172.21.15.104:0/3605254219 conn(0x7f0e80073530 msgr2=0x7f0e801411c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T14:07:57.645 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:57.643+0000 7f0e86a82640 1 -- 172.21.15.104:0/3605254219 shutdown_connections 2024-03-28T14:07:57.645 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:57.643+0000 7f0e86a82640 1 -- 172.21.15.104:0/3605254219 wait complete. 2024-03-28T14:07:57.673 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-03-28T14:07:57.808 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:57.807+0000 7f12815ed640 1 Processor -- start 2024-03-28T14:07:57.808 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:57.807+0000 7f12815ed640 1 -- start start 2024-03-28T14:07:57.809 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:57.807+0000 7f12815ed640 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f127c02c460 0x7f127c070790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:07:57.809 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:57.807+0000 7f12815ed640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f127c02c840 con 0x7f127c141440 2024-03-28T14:07:57.809 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:57.807+0000 7f12815ed640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f127c06e0f0 con 0x7f127c072290 2024-03-28T14:07:57.809 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:57.807+0000 7f12815ed640 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 -- 0x7f127c06e740 con 0x7f127c02c460 2024-03-28T14:07:57.809 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:57.807+0000 7f127affd640 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f127c02c460 0x7f127c070790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:07:57.809 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:57.807+0000 7f127affd640 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f127c02c460 0x7f127c070790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:42182/0 (socket says 172.21.15.104:42182) 2024-03-28T14:07:57.809 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:57.807+0000 7f127affd640 1 -- 172.21.15.104:0/2366319031 learned_addr learned my addr 172.21.15.104:0/2366319031 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-03-28T14:07:57.810 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:57.807+0000 7f127affd640 1 -- 172.21.15.104:0/2366319031 >> v1:172.21.15.171:6789/0 conn(0x7f127c072290 legacy=0x7f127c0726d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:07:57.810 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:57.807+0000 7f127affd640 1 -- 172.21.15.104:0/2366319031 >> v1:172.21.15.104:6789/0 conn(0x7f127c141440 legacy=0x7f127c141890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:07:57.810 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:57.807+0000 7f127affd640 1 -- 172.21.15.104:0/2366319031 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f127c06e3d0 con 0x7f127c02c460 2024-03-28T14:07:57.810 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:57.807+0000 7f1279ffb640 1 -- 172.21.15.104:0/2366319031 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 223508764 0 0) 0x7f127c02c840 con 0x7f127c141440 2024-03-28T14:07:57.812 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:57.811+0000 7f127affd640 1 --2- 172.21.15.104:0/2366319031 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f127c02c460 0x7f127c070790 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f1264009870 tx=0x7f126401c410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ebc5ed126f8ac2e server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:07:57.812 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:57.811+0000 7f1279ffb640 1 -- 172.21.15.104:0/2366319031 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f12640b0070 con 0x7f127c02c460 2024-03-28T14:07:57.812 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:57.811+0000 7f1279ffb640 1 -- 172.21.15.104:0/2366319031 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f12640ab600 con 0x7f127c02c460 2024-03-28T14:07:57.812 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:57.811+0000 7f1279ffb640 1 -- 172.21.15.104:0/2366319031 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f12640ab9d0 con 0x7f127c02c460 2024-03-28T14:07:57.813 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:57.811+0000 7f12815ed640 1 -- 172.21.15.104:0/2366319031 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f127c02c460 msgr2=0x7f127c070790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:07:57.813 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:57.811+0000 7f12815ed640 1 --2- 172.21.15.104:0/2366319031 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f127c02c460 0x7f127c070790 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f1264009870 tx=0x7f126401c410 comp rx=0 tx=0).stop 2024-03-28T14:07:57.813 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:57.811+0000 7f12815ed640 1 -- 172.21.15.104:0/2366319031 shutdown_connections 2024-03-28T14:07:57.813 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:57.811+0000 7f12815ed640 1 --2- 172.21.15.104:0/2366319031 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f127c02c460 0x7f127c070790 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:07:57.813 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:57.811+0000 7f12815ed640 1 -- 172.21.15.104:0/2366319031 >> 172.21.15.104:0/2366319031 conn(0x7f127c0b60e0 msgr2=0x7f127c0b64f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T14:07:57.813 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:57.811+0000 7f12815ed640 1 -- 172.21.15.104:0/2366319031 shutdown_connections 2024-03-28T14:07:57.813 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:57.811+0000 7f12815ed640 1 -- 172.21.15.104:0/2366319031 wait complete. 2024-03-28T14:07:57.814 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:57.811+0000 7f12815ed640 1 Processor -- start 2024-03-28T14:07:57.814 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:57.811+0000 7f12815ed640 1 -- start start 2024-03-28T14:07:57.815 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:57.811+0000 7f12815ed640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f127c02c460 0x7f127c1338d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:07:57.815 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:57.811+0000 7f12815ed640 1 --2- >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f127c072290 0x7f127c1318a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:07:57.815 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:57.811+0000 7f12815ed640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f127c141440 0x7f127c13ae70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:07:57.816 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:57.811+0000 7f12815ed640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f127c155be0 con 0x7f127c141440 2024-03-28T14:07:57.816 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:57.811+0000 7f12815ed640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f127c155a60 con 0x7f127c02c460 2024-03-28T14:07:57.816 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:57.811+0000 7f12815ed640 1 -- --> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] -- mon_getmap magic: 0 -- 0x7f127c155d60 con 0x7f127c072290 2024-03-28T14:07:57.816 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:57.811+0000 7f127b7fe640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f127c141440 0x7f127c13ae70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:07:57.816 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:57.811+0000 7f127b7fe640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f127c141440 0x7f127c13ae70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:42186/0 (socket says 172.21.15.104:42186) 2024-03-28T14:07:57.816 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:57.811+0000 7f127b7fe640 1 -- 172.21.15.104:0/1123069309 learned_addr learned my addr 172.21.15.104:0/1123069309 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-03-28T14:07:57.816 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:57.811+0000 7f127a7fc640 1 --2- 172.21.15.104:0/1123069309 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f127c072290 0x7f127c1318a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:07:57.816 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:57.811+0000 7f127affd640 1 --2- 172.21.15.104:0/1123069309 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f127c02c460 0x7f127c1338d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:07:57.816 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:57.811+0000 7f127b7fe640 1 -- 172.21.15.104:0/1123069309 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f127c072290 msgr2=0x7f127c1318a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:07:57.816 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:57.811+0000 7f127b7fe640 1 --2- 172.21.15.104:0/1123069309 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f127c072290 0x7f127c1318a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:07:57.816 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:57.811+0000 7f127b7fe640 1 -- 172.21.15.104:0/1123069309 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f127c02c460 msgr2=0x7f127c1338d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:07:57.816 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:57.811+0000 7f127b7fe640 1 --2- 172.21.15.104:0/1123069309 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f127c02c460 0x7f127c1338d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:07:57.816 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:57.811+0000 7f127b7fe640 1 -- 172.21.15.104:0/1123069309 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f127c1e3710 con 0x7f127c141440 2024-03-28T14:07:57.816 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:57.811+0000 7f127b7fe640 1 --2- 172.21.15.104:0/1123069309 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f127c141440 0x7f127c13ae70 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f127400c780 tx=0x7f127400cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:07:57.817 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:57.815+0000 7f12737fe640 1 -- 172.21.15.104:0/1123069309 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1274003db0 con 0x7f127c141440 2024-03-28T14:07:57.817 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:57.815+0000 7f12737fe640 1 -- 172.21.15.104:0/1123069309 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f127400ce30 con 0x7f127c141440 2024-03-28T14:07:57.817 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:57.815+0000 7f12815ed640 1 -- 172.21.15.104:0/1123069309 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f127c135230 con 0x7f127c141440 2024-03-28T14:07:57.817 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:57.815+0000 7f12737fe640 1 -- 172.21.15.104:0/1123069309 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1274005290 con 0x7f127c141440 2024-03-28T14:07:57.817 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:57.815+0000 7f12815ed640 1 -- 172.21.15.104:0/1123069309 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f127c1357f0 con 0x7f127c141440 2024-03-28T14:07:57.818 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:57.815+0000 7f12815ed640 1 -- 172.21.15.104:0/1123069309 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f127c0687c0 con 0x7f127c141440 2024-03-28T14:07:57.818 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:57.815+0000 7f12737fe640 1 -- 172.21.15.104:0/1123069309 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) ==== 87803+0+0 (secure 0 0 0) 0x7f1274005430 con 0x7f127c141440 2024-03-28T14:07:57.818 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:57.815+0000 7f12737fe640 1 --2- 172.21.15.104:0/1123069309 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f125c067fc0 0x7f125c06a480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:07:57.818 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:57.815+0000 7f12737fe640 1 -- 172.21.15.104:0/1123069309 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(18..18 src has 1..18) ==== 5227+0+0 (secure 0 0 0) 0x7f1274086b10 con 0x7f127c141440 2024-03-28T14:07:57.818 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:57.815+0000 7f127affd640 1 --2- 172.21.15.104:0/1123069309 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f125c067fc0 0x7f125c06a480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:07:57.818 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:57.815+0000 7f127affd640 1 --2- 172.21.15.104:0/1123069309 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f125c067fc0 0x7f125c06a480 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f12640ad000 tx=0x7f12640a6000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:07:57.821 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:57.819+0000 7f12737fe640 1 -- 172.21.15.104:0/1123069309 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179117 (secure 0 0 0) 0x7f1274056f30 con 0x7f127c141440 2024-03-28T14:07:57.982 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:57.979+0000 7f12815ed640 1 -- 172.21.15.104:0/1123069309 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) -- 0x7f127c152e10 con 0x7f127c141440 2024-03-28T14:07:58.644 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:58.643+0000 7f12737fe640 1 -- 172.21.15.104:0/1123069309 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v19) ==== 170+0+0 (secure 0 0 0) 0x7f12740590d0 con 0x7f127c141440 2024-03-28T14:07:58.644 INFO:teuthology.orchestra.run.smithi104.stderr:pool 'cephfs_data' created 2024-03-28T14:07:58.648 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:58.647+0000 7f12815ed640 1 -- 172.21.15.104:0/1123069309 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f125c067fc0 msgr2=0x7f125c06a480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:07:58.648 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:58.647+0000 7f12815ed640 1 --2- 172.21.15.104:0/1123069309 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f125c067fc0 0x7f125c06a480 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f12640ad000 tx=0x7f12640a6000 comp rx=0 tx=0).stop 2024-03-28T14:07:58.648 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:58.647+0000 7f12815ed640 1 -- 172.21.15.104:0/1123069309 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f127c141440 msgr2=0x7f127c13ae70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:07:58.649 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:58.647+0000 7f12815ed640 1 --2- 172.21.15.104:0/1123069309 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f127c141440 0x7f127c13ae70 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f127400c780 tx=0x7f127400cc40 comp rx=0 tx=0).stop 2024-03-28T14:07:58.649 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:58.647+0000 7f12815ed640 1 -- 172.21.15.104:0/1123069309 shutdown_connections 2024-03-28T14:07:58.649 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:58.647+0000 7f12815ed640 1 --2- 172.21.15.104:0/1123069309 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f125c067fc0 0x7f125c06a480 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:07:58.649 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:58.647+0000 7f12815ed640 1 --2- 172.21.15.104:0/1123069309 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f127c141440 0x7f127c13ae70 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:07:58.649 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:58.647+0000 7f12815ed640 1 --2- 172.21.15.104:0/1123069309 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f127c072290 0x7f127c1318a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:07:58.649 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:58.647+0000 7f12815ed640 1 --2- 172.21.15.104:0/1123069309 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f127c02c460 0x7f127c1338d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:07:58.650 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:58.647+0000 7f12815ed640 1 -- 172.21.15.104:0/1123069309 >> 172.21.15.104:0/1123069309 conn(0x7f127c0b60e0 msgr2=0x7f127c136380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T14:07:58.650 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:58.647+0000 7f12815ed640 1 -- 172.21.15.104:0/1123069309 shutdown_connections 2024-03-28T14:07:58.650 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:58.647+0000 7f12815ed640 1 -- 172.21.15.104:0/1123069309 wait complete. 2024-03-28T14:07:58.685 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-03-28T14:07:58.807 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:58.807+0000 7f25dd2b0640 1 Processor -- start 2024-03-28T14:07:58.807 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:58.807+0000 7f25dd2b0640 1 -- start start 2024-03-28T14:07:58.809 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:58.807+0000 7f25dd2b0640 1 --2- >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f25d806c8f0 0x7f25d806f8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:07:58.809 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:58.807+0000 7f25dd2b0640 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f25d80ae3c0 0x7f25d80b3730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:07:58.809 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:58.807+0000 7f25dd2b0640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f25d80b4830 con 0x7f25d80b3c70 2024-03-28T14:07:58.809 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:58.807+0000 7f25dd2b0640 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 -- 0x7f25d806ccd0 con 0x7f25d80ae3c0 2024-03-28T14:07:58.809 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:58.807+0000 7f25dd2b0640 1 -- --> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] -- mon_getmap magic: 0 -- 0x7f25d80af660 con 0x7f25d806c8f0 2024-03-28T14:07:58.809 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:58.807+0000 7f25cffff640 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f25d80ae3c0 0x7f25d80b3730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:07:58.809 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:58.807+0000 7f25cffff640 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f25d80ae3c0 0x7f25d80b3730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:42200/0 (socket says 172.21.15.104:42200) 2024-03-28T14:07:58.809 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:58.807+0000 7f25cffff640 1 -- 172.21.15.104:0/2859068890 learned_addr learned my addr 172.21.15.104:0/2859068890 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-03-28T14:07:58.809 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:58.807+0000 7f25d6ffd640 1 --2- 172.21.15.104:0/2859068890 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f25d806c8f0 0x7f25d806f8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:07:58.809 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:58.807+0000 7f25cffff640 1 -- 172.21.15.104:0/2859068890 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f25d806c8f0 msgr2=0x7f25d806f8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T14:07:58.809 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:58.807+0000 7f25cffff640 1 --2- 172.21.15.104:0/2859068890 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f25d806c8f0 0x7f25d806f8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:07:58.809 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:58.807+0000 7f25cffff640 1 -- 172.21.15.104:0/2859068890 >> v1:172.21.15.171:6789/0 conn(0x7f25d80b3c70 legacy=0x7f25d8155780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:07:58.809 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:58.807+0000 7f25cffff640 1 -- 172.21.15.104:0/2859068890 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f25d80af940 con 0x7f25d80ae3c0 2024-03-28T14:07:58.810 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:58.807+0000 7f25d6ffd640 1 --2- 172.21.15.104:0/2859068890 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f25d806c8f0 0x7f25d806f8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T14:07:58.811 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:58.811+0000 7f25cffff640 1 --2- 172.21.15.104:0/2859068890 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f25d80ae3c0 0x7f25d80b3730 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f25c0009870 tx=0x7f25c001c410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=24194574ddd95980 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:07:58.812 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:58.811+0000 7f25d67fc640 1 -- 172.21.15.104:0/2859068890 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25c00ab610 con 0x7f25d80ae3c0 2024-03-28T14:07:58.812 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:58.811+0000 7f25d67fc640 1 -- 172.21.15.104:0/2859068890 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f25c00b0070 con 0x7f25d80ae3c0 2024-03-28T14:07:58.812 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:58.811+0000 7f25d67fc640 1 -- 172.21.15.104:0/2859068890 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25c00ab9d0 con 0x7f25d80ae3c0 2024-03-28T14:07:58.812 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:58.811+0000 7f25dd2b0640 1 -- 172.21.15.104:0/2859068890 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f25d80ae3c0 msgr2=0x7f25d80b3730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:07:58.813 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:58.811+0000 7f25dd2b0640 1 --2- 172.21.15.104:0/2859068890 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f25d80ae3c0 0x7f25d80b3730 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f25c0009870 tx=0x7f25c001c410 comp rx=0 tx=0).stop 2024-03-28T14:07:58.813 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:58.811+0000 7f25dd2b0640 1 -- 172.21.15.104:0/2859068890 shutdown_connections 2024-03-28T14:07:58.813 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:58.811+0000 7f25dd2b0640 1 --2- 172.21.15.104:0/2859068890 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f25d80ae3c0 0x7f25d80b3730 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:07:58.813 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:58.811+0000 7f25dd2b0640 1 --2- 172.21.15.104:0/2859068890 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f25d806c8f0 0x7f25d806f8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:07:58.813 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:58.811+0000 7f25dd2b0640 1 -- 172.21.15.104:0/2859068890 >> 172.21.15.104:0/2859068890 conn(0x7f25d80b8f00 msgr2=0x7f25d80b9310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T14:07:58.813 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:58.811+0000 7f25dd2b0640 1 -- 172.21.15.104:0/2859068890 shutdown_connections 2024-03-28T14:07:58.813 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:58.811+0000 7f25dd2b0640 1 -- 172.21.15.104:0/2859068890 wait complete. 2024-03-28T14:07:58.816 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:58.811+0000 7f25dd2b0640 1 Processor -- start 2024-03-28T14:07:58.816 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:58.811+0000 7f25dd2b0640 1 -- start start 2024-03-28T14:07:58.816 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:58.811+0000 7f25dd2b0640 1 --2- >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f25d806c8f0 0x7f25d809bcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:07:58.816 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:58.811+0000 7f25dd2b0640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f25d80ae3c0 0x7f25d8155820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:07:58.816 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:58.811+0000 7f25dd2b0640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f25d80b3c70 0x7f25d8155ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:07:58.816 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:58.811+0000 7f25dd2b0640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f25d80ae820 con 0x7f25d80b3c70 2024-03-28T14:07:58.816 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:58.811+0000 7f25dd2b0640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f25d80afe80 con 0x7f25d80ae3c0 2024-03-28T14:07:58.817 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:58.811+0000 7f25dd2b0640 1 -- --> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] -- mon_getmap magic: 0 -- 0x7f25d80af2e0 con 0x7f25d806c8f0 2024-03-28T14:07:58.817 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:58.811+0000 7f25d77fe640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f25d80b3c70 0x7f25d8155ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:07:58.817 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:58.811+0000 7f25d77fe640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f25d80b3c70 0x7f25d8155ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:42208/0 (socket says 172.21.15.104:42208) 2024-03-28T14:07:58.817 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:58.811+0000 7f25d77fe640 1 -- 172.21.15.104:0/1635101436 learned_addr learned my addr 172.21.15.104:0/1635101436 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-03-28T14:07:58.817 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:58.811+0000 7f25d6ffd640 1 --2- 172.21.15.104:0/1635101436 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f25d806c8f0 0x7f25d809bcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:07:58.817 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:58.811+0000 7f25cffff640 1 --2- 172.21.15.104:0/1635101436 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f25d80ae3c0 0x7f25d8155820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:07:58.817 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:58.811+0000 7f25d77fe640 1 -- 172.21.15.104:0/1635101436 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f25d806c8f0 msgr2=0x7f25d809bcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:07:58.817 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:58.811+0000 7f25d77fe640 1 --2- 172.21.15.104:0/1635101436 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f25d806c8f0 0x7f25d809bcb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:07:58.817 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:58.811+0000 7f25d77fe640 1 -- 172.21.15.104:0/1635101436 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f25d80ae3c0 msgr2=0x7f25d8155820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:07:58.817 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:58.811+0000 7f25d77fe640 1 --2- 172.21.15.104:0/1635101436 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f25d80ae3c0 0x7f25d8155820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:07:58.817 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:58.811+0000 7f25d77fe640 1 -- 172.21.15.104:0/1635101436 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f25d81e8990 con 0x7f25d80b3c70 2024-03-28T14:07:58.817 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:58.811+0000 7f25d77fe640 1 --2- 172.21.15.104:0/1635101436 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f25d80b3c70 0x7f25d8155ee0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f25d000c710 tx=0x7f25d000cbd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:07:58.817 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:58.815+0000 7f25cf7fe640 1 -- 172.21.15.104:0/1635101436 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25d0016020 con 0x7f25d80b3c70 2024-03-28T14:07:58.817 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:58.815+0000 7f25cf7fe640 1 -- 172.21.15.104:0/1635101436 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f25d0006d60 con 0x7f25d80b3c70 2024-03-28T14:07:58.818 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:58.815+0000 7f25dd2b0640 1 -- 172.21.15.104:0/1635101436 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f25d80b0710 con 0x7f25d80b3c70 2024-03-28T14:07:58.818 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:58.815+0000 7f25cf7fe640 1 -- 172.21.15.104:0/1635101436 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25d0003bc0 con 0x7f25d80b3c70 2024-03-28T14:07:58.818 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:58.815+0000 7f25dd2b0640 1 -- 172.21.15.104:0/1635101436 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f25d81e7e70 con 0x7f25d80b3c70 2024-03-28T14:07:58.818 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:58.815+0000 7f25dd2b0640 1 -- 172.21.15.104:0/1635101436 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f25d80687c0 con 0x7f25d80b3c70 2024-03-28T14:07:58.818 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:58.815+0000 7f25cf7fe640 1 -- 172.21.15.104:0/1635101436 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) ==== 87803+0+0 (secure 0 0 0) 0x7f25d000a850 con 0x7f25d80b3c70 2024-03-28T14:07:58.819 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:58.815+0000 7f25cf7fe640 1 --2- 172.21.15.104:0/1635101436 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f25b8067fc0 0x7f25b806a480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:07:58.819 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:58.815+0000 7f25cf7fe640 1 -- 172.21.15.104:0/1635101436 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(19..19 src has 1..19) ==== 5609+0+0 (secure 0 0 0) 0x7f25d0086150 con 0x7f25d80b3c70 2024-03-28T14:07:58.819 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:58.815+0000 7f25d6ffd640 1 --2- 172.21.15.104:0/1635101436 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f25b8067fc0 0x7f25b806a480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:07:58.819 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:58.815+0000 7f25d6ffd640 1 --2- 172.21.15.104:0/1635101436 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f25b8067fc0 0x7f25b806a480 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f25c800b930 tx=0x7f25c8008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:07:58.821 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:58.819+0000 7f25cf7fe640 1 -- 172.21.15.104:0/1635101436 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179117 (secure 0 0 0) 0x7f25d0020020 con 0x7f25d80b3c70 2024-03-28T14:07:58.858 DEBUG:teuthology.orchestra.run.smithi104:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-28T14:07:58.861 DEBUG:teuthology.orchestra.run.smithi171:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-28T14:07:59.024 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:59.023+0000 7f25dd2b0640 1 -- 172.21.15.104:0/1635101436 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) -- 0x7f25d80aeae0 con 0x7f25d80b3c70 2024-03-28T14:07:59.690 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:59.687+0000 7f25cf7fe640 1 -- 172.21.15.104:0/1635101436 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 2 and data pool 3 v4) ==== 172+0+0 (secure 0 0 0) 0x7f25d80b4830 con 0x7f25d80b3c70 2024-03-28T14:07:59.690 INFO:teuthology.orchestra.run.smithi104.stderr:new fs with metadata pool 2 and data pool 3 2024-03-28T14:07:59.697 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:59.695+0000 7f25dd2b0640 1 -- 172.21.15.104:0/1635101436 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f25b8067fc0 msgr2=0x7f25b806a480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:07:59.697 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:59.695+0000 7f25dd2b0640 1 --2- 172.21.15.104:0/1635101436 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f25b8067fc0 0x7f25b806a480 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f25c800b930 tx=0x7f25c8008040 comp rx=0 tx=0).stop 2024-03-28T14:07:59.697 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:59.695+0000 7f25dd2b0640 1 -- 172.21.15.104:0/1635101436 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f25d80b3c70 msgr2=0x7f25d8155ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:07:59.697 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:59.695+0000 7f25dd2b0640 1 --2- 172.21.15.104:0/1635101436 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f25d80b3c70 0x7f25d8155ee0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f25d000c710 tx=0x7f25d000cbd0 comp rx=0 tx=0).stop 2024-03-28T14:07:59.697 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:59.695+0000 7f25dd2b0640 1 -- 172.21.15.104:0/1635101436 shutdown_connections 2024-03-28T14:07:59.697 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:59.695+0000 7f25dd2b0640 1 --2- 172.21.15.104:0/1635101436 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f25b8067fc0 0x7f25b806a480 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:07:59.697 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:59.695+0000 7f25dd2b0640 1 --2- 172.21.15.104:0/1635101436 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f25d80b3c70 0x7f25d8155ee0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:07:59.698 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:59.695+0000 7f25dd2b0640 1 --2- 172.21.15.104:0/1635101436 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f25d80ae3c0 0x7f25d8155820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:07:59.698 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:59.695+0000 7f25dd2b0640 1 --2- 172.21.15.104:0/1635101436 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f25d806c8f0 0x7f25d809bcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:07:59.698 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:59.695+0000 7f25dd2b0640 1 -- 172.21.15.104:0/1635101436 >> 172.21.15.104:0/1635101436 conn(0x7f25d80b8f00 msgr2=0x7f25d8071a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T14:07:59.711 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:59.711+0000 7f25dd2b0640 1 -- 172.21.15.104:0/1635101436 shutdown_connections 2024-03-28T14:07:59.712 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:59.711+0000 7f25dd2b0640 1 -- 172.21.15.104:0/1635101436 wait complete. 2024-03-28T14:07:59.733 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-28T14:07:59.850 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:59.847+0000 7f957b254640 1 Processor -- start 2024-03-28T14:07:59.850 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:59.847+0000 7f957b254640 1 -- start start 2024-03-28T14:07:59.851 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:59.847+0000 7f957b254640 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f95740bd9e0 0x7f957414af30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:07:59.851 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:59.847+0000 7f957b254640 1 --2- >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f9574141b00 0x7f9574152080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:07:59.852 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:59.847+0000 7f957b254640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f957414b900 con 0x7f957414b470 2024-03-28T14:07:59.852 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:59.847+0000 7f957b254640 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 -- 0x7f95740b70b0 con 0x7f95740bd9e0 2024-03-28T14:07:59.852 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:59.847+0000 7f957b254640 1 -- --> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] -- mon_getmap magic: 0 -- 0x7f95740b5050 con 0x7f9574141b00 2024-03-28T14:07:59.852 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:59.847+0000 7f9578ff0640 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f95740bd9e0 0x7f957414af30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:07:59.852 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:59.847+0000 7f9573fff640 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f957414b470 0x7f95741413f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:47266/0 (socket says 172.21.15.104:47266) 2024-03-28T14:07:59.852 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:59.847+0000 7f9573fff640 1 -- 172.21.15.104:0/3941276517 learned_addr learned my addr 172.21.15.104:0/3941276517 (peer_addr_for_me v1:172.21.15.104:0/0) 2024-03-28T14:07:59.852 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:59.847+0000 7f9578ff0640 1 -- 172.21.15.104:0/3941276517 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f9574141b00 msgr2=0x7f9574152080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T14:07:59.852 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:59.847+0000 7f95797f1640 1 --2- 172.21.15.104:0/3941276517 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f9574141b00 0x7f9574152080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:07:59.852 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:59.851+0000 7f9578ff0640 1 --2- 172.21.15.104:0/3941276517 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f9574141b00 0x7f9574152080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:07:59.852 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:59.851+0000 7f95737fe640 1 -- 172.21.15.104:0/3941276517 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3323181707 0 0) 0x7f957414b900 con 0x7f957414b470 2024-03-28T14:07:59.852 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:59.851+0000 7f9578ff0640 1 -- 172.21.15.104:0/3941276517 >> v1:172.21.15.104:6789/0 conn(0x7f957414b470 legacy=0x7f95741413f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:07:59.853 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:59.851+0000 7f9578ff0640 1 -- 172.21.15.104:0/3941276517 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f95740bddc0 con 0x7f95740bd9e0 2024-03-28T14:07:59.853 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:59.851+0000 7f95797f1640 1 --2- 172.21.15.104:0/3941276517 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f9574141b00 0x7f9574152080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T14:07:59.855 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:59.851+0000 7f9578ff0640 1 --2- 172.21.15.104:0/3941276517 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f95740bd9e0 0x7f957414af30 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f95640097c0 tx=0x7f956401c350 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8da289e69d8f2cf4 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:07:59.855 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:59.851+0000 7f95737fe640 1 -- 172.21.15.104:0/3941276517 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95640b0070 con 0x7f95740bd9e0 2024-03-28T14:07:59.855 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:59.851+0000 7f95737fe640 1 -- 172.21.15.104:0/3941276517 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9564004230 con 0x7f95740bd9e0 2024-03-28T14:07:59.855 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:59.851+0000 7f95737fe640 1 -- 172.21.15.104:0/3941276517 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9564003620 con 0x7f95740bd9e0 2024-03-28T14:07:59.855 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:59.851+0000 7f957b254640 1 -- 172.21.15.104:0/3941276517 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f95740bd9e0 msgr2=0x7f957414af30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:07:59.855 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:59.851+0000 7f957b254640 1 --2- 172.21.15.104:0/3941276517 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f95740bd9e0 0x7f957414af30 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f95640097c0 tx=0x7f956401c350 comp rx=0 tx=0).stop 2024-03-28T14:07:59.856 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:59.851+0000 7f957b254640 1 -- 172.21.15.104:0/3941276517 shutdown_connections 2024-03-28T14:07:59.856 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:59.851+0000 7f957b254640 1 --2- 172.21.15.104:0/3941276517 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f9574141b00 0x7f9574152080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:07:59.856 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:59.851+0000 7f957b254640 1 --2- 172.21.15.104:0/3941276517 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f95740bd9e0 0x7f957414af30 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:07:59.856 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:59.851+0000 7f957b254640 1 -- 172.21.15.104:0/3941276517 >> 172.21.15.104:0/3941276517 conn(0x7f9574073530 msgr2=0x7f95740bd070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T14:07:59.856 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:59.851+0000 7f957b254640 1 -- 172.21.15.104:0/3941276517 shutdown_connections 2024-03-28T14:07:59.856 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:59.851+0000 7f957b254640 1 -- 172.21.15.104:0/3941276517 wait complete. 2024-03-28T14:07:59.858 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:59.855+0000 7f957b254640 1 Processor -- start 2024-03-28T14:07:59.858 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:59.855+0000 7f957b254640 1 -- start start 2024-03-28T14:07:59.858 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:59.855+0000 7f957b254640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f95740bd9e0 0x7f9574142dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:07:59.858 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:59.855+0000 7f957b254640 1 --2- >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f9574141b00 0x7f95741dff10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:07:59.859 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:59.855+0000 7f957b254640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f957414b470 0x7f95741e0450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:07:59.859 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:59.855+0000 7f957b254640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f95740b62c0 con 0x7f95740bd9e0 2024-03-28T14:07:59.859 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:59.855+0000 7f957b254640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f95740b6140 con 0x7f957414b470 2024-03-28T14:07:59.859 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:59.855+0000 7f957b254640 1 -- --> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] -- mon_getmap magic: 0 -- 0x7f95740b4940 con 0x7f9574141b00 2024-03-28T14:07:59.859 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:59.855+0000 7f9578ff0640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f95740bd9e0 0x7f9574142dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:07:59.859 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:59.855+0000 7f9578ff0640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f95740bd9e0 0x7f9574142dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:42222/0 (socket says 172.21.15.104:42222) 2024-03-28T14:07:59.859 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:59.855+0000 7f9578ff0640 1 -- 172.21.15.104:0/3049941074 learned_addr learned my addr 172.21.15.104:0/3049941074 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-03-28T14:07:59.859 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:59.855+0000 7f9573fff640 1 --2- 172.21.15.104:0/3049941074 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f9574141b00 0x7f95741dff10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:07:59.859 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:59.855+0000 7f9578ff0640 1 -- 172.21.15.104:0/3049941074 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f9574141b00 msgr2=0x7f95741dff10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:07:59.859 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:59.855+0000 7f9578ff0640 1 --2- 172.21.15.104:0/3049941074 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f9574141b00 0x7f95741dff10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:07:59.859 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:59.855+0000 7f9578ff0640 1 -- 172.21.15.104:0/3049941074 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f957414b470 msgr2=0x7f95741e0450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:07:59.859 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:59.855+0000 7f9578ff0640 1 --2- 172.21.15.104:0/3049941074 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f957414b470 0x7f95741e0450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:07:59.859 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:59.855+0000 7f9578ff0640 1 -- 172.21.15.104:0/3049941074 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9574144800 con 0x7f95740bd9e0 2024-03-28T14:07:59.859 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:59.855+0000 7f9573fff640 1 --2- 172.21.15.104:0/3049941074 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f9574141b00 0x7f95741dff10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T14:07:59.860 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:59.855+0000 7f9578ff0640 1 --2- 172.21.15.104:0/3049941074 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f95740bd9e0 0x7f9574142dd0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f9564019f80 tx=0x7f95640190f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:07:59.860 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:59.855+0000 7f95717fa640 1 -- 172.21.15.104:0/3049941074 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95640b9070 con 0x7f95740bd9e0 2024-03-28T14:07:59.860 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:59.855+0000 7f95717fa640 1 -- 172.21.15.104:0/3049941074 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f95640043e0 con 0x7f95740bd9e0 2024-03-28T14:07:59.860 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:59.855+0000 7f957b254640 1 -- 172.21.15.104:0/3049941074 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9574142200 con 0x7f95740bd9e0 2024-03-28T14:07:59.860 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:59.855+0000 7f95717fa640 1 -- 172.21.15.104:0/3049941074 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95640b0040 con 0x7f95740bd9e0 2024-03-28T14:07:59.860 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:59.855+0000 7f957b254640 1 -- 172.21.15.104:0/3049941074 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9574068de0 con 0x7f95740bd9e0 2024-03-28T14:07:59.860 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:59.855+0000 7f957b254640 1 -- 172.21.15.104:0/3049941074 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f95740687c0 con 0x7f95740bd9e0 2024-03-28T14:07:59.861 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:59.859+0000 7f95717fa640 1 -- 172.21.15.104:0/3049941074 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) ==== 87803+0+0 (secure 0 0 0) 0x7f95640bc070 con 0x7f95740bd9e0 2024-03-28T14:07:59.861 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:59.859+0000 7f95717fa640 1 --2- 172.21.15.104:0/3049941074 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f9560067ff0 0x7f956006a4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:07:59.861 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:59.859+0000 7f95717fa640 1 -- 172.21.15.104:0/3049941074 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f9564120a60 con 0x7f95740bd9e0 2024-03-28T14:07:59.861 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:59.859+0000 7f9573fff640 1 --2- 172.21.15.104:0/3049941074 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f9560067ff0 0x7f956006a4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:07:59.861 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:59.859+0000 7f9573fff640 1 --2- 172.21.15.104:0/3049941074 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f9560067ff0 0x7f956006a4b0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f9568007fd0 tx=0x7f9568009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:07:59.863 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:07:59.863+0000 7f95717fa640 1 -- 172.21.15.104:0/3049941074 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179117 (secure 0 0 0) 0x7f9564016090 con 0x7f95740bd9e0 2024-03-28T14:08:00.025 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:00.023+0000 7f957b254640 1 -- 172.21.15.104:0/3049941074 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f9574145120 con 0x7f95740bd9e0 2024-03-28T14:08:00.028 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:00.027+0000 7f95717fa640 1 -- 172.21.15.104:0/3049941074 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) ==== 74+0+16642 (secure 0 0 0) 0x7f95640f1380 con 0x7f95740bd9e0 2024-03-28T14:08:00.028 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-03-28T14:08:00.029 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":20,"fsid":"428a90f2-3351-435f-8fcf-f7e66c78a13e","created":"2024-03-28T14:07:34.862458+0000","modified":"2024-03-28T14:07:59.026677+0000","last_up_change":"2024-03-28T14:07:46.028122+0000","last_in_change":"2024-03-28T14:07:37.157486+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-28T14:07:47.622988+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-28T14:07:57.147450+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-28T14:07:57.990975+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"1cb70540-4e42-402f-a446-aaec4ca83007","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6800","nonce":3745690918},{"type":"v1","addr":"172.21.15.104:6801","nonce":3745690918}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":3745690918},{"type":"v1","addr":"172.21.15.104:6803","nonce":3745690918}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":3745690918},{"type":"v1","addr":"172.21.15.104:6807","nonce":3745690918}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":3745690918},{"type":"v1","addr":"172.21.15.104:6805","nonce":3745690918}]},"public_addr":"172.21.15.104:6801/3745690918","cluster_addr":"172.21.15.104:6803/3745690918","heartbeat_back_addr":"172.21.15.104:6807/3745690918","heartbeat_front_addr":"172.21.15.104:6805/3745690918","state":["exists","up"]},{"osd":1,"uuid":"3ab7afdd-b980-418c-81f2-004e16de3dca","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":19438674},{"type":"v1","addr":"172.21.15.104:6810","nonce":19438674}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":19438674},{"type":"v1","addr":"172.21.15.104:6814","nonce":19438674}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":19438674},{"type":"v1","addr":"172.21.15.104:6823","nonce":19438674}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":19438674},{"type":"v1","addr":"172.21.15.104:6819","nonce":19438674}]},"public_addr":"172.21.15.104:6810/19438674","cluster_addr":"172.21.15.104:6814/19438674","heartbeat_back_addr":"172.21.15.104:6823/19438674","heartbeat_front_addr":"172.21.15.104:6819/19438674","state":["exists","up"]},{"osd":2,"uuid":"3e2651c8-ecae-4a9e-a053-27240a79eac2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":2352012626},{"type":"v1","addr":"172.21.15.104:6825","nonce":2352012626}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6826","nonce":2352012626},{"type":"v1","addr":"172.21.15.104:6827","nonce":2352012626}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":2352012626},{"type":"v1","addr":"172.21.15.104:6831","nonce":2352012626}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6828","nonce":2352012626},{"type":"v1","addr":"172.21.15.104:6829","nonce":2352012626}]},"public_addr":"172.21.15.104:6825/2352012626","cluster_addr":"172.21.15.104:6827/2352012626","heartbeat_back_addr":"172.21.15.104:6831/2352012626","heartbeat_front_addr":"172.21.15.104:6829/2352012626","state":["exists","up"]},{"osd":3,"uuid":"925c7e0a-38ef-47cd-b288-8bec8456fb17","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6809","nonce":3568347381},{"type":"v1","addr":"172.21.15.104:6811","nonce":3568347381}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6813","nonce":3568347381},{"type":"v1","addr":"172.21.15.104:6815","nonce":3568347381}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":3568347381},{"type":"v1","addr":"172.21.15.104:6822","nonce":3568347381}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6817","nonce":3568347381},{"type":"v1","addr":"172.21.15.104:6818","nonce":3568347381}]},"public_addr":"172.21.15.104:6811/3568347381","cluster_addr":"172.21.15.104:6815/3568347381","heartbeat_back_addr":"172.21.15.104:6822/3568347381","heartbeat_front_addr":"172.21.15.104:6818/3568347381","state":["exists","up"]},{"osd":4,"uuid":"4a8fac86-e595-4a0e-96e3-57edb3cc3f57","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6800","nonce":2107001077},{"type":"v1","addr":"172.21.15.171:6801","nonce":2107001077}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6802","nonce":2107001077},{"type":"v1","addr":"172.21.15.171:6803","nonce":2107001077}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6806","nonce":2107001077},{"type":"v1","addr":"172.21.15.171:6807","nonce":2107001077}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6804","nonce":2107001077},{"type":"v1","addr":"172.21.15.171:6805","nonce":2107001077}]},"public_addr":"172.21.15.171:6801/2107001077","cluster_addr":"172.21.15.171:6803/2107001077","heartbeat_back_addr":"172.21.15.171:6807/2107001077","heartbeat_front_addr":"172.21.15.171:6805/2107001077","state":["exists","up"]},{"osd":5,"uuid":"e48d50b0-beae-40ae-b15a-9a84dbfe88ab","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6808","nonce":2934826925},{"type":"v1","addr":"172.21.15.171:6810","nonce":2934826925}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6812","nonce":2934826925},{"type":"v1","addr":"172.21.15.171:6814","nonce":2934826925}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6819","nonce":2934826925},{"type":"v1","addr":"172.21.15.171:6821","nonce":2934826925}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6816","nonce":2934826925},{"type":"v1","addr":"172.21.15.171:6817","nonce":2934826925}]},"public_addr":"172.21.15.171:6810/2934826925","cluster_addr":"172.21.15.171:6814/2934826925","heartbeat_back_addr":"172.21.15.171:6821/2934826925","heartbeat_front_addr":"172.21.15.171:6817/2934826925","state":["exists","up"]},{"osd":6,"uuid":"5e7c3292-9936-4f35-b267-aed9d4e49323","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6824","nonce":436742919},{"type":"v1","addr":"172.21.15.171:6825","nonce":436742919}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6826","nonce":436742919},{"type":"v1","addr":"172.21.15.171:6827","nonce":436742919}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6830","nonce":436742919},{"type":"v1","addr":"172.21.15.171:6831","nonce":436742919}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6828","nonce":436742919},{"type":"v1","addr":"172.21.15.171:6829","nonce":436742919}]},"public_addr":"172.21.15.171:6825/436742919","cluster_addr":"172.21.15.171:6827/436742919","heartbeat_back_addr":"172.21.15.171:6831/436742919","heartbeat_front_addr":"172.21.15.171:6829/436742919","state":["exists","up"]},{"osd":7,"uuid":"339b8414-d7f0-4fb0-a446-efaf635cb7ff","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6809","nonce":2055631175},{"type":"v1","addr":"172.21.15.171:6811","nonce":2055631175}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6813","nonce":2055631175},{"type":"v1","addr":"172.21.15.171:6815","nonce":2055631175}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6822","nonce":2055631175},{"type":"v1","addr":"172.21.15.171:6823","nonce":2055631175}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6818","nonce":2055631175},{"type":"v1","addr":"172.21.15.171:6820","nonce":2055631175}]},"public_addr":"172.21.15.171:6811/2055631175","cluster_addr":"172.21.15.171:6815/2055631175","heartbeat_back_addr":"172.21.15.171:6823/2055631175","heartbeat_front_addr":"172.21.15.171:6820/2055631175","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T14:07:42.668881+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T14:07:42.708397+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T14:07:42.634191+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T14:07:42.733662+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T14:07:42.651541+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T14:07:42.783181+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T14:07:42.766310+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T14:07:42.670797+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-28T14:08:00.030 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:00.027+0000 7f957b254640 1 -- 172.21.15.104:0/3049941074 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f9560067ff0 msgr2=0x7f956006a4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:00.031 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:00.027+0000 7f957b254640 1 --2- 172.21.15.104:0/3049941074 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f9560067ff0 0x7f956006a4b0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f9568007fd0 tx=0x7f9568009040 comp rx=0 tx=0).stop 2024-03-28T14:08:00.031 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:00.027+0000 7f957b254640 1 -- 172.21.15.104:0/3049941074 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f95740bd9e0 msgr2=0x7f9574142dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:00.031 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:00.027+0000 7f957b254640 1 --2- 172.21.15.104:0/3049941074 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f95740bd9e0 0x7f9574142dd0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f9564019f80 tx=0x7f95640190f0 comp rx=0 tx=0).stop 2024-03-28T14:08:00.031 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:00.027+0000 7f957b254640 1 -- 172.21.15.104:0/3049941074 shutdown_connections 2024-03-28T14:08:00.031 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:00.027+0000 7f957b254640 1 --2- 172.21.15.104:0/3049941074 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f957414b470 0x7f95741e0450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:00.031 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:00.027+0000 7f957b254640 1 --2- 172.21.15.104:0/3049941074 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f9574141b00 0x7f95741dff10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:00.031 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:00.027+0000 7f957b254640 1 --2- 172.21.15.104:0/3049941074 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f9560067ff0 0x7f956006a4b0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:00.031 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:00.027+0000 7f957b254640 1 --2- 172.21.15.104:0/3049941074 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f95740bd9e0 0x7f9574142dd0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:00.031 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:00.027+0000 7f957b254640 1 -- 172.21.15.104:0/3049941074 >> 172.21.15.104:0/3049941074 conn(0x7f9574073530 msgr2=0x7f95740bbf50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T14:08:00.031 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:00.027+0000 7f957b254640 1 -- 172.21.15.104:0/3049941074 shutdown_connections 2024-03-28T14:08:00.031 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:00.027+0000 7f957b254640 1 -- 172.21.15.104:0/3049941074 wait complete. 2024-03-28T14:08:00.048 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-28T14:08:00.164 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:00.163+0000 7fd280170640 1 Processor -- start 2024-03-28T14:08:00.164 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:00.163+0000 7fd280170640 1 -- start start 2024-03-28T14:08:00.165 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:00.163+0000 7fd280170640 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fd2780b5330 0x7fd278148e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:00.165 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:00.163+0000 7fd280170640 1 --2- >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fd27809de70 0x7fd278149390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:00.165 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:00.163+0000 7fd280170640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd2780bad50 con 0x7fd2780ba970 2024-03-28T14:08:00.165 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:00.163+0000 7fd280170640 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 -- 0x7fd27809d790 con 0x7fd2780b5330 2024-03-28T14:08:00.165 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:00.163+0000 7fd280170640 1 -- --> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] -- mon_getmap magic: 0 -- 0x7fd27809ea30 con 0x7fd27809de70 2024-03-28T14:08:00.165 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:00.163+0000 7fd27df0c640 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7fd2780ba970 0x7fd2780b4c20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:47282/0 (socket says 172.21.15.104:47282) 2024-03-28T14:08:00.166 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:00.163+0000 7fd27df0c640 1 -- 172.21.15.104:0/1715273234 learned_addr learned my addr 172.21.15.104:0/1715273234 (peer_addr_for_me v1:172.21.15.104:0/0) 2024-03-28T14:08:00.166 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:00.163+0000 7fd27d70b640 1 --2- 172.21.15.104:0/1715273234 >> v2:172.21.15.104:3300/0 conn(0x7fd2780b5330 0x7fd278148e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:00.166 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:00.163+0000 7fd27e70d640 1 --2- 172.21.15.104:0/1715273234 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fd27809de70 0x7fd278149390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:00.166 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:00.163+0000 7fd27cf0a640 1 -- 172.21.15.104:0/1715273234 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3797238441 0 0) 0x7fd2780bad50 con 0x7fd2780ba970 2024-03-28T14:08:00.166 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:00.163+0000 7fd27cf0a640 1 -- 172.21.15.104:0/1715273234 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd264003530 con 0x7fd2780ba970 2024-03-28T14:08:00.166 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:00.163+0000 7fd27d70b640 1 -- 172.21.15.104:0/1715273234 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fd27809de70 msgr2=0x7fd278149390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T14:08:00.166 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:00.163+0000 7fd27d70b640 1 --2- 172.21.15.104:0/1715273234 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fd27809de70 0x7fd278149390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:00.166 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:00.163+0000 7fd27d70b640 1 -- 172.21.15.104:0/1715273234 >> v1:172.21.15.104:6789/0 conn(0x7fd2780ba970 legacy=0x7fd2780b4c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:00.166 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:00.163+0000 7fd27d70b640 1 -- 172.21.15.104:0/1715273234 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd27809e3e0 con 0x7fd2780b5330 2024-03-28T14:08:00.167 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:00.167+0000 7fd27d70b640 1 --2- 172.21.15.104:0/1715273234 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd2780b5330 0x7fd278148e50 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fd2680097c0 tx=0x7fd26801c410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ca4a1ebae729f30b server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:08:00.167 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:00.167+0000 7fd27cf0a640 1 -- 172.21.15.104:0/1715273234 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2680b0070 con 0x7fd2780b5330 2024-03-28T14:08:00.167 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:00.167+0000 7fd27cf0a640 1 -- 172.21.15.104:0/1715273234 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd2680ab600 con 0x7fd2780b5330 2024-03-28T14:08:00.168 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:00.167+0000 7fd27cf0a640 1 -- 172.21.15.104:0/1715273234 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2680ab9d0 con 0x7fd2780b5330 2024-03-28T14:08:00.168 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:00.167+0000 7fd280170640 1 -- 172.21.15.104:0/1715273234 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd2780b5330 msgr2=0x7fd278148e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:00.168 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:00.167+0000 7fd280170640 1 --2- 172.21.15.104:0/1715273234 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd2780b5330 0x7fd278148e50 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fd2680097c0 tx=0x7fd26801c410 comp rx=0 tx=0).stop 2024-03-28T14:08:00.169 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:00.167+0000 7fd280170640 1 -- 172.21.15.104:0/1715273234 shutdown_connections 2024-03-28T14:08:00.169 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:00.167+0000 7fd280170640 1 --2- 172.21.15.104:0/1715273234 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fd27809de70 0x7fd278149390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:00.169 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:00.167+0000 7fd280170640 1 --2- 172.21.15.104:0/1715273234 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd2780b5330 0x7fd278148e50 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:00.169 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:00.167+0000 7fd280170640 1 -- 172.21.15.104:0/1715273234 >> 172.21.15.104:0/1715273234 conn(0x7fd27806cd70 msgr2=0x7fd2780ba000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T14:08:00.169 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:00.167+0000 7fd280170640 1 -- 172.21.15.104:0/1715273234 shutdown_connections 2024-03-28T14:08:00.169 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:00.167+0000 7fd280170640 1 -- 172.21.15.104:0/1715273234 wait complete. 2024-03-28T14:08:00.172 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:00.167+0000 7fd280170640 1 Processor -- start 2024-03-28T14:08:00.172 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:00.167+0000 7fd280170640 1 -- start start 2024-03-28T14:08:00.172 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:00.167+0000 7fd280170640 1 --2- >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fd27809de70 0x7fd27814a4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:00.172 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:00.167+0000 7fd280170640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd2780b5330 0x7fd2781de510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:00.172 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:00.167+0000 7fd280170640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fd2780ba970 0x7fd278144150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:00.172 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:00.167+0000 7fd280170640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7fd278152e60 con 0x7fd2780b5330 2024-03-28T14:08:00.172 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:00.167+0000 7fd280170640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7fd278152ce0 con 0x7fd2780ba970 2024-03-28T14:08:00.172 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:00.167+0000 7fd280170640 1 -- --> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] -- mon_getmap magic: 0 -- 0x7fd278152fe0 con 0x7fd27809de70 2024-03-28T14:08:00.173 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:00.167+0000 7fd27d70b640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd2780b5330 0x7fd2781de510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:00.173 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:00.167+0000 7fd27d70b640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd2780b5330 0x7fd2781de510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:42244/0 (socket says 172.21.15.104:42244) 2024-03-28T14:08:00.173 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:00.167+0000 7fd27d70b640 1 -- 172.21.15.104:0/630734701 learned_addr learned my addr 172.21.15.104:0/630734701 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-03-28T14:08:00.173 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:00.167+0000 7fd27df0c640 1 --2- 172.21.15.104:0/630734701 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fd27809de70 0x7fd27814a4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:00.173 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:00.167+0000 7fd27d70b640 1 -- 172.21.15.104:0/630734701 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fd27809de70 msgr2=0x7fd27814a4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:00.173 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:00.167+0000 7fd27d70b640 1 --2- 172.21.15.104:0/630734701 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fd27809de70 0x7fd27814a4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:00.173 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:00.167+0000 7fd27e70d640 1 --2- 172.21.15.104:0/630734701 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fd2780ba970 0x7fd278144150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:00.173 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:00.167+0000 7fd27d70b640 1 -- 172.21.15.104:0/630734701 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fd2780ba970 msgr2=0x7fd278144150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:00.173 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:00.167+0000 7fd27d70b640 1 --2- 172.21.15.104:0/630734701 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fd2780ba970 0x7fd278144150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:00.174 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:00.167+0000 7fd27d70b640 1 -- 172.21.15.104:0/630734701 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd2781496d0 con 0x7fd2780b5330 2024-03-28T14:08:00.174 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:00.167+0000 7fd27d70b640 1 --2- 172.21.15.104:0/630734701 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd2780b5330 0x7fd2781de510 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fd2680ae040 tx=0x7fd2680191b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:08:00.174 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:00.167+0000 7fd26e7fc640 1 -- 172.21.15.104:0/630734701 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2680b0040 con 0x7fd2780b5330 2024-03-28T14:08:00.174 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:00.167+0000 7fd26e7fc640 1 -- 172.21.15.104:0/630734701 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd2680b9070 con 0x7fd2780b5330 2024-03-28T14:08:00.174 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:00.167+0000 7fd280170640 1 -- 172.21.15.104:0/630734701 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd278149130 con 0x7fd2780b5330 2024-03-28T14:08:00.174 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:00.167+0000 7fd26e7fc640 1 -- 172.21.15.104:0/630734701 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2680b4850 con 0x7fd2780b5330 2024-03-28T14:08:00.174 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:00.167+0000 7fd280170640 1 -- 172.21.15.104:0/630734701 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd27814c230 con 0x7fd2780b5330 2024-03-28T14:08:00.174 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:00.171+0000 7fd280170640 1 -- 172.21.15.104:0/630734701 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd2780687c0 con 0x7fd2780b5330 2024-03-28T14:08:00.175 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:00.171+0000 7fd26e7fc640 1 -- 172.21.15.104:0/630734701 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) ==== 87803+0+0 (secure 0 0 0) 0x7fd2680b49f0 con 0x7fd2780b5330 2024-03-28T14:08:00.175 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:00.171+0000 7fd26e7fc640 1 --2- 172.21.15.104:0/630734701 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7fd25c067fc0 0x7fd25c06a480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:00.175 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:00.171+0000 7fd26e7fc640 1 -- 172.21.15.104:0/630734701 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7fd2681203e0 con 0x7fd2780b5330 2024-03-28T14:08:00.175 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:00.171+0000 7fd27df0c640 1 --2- 172.21.15.104:0/630734701 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7fd25c067fc0 0x7fd25c06a480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:00.175 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:00.171+0000 7fd27df0c640 1 --2- 172.21.15.104:0/630734701 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7fd25c067fc0 0x7fd25c06a480 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fd274007fd0 tx=0x7fd274009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:08:00.177 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:00.175+0000 7fd26e7fc640 1 -- 172.21.15.104:0/630734701 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179117 (secure 0 0 0) 0x7fd268016090 con 0x7fd2780b5330 2024-03-28T14:08:00.340 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:00.339+0000 7fd280170640 1 -- 172.21.15.104:0/630734701 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fd27809aa10 con 0x7fd2780b5330 2024-03-28T14:08:00.341 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:00.339+0000 7fd26e7fc640 1 -- 172.21.15.104:0/630734701 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) ==== 74+0+16642 (secure 0 0 0) 0x7fd2680f0c20 con 0x7fd2780b5330 2024-03-28T14:08:00.342 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-03-28T14:08:00.342 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":20,"fsid":"428a90f2-3351-435f-8fcf-f7e66c78a13e","created":"2024-03-28T14:07:34.862458+0000","modified":"2024-03-28T14:07:59.026677+0000","last_up_change":"2024-03-28T14:07:46.028122+0000","last_in_change":"2024-03-28T14:07:37.157486+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-28T14:07:47.622988+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-28T14:07:57.147450+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-28T14:07:57.990975+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"1cb70540-4e42-402f-a446-aaec4ca83007","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6800","nonce":3745690918},{"type":"v1","addr":"172.21.15.104:6801","nonce":3745690918}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":3745690918},{"type":"v1","addr":"172.21.15.104:6803","nonce":3745690918}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":3745690918},{"type":"v1","addr":"172.21.15.104:6807","nonce":3745690918}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":3745690918},{"type":"v1","addr":"172.21.15.104:6805","nonce":3745690918}]},"public_addr":"172.21.15.104:6801/3745690918","cluster_addr":"172.21.15.104:6803/3745690918","heartbeat_back_addr":"172.21.15.104:6807/3745690918","heartbeat_front_addr":"172.21.15.104:6805/3745690918","state":["exists","up"]},{"osd":1,"uuid":"3ab7afdd-b980-418c-81f2-004e16de3dca","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":19438674},{"type":"v1","addr":"172.21.15.104:6810","nonce":19438674}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":19438674},{"type":"v1","addr":"172.21.15.104:6814","nonce":19438674}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":19438674},{"type":"v1","addr":"172.21.15.104:6823","nonce":19438674}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":19438674},{"type":"v1","addr":"172.21.15.104:6819","nonce":19438674}]},"public_addr":"172.21.15.104:6810/19438674","cluster_addr":"172.21.15.104:6814/19438674","heartbeat_back_addr":"172.21.15.104:6823/19438674","heartbeat_front_addr":"172.21.15.104:6819/19438674","state":["exists","up"]},{"osd":2,"uuid":"3e2651c8-ecae-4a9e-a053-27240a79eac2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":2352012626},{"type":"v1","addr":"172.21.15.104:6825","nonce":2352012626}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6826","nonce":2352012626},{"type":"v1","addr":"172.21.15.104:6827","nonce":2352012626}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":2352012626},{"type":"v1","addr":"172.21.15.104:6831","nonce":2352012626}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6828","nonce":2352012626},{"type":"v1","addr":"172.21.15.104:6829","nonce":2352012626}]},"public_addr":"172.21.15.104:6825/2352012626","cluster_addr":"172.21.15.104:6827/2352012626","heartbeat_back_addr":"172.21.15.104:6831/2352012626","heartbeat_front_addr":"172.21.15.104:6829/2352012626","state":["exists","up"]},{"osd":3,"uuid":"925c7e0a-38ef-47cd-b288-8bec8456fb17","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6809","nonce":3568347381},{"type":"v1","addr":"172.21.15.104:6811","nonce":3568347381}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6813","nonce":3568347381},{"type":"v1","addr":"172.21.15.104:6815","nonce":3568347381}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":3568347381},{"type":"v1","addr":"172.21.15.104:6822","nonce":3568347381}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6817","nonce":3568347381},{"type":"v1","addr":"172.21.15.104:6818","nonce":3568347381}]},"public_addr":"172.21.15.104:6811/3568347381","cluster_addr":"172.21.15.104:6815/3568347381","heartbeat_back_addr":"172.21.15.104:6822/3568347381","heartbeat_front_addr":"172.21.15.104:6818/3568347381","state":["exists","up"]},{"osd":4,"uuid":"4a8fac86-e595-4a0e-96e3-57edb3cc3f57","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6800","nonce":2107001077},{"type":"v1","addr":"172.21.15.171:6801","nonce":2107001077}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6802","nonce":2107001077},{"type":"v1","addr":"172.21.15.171:6803","nonce":2107001077}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6806","nonce":2107001077},{"type":"v1","addr":"172.21.15.171:6807","nonce":2107001077}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6804","nonce":2107001077},{"type":"v1","addr":"172.21.15.171:6805","nonce":2107001077}]},"public_addr":"172.21.15.171:6801/2107001077","cluster_addr":"172.21.15.171:6803/2107001077","heartbeat_back_addr":"172.21.15.171:6807/2107001077","heartbeat_front_addr":"172.21.15.171:6805/2107001077","state":["exists","up"]},{"osd":5,"uuid":"e48d50b0-beae-40ae-b15a-9a84dbfe88ab","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6808","nonce":2934826925},{"type":"v1","addr":"172.21.15.171:6810","nonce":2934826925}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6812","nonce":2934826925},{"type":"v1","addr":"172.21.15.171:6814","nonce":2934826925}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6819","nonce":2934826925},{"type":"v1","addr":"172.21.15.171:6821","nonce":2934826925}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6816","nonce":2934826925},{"type":"v1","addr":"172.21.15.171:6817","nonce":2934826925}]},"public_addr":"172.21.15.171:6810/2934826925","cluster_addr":"172.21.15.171:6814/2934826925","heartbeat_back_addr":"172.21.15.171:6821/2934826925","heartbeat_front_addr":"172.21.15.171:6817/2934826925","state":["exists","up"]},{"osd":6,"uuid":"5e7c3292-9936-4f35-b267-aed9d4e49323","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6824","nonce":436742919},{"type":"v1","addr":"172.21.15.171:6825","nonce":436742919}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6826","nonce":436742919},{"type":"v1","addr":"172.21.15.171:6827","nonce":436742919}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6830","nonce":436742919},{"type":"v1","addr":"172.21.15.171:6831","nonce":436742919}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6828","nonce":436742919},{"type":"v1","addr":"172.21.15.171:6829","nonce":436742919}]},"public_addr":"172.21.15.171:6825/436742919","cluster_addr":"172.21.15.171:6827/436742919","heartbeat_back_addr":"172.21.15.171:6831/436742919","heartbeat_front_addr":"172.21.15.171:6829/436742919","state":["exists","up"]},{"osd":7,"uuid":"339b8414-d7f0-4fb0-a446-efaf635cb7ff","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6809","nonce":2055631175},{"type":"v1","addr":"172.21.15.171:6811","nonce":2055631175}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6813","nonce":2055631175},{"type":"v1","addr":"172.21.15.171:6815","nonce":2055631175}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6822","nonce":2055631175},{"type":"v1","addr":"172.21.15.171:6823","nonce":2055631175}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6818","nonce":2055631175},{"type":"v1","addr":"172.21.15.171:6820","nonce":2055631175}]},"public_addr":"172.21.15.171:6811/2055631175","cluster_addr":"172.21.15.171:6815/2055631175","heartbeat_back_addr":"172.21.15.171:6823/2055631175","heartbeat_front_addr":"172.21.15.171:6820/2055631175","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T14:07:42.668881+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T14:07:42.708397+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T14:07:42.634191+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T14:07:42.733662+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T14:07:42.651541+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T14:07:42.783181+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T14:07:42.766310+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T14:07:42.670797+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-28T14:08:00.344 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:00.343+0000 7fd280170640 1 -- 172.21.15.104:0/630734701 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7fd25c067fc0 msgr2=0x7fd25c06a480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:00.344 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:00.343+0000 7fd280170640 1 --2- 172.21.15.104:0/630734701 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7fd25c067fc0 0x7fd25c06a480 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fd274007fd0 tx=0x7fd274009040 comp rx=0 tx=0).stop 2024-03-28T14:08:00.344 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:00.343+0000 7fd280170640 1 -- 172.21.15.104:0/630734701 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd2780b5330 msgr2=0x7fd2781de510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:00.344 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:00.343+0000 7fd280170640 1 --2- 172.21.15.104:0/630734701 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd2780b5330 0x7fd2781de510 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fd2680ae040 tx=0x7fd2680191b0 comp rx=0 tx=0).stop 2024-03-28T14:08:00.344 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:00.343+0000 7fd280170640 1 -- 172.21.15.104:0/630734701 shutdown_connections 2024-03-28T14:08:00.344 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:00.343+0000 7fd280170640 1 --2- 172.21.15.104:0/630734701 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fd2780ba970 0x7fd278144150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:00.344 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:00.343+0000 7fd280170640 1 --2- 172.21.15.104:0/630734701 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7fd25c067fc0 0x7fd25c06a480 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:00.344 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:00.343+0000 7fd280170640 1 --2- 172.21.15.104:0/630734701 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd2780b5330 0x7fd2781de510 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:00.344 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:00.343+0000 7fd280170640 1 --2- 172.21.15.104:0/630734701 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fd27809de70 0x7fd27814a4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:00.344 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:00.343+0000 7fd280170640 1 -- 172.21.15.104:0/630734701 >> 172.21.15.104:0/630734701 conn(0x7fd27806cd70 msgr2=0x7fd2780b9fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T14:08:00.344 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:00.343+0000 7fd280170640 1 -- 172.21.15.104:0/630734701 shutdown_connections 2024-03-28T14:08:00.345 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:00.343+0000 7fd280170640 1 -- 172.21.15.104:0/630734701 wait complete. 2024-03-28T14:08:00.361 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-03-28T14:08:00.486 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:00.483+0000 7f7c76a85640 1 Processor -- start 2024-03-28T14:08:00.486 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:00.483+0000 7f7c76a85640 1 -- start start 2024-03-28T14:08:00.488 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:00.483+0000 7f7c76a85640 1 --2- >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f7c700b4aa0 0x7f7c700af660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:00.488 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:00.483+0000 7f7c76a85640 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f7c70147bf0 0x7f7c700b22d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:00.488 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:00.483+0000 7f7c76a85640 1 --2- >> v2:172.21.15.171:3300/0 conn(0x7f7c700b2810 0x7f7c7006c690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:00.488 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:00.483+0000 7f7c76a85640 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 -- 0x7f7c70148030 con 0x7f7c70147bf0 2024-03-28T14:08:00.488 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:00.483+0000 7f7c76a85640 1 -- --> v2:172.21.15.171:3300/0 -- mon_getmap magic: 0 -- 0x7f7c700b4e80 con 0x7f7c700b2810 2024-03-28T14:08:00.488 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:00.483+0000 7f7c76a85640 1 -- --> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] -- mon_getmap magic: 0 -- 0x7f7c700b0890 con 0x7f7c700b4aa0 2024-03-28T14:08:00.488 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:00.483+0000 7f7c6ffff640 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f7c70147bf0 0x7f7c700b22d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:00.488 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:00.487+0000 7f7c6ffff640 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f7c70147bf0 0x7f7c700b22d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:42246/0 (socket says 172.21.15.104:42246) 2024-03-28T14:08:00.488 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:00.487+0000 7f7c6ffff640 1 -- 172.21.15.104:0/4294252078 learned_addr learned my addr 172.21.15.104:0/4294252078 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-03-28T14:08:00.488 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:00.487+0000 7f7c74821640 1 --2- 172.21.15.104:0/4294252078 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f7c700b4aa0 0x7f7c700af660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:00.488 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:00.487+0000 7f7c75022640 1 --2- 172.21.15.104:0/4294252078 >> v2:172.21.15.171:3300/0 conn(0x7f7c700b2810 0x7f7c7006c690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:00.488 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:00.487+0000 7f7c6ffff640 1 -- 172.21.15.104:0/4294252078 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f7c700b4aa0 msgr2=0x7f7c700af660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T14:08:00.489 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:00.487+0000 7f7c6ffff640 1 --2- 172.21.15.104:0/4294252078 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f7c700b4aa0 0x7f7c700af660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:00.489 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:00.487+0000 7f7c6ffff640 1 -- 172.21.15.104:0/4294252078 >> v2:172.21.15.171:3300/0 conn(0x7f7c700b2810 msgr2=0x7f7c7006c690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T14:08:00.489 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:00.487+0000 7f7c6ffff640 1 --2- 172.21.15.104:0/4294252078 >> v2:172.21.15.171:3300/0 conn(0x7f7c700b2810 0x7f7c7006c690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:00.489 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:00.487+0000 7f7c6ffff640 1 -- 172.21.15.104:0/4294252078 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7c700ae0a0 con 0x7f7c70147bf0 2024-03-28T14:08:00.489 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:00.487+0000 7f7c6ffff640 1 --2- 172.21.15.104:0/4294252078 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7c70147bf0 0x7f7c700b22d0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f7c700732a0 tx=0x7f7c6401c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b270d9a053fece1f server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:08:00.490 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:00.487+0000 7f7c6f7fe640 1 -- 172.21.15.104:0/4294252078 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c640b2070 con 0x7f7c70147bf0 2024-03-28T14:08:00.490 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:00.487+0000 7f7c6f7fe640 1 -- 172.21.15.104:0/4294252078 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7c640a9070 con 0x7f7c70147bf0 2024-03-28T14:08:00.490 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:00.487+0000 7f7c6f7fe640 1 -- 172.21.15.104:0/4294252078 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c640ad880 con 0x7f7c70147bf0 2024-03-28T14:08:00.497 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:00.487+0000 7f7c76a85640 1 -- 172.21.15.104:0/4294252078 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7c70147bf0 msgr2=0x7f7c700b22d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:00.497 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:00.487+0000 7f7c76a85640 1 --2- 172.21.15.104:0/4294252078 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7c70147bf0 0x7f7c700b22d0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f7c700732a0 tx=0x7f7c6401c490 comp rx=0 tx=0).stop 2024-03-28T14:08:00.497 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:00.487+0000 7f7c76a85640 1 -- 172.21.15.104:0/4294252078 shutdown_connections 2024-03-28T14:08:00.497 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:00.487+0000 7f7c76a85640 1 --2- 172.21.15.104:0/4294252078 >> v2:172.21.15.171:3300/0 conn(0x7f7c700b2810 0x7f7c7006c690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:00.497 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:00.487+0000 7f7c76a85640 1 --2- 172.21.15.104:0/4294252078 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7c70147bf0 0x7f7c700b22d0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:00.497 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:00.487+0000 7f7c76a85640 1 --2- 172.21.15.104:0/4294252078 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f7c700b4aa0 0x7f7c700af660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:00.497 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:00.487+0000 7f7c76a85640 1 -- 172.21.15.104:0/4294252078 >> 172.21.15.104:0/4294252078 conn(0x7f7c700b8f00 msgr2=0x7f7c700b9310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T14:08:00.497 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:00.487+0000 7f7c76a85640 1 -- 172.21.15.104:0/4294252078 shutdown_connections 2024-03-28T14:08:00.497 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:00.487+0000 7f7c76a85640 1 -- 172.21.15.104:0/4294252078 wait complete. 2024-03-28T14:08:00.497 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:00.491+0000 7f7c76a85640 1 Processor -- start 2024-03-28T14:08:00.497 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:00.491+0000 7f7c76a85640 1 -- start start 2024-03-28T14:08:00.497 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:00.491+0000 7f7c76a85640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7c700b2810 0x7f7c7013e240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:00.498 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:00.491+0000 7f7c76a85640 1 --2- >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f7c700b4aa0 0x7f7c70142090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:00.498 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:00.491+0000 7f7c76a85640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f7c70147bf0 0x7f7c701425d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:00.498 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:00.491+0000 7f7c76a85640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f7c700aeb20 con 0x7f7c700b2810 2024-03-28T14:08:00.498 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:00.491+0000 7f7c76a85640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f7c700ae9a0 con 0x7f7c70147bf0 2024-03-28T14:08:00.498 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:00.491+0000 7f7c76a85640 1 -- --> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] -- mon_getmap magic: 0 -- 0x7f7c70150f30 con 0x7f7c700b4aa0 2024-03-28T14:08:00.498 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:00.491+0000 7f7c74821640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7c700b2810 0x7f7c7013e240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:00.498 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:00.491+0000 7f7c74821640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7c700b2810 0x7f7c7013e240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:42256/0 (socket says 172.21.15.104:42256) 2024-03-28T14:08:00.498 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:00.491+0000 7f7c74821640 1 -- 172.21.15.104:0/3591070289 learned_addr learned my addr 172.21.15.104:0/3591070289 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-03-28T14:08:00.498 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:00.491+0000 7f7c75022640 1 --2- 172.21.15.104:0/3591070289 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f7c70147bf0 0x7f7c701425d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:00.498 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:00.491+0000 7f7c6ffff640 1 --2- 172.21.15.104:0/3591070289 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f7c700b4aa0 0x7f7c70142090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:00.498 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:00.491+0000 7f7c74821640 1 -- 172.21.15.104:0/3591070289 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f7c700b4aa0 msgr2=0x7f7c70142090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:00.498 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:00.491+0000 7f7c74821640 1 --2- 172.21.15.104:0/3591070289 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f7c700b4aa0 0x7f7c70142090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:00.498 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:00.491+0000 7f7c74821640 1 -- 172.21.15.104:0/3591070289 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f7c70147bf0 msgr2=0x7f7c701425d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:00.499 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:00.491+0000 7f7c74821640 1 --2- 172.21.15.104:0/3591070289 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f7c70147bf0 0x7f7c701425d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:00.499 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:00.491+0000 7f7c74821640 1 -- 172.21.15.104:0/3591070289 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7c7009d750 con 0x7f7c700b2810 2024-03-28T14:08:00.499 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:00.491+0000 7f7c75022640 1 --2- 172.21.15.104:0/3591070289 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f7c70147bf0 0x7f7c701425d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T14:08:00.499 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:00.491+0000 7f7c74821640 1 --2- 172.21.15.104:0/3591070289 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7c700b2810 0x7f7c7013e240 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f7c60007f20 tx=0x7f7c6000e570 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:08:00.499 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:00.491+0000 7f7c6d7fa640 1 -- 172.21.15.104:0/3591070289 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c60014620 con 0x7f7c700b2810 2024-03-28T14:08:00.499 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:00.491+0000 7f7c6d7fa640 1 -- 172.21.15.104:0/3591070289 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7c60019070 con 0x7f7c700b2810 2024-03-28T14:08:00.499 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:00.491+0000 7f7c6d7fa640 1 -- 172.21.15.104:0/3591070289 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c60014a00 con 0x7f7c700b2810 2024-03-28T14:08:00.499 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:00.491+0000 7f7c76a85640 1 -- 172.21.15.104:0/3591070289 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7c70144ac0 con 0x7f7c700b2810 2024-03-28T14:08:00.499 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:00.491+0000 7f7c76a85640 1 -- 172.21.15.104:0/3591070289 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7c70143720 con 0x7f7c700b2810 2024-03-28T14:08:00.499 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:00.491+0000 7f7c76a85640 1 -- 172.21.15.104:0/3591070289 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7c70143d70 con 0x7f7c700b2810 2024-03-28T14:08:00.499 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:00.491+0000 7f7c6d7fa640 1 -- 172.21.15.104:0/3591070289 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) ==== 87803+0+0 (secure 0 0 0) 0x7f7c60014ba0 con 0x7f7c700b2810 2024-03-28T14:08:00.499 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:00.495+0000 7f7c6d7fa640 1 --2- 172.21.15.104:0/3591070289 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f7c5c067fc0 0x7f7c5c06a480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:00.499 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:00.495+0000 7f7c6d7fa640 1 -- 172.21.15.104:0/3591070289 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f7c60086d50 con 0x7f7c700b2810 2024-03-28T14:08:00.499 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:00.495+0000 7f7c6ffff640 1 --2- 172.21.15.104:0/3591070289 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f7c5c067fc0 0x7f7c5c06a480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:00.500 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:00.495+0000 7f7c6ffff640 1 --2- 172.21.15.104:0/3591070289 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f7c5c067fc0 0x7f7c5c06a480 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f7c640af000 tx=0x7f7c64008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:08:00.500 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:00.499+0000 7f7c6d7fa640 1 -- 172.21.15.104:0/3591070289 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179117 (secure 0 0 0) 0x7f7c6008a030 con 0x7f7c700b2810 2024-03-28T14:08:00.691 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:00.687+0000 7f7c76a85640 1 -- 172.21.15.104:0/3591070289 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) -- 0x7f7c700687c0 con 0x7f7c700b2810 2024-03-28T14:08:01.696 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:01.695+0000 7f7c6d7fa640 1 -- 172.21.15.104:0/3591070289 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v6) ==== 118+0+0 (secure 0 0 0) 0x7f7c7009d750 con 0x7f7c700b2810 2024-03-28T14:08:01.699 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:01.695+0000 7f7c76a85640 1 -- 172.21.15.104:0/3591070289 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f7c5c067fc0 msgr2=0x7f7c5c06a480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:01.699 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:01.695+0000 7f7c76a85640 1 --2- 172.21.15.104:0/3591070289 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f7c5c067fc0 0x7f7c5c06a480 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f7c640af000 tx=0x7f7c64008040 comp rx=0 tx=0).stop 2024-03-28T14:08:01.699 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:01.695+0000 7f7c76a85640 1 -- 172.21.15.104:0/3591070289 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7c700b2810 msgr2=0x7f7c7013e240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:01.699 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:01.695+0000 7f7c76a85640 1 --2- 172.21.15.104:0/3591070289 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7c700b2810 0x7f7c7013e240 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f7c60007f20 tx=0x7f7c6000e570 comp rx=0 tx=0).stop 2024-03-28T14:08:01.700 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:01.695+0000 7f7c76a85640 1 -- 172.21.15.104:0/3591070289 shutdown_connections 2024-03-28T14:08:01.700 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:01.695+0000 7f7c76a85640 1 --2- 172.21.15.104:0/3591070289 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f7c70147bf0 0x7f7c701425d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:01.700 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:01.695+0000 7f7c76a85640 1 --2- 172.21.15.104:0/3591070289 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f7c700b4aa0 0x7f7c70142090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:01.700 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:01.695+0000 7f7c76a85640 1 --2- 172.21.15.104:0/3591070289 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f7c5c067fc0 0x7f7c5c06a480 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:01.700 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:01.695+0000 7f7c76a85640 1 --2- 172.21.15.104:0/3591070289 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7c700b2810 0x7f7c7013e240 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:01.700 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:01.699+0000 7f7c76a85640 1 -- 172.21.15.104:0/3591070289 >> 172.21.15.104:0/3591070289 conn(0x7f7c700b8f00 msgr2=0x7f7c700b92e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T14:08:01.700 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:01.699+0000 7f7c76a85640 1 -- 172.21.15.104:0/3591070289 shutdown_connections 2024-03-28T14:08:01.700 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:01.699+0000 7f7c76a85640 1 -- 172.21.15.104:0/3591070289 wait complete. 2024-03-28T14:08:01.722 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-03-28T14:08:01.722 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-03-28T14:08:01.851 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:01.847+0000 7f721fccc640 1 Processor -- start 2024-03-28T14:08:01.851 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:01.847+0000 7f721fccc640 1 -- start start 2024-03-28T14:08:01.852 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:01.851+0000 7f721fccc640 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f7218072bf0 0x7f72180b1e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:01.852 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:01.851+0000 7f721fccc640 1 --2- >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f7218071be0 0x7f72180b12f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:01.852 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:01.851+0000 7f721fccc640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7218072020 con 0x7f72180b1830 2024-03-28T14:08:01.852 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:01.851+0000 7f721fccc640 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 -- 0x7f7218073250 con 0x7f7218072bf0 2024-03-28T14:08:01.852 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:01.851+0000 7f721fccc640 1 -- --> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] -- mon_getmap magic: 0 -- 0x7f7218072fd0 con 0x7f7218071be0 2024-03-28T14:08:01.852 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:01.851+0000 7f721da68640 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f7218072bf0 0x7f72180b1e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:01.853 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:01.851+0000 7f721da68640 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f7218072bf0 0x7f72180b1e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:42260/0 (socket says 172.21.15.104:42260) 2024-03-28T14:08:01.853 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:01.851+0000 7f721da68640 1 -- 172.21.15.104:0/1654825546 learned_addr learned my addr 172.21.15.104:0/1654825546 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-03-28T14:08:01.853 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:01.851+0000 7f721d267640 1 --2- 172.21.15.104:0/1654825546 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f7218071be0 0x7f72180b12f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:01.853 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:01.851+0000 7f721da68640 1 -- 172.21.15.104:0/1654825546 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f7218071be0 msgr2=0x7f72180b12f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T14:08:01.853 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:01.851+0000 7f721da68640 1 --2- 172.21.15.104:0/1654825546 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f7218071be0 0x7f72180b12f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:01.853 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:01.851+0000 7f721da68640 1 -- 172.21.15.104:0/1654825546 >> v1:172.21.15.171:6789/0 conn(0x7f72180b1830 legacy=0x7f7218145ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T14:08:01.853 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:01.851+0000 7f721da68640 1 -- 172.21.15.104:0/1654825546 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f72180734a0 con 0x7f7218072bf0 2024-03-28T14:08:01.853 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:01.851+0000 7f721d267640 1 --2- 172.21.15.104:0/1654825546 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f7218071be0 0x7f72180b12f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T14:08:01.855 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:01.851+0000 7f721da68640 1 --2- 172.21.15.104:0/1654825546 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7218072bf0 0x7f72180b1e80 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f7210009870 tx=0x7f721001c400 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1b71409d62cda3d4 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:08:01.855 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:01.851+0000 7f721ca66640 1 -- 172.21.15.104:0/1654825546 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72100b0070 con 0x7f7218072bf0 2024-03-28T14:08:01.855 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:01.851+0000 7f721ca66640 1 -- 172.21.15.104:0/1654825546 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f72100ab600 con 0x7f7218072bf0 2024-03-28T14:08:01.855 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:01.851+0000 7f721ca66640 1 -- 172.21.15.104:0/1654825546 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72100ab9d0 con 0x7f7218072bf0 2024-03-28T14:08:01.856 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:01.855+0000 7f721fccc640 1 -- 172.21.15.104:0/1654825546 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7218072bf0 msgr2=0x7f72180b1e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:01.856 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:01.855+0000 7f721fccc640 1 --2- 172.21.15.104:0/1654825546 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7218072bf0 0x7f72180b1e80 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f7210009870 tx=0x7f721001c400 comp rx=0 tx=0).stop 2024-03-28T14:08:01.856 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:01.855+0000 7f721fccc640 1 -- 172.21.15.104:0/1654825546 shutdown_connections 2024-03-28T14:08:01.856 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:01.855+0000 7f721fccc640 1 --2- 172.21.15.104:0/1654825546 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f7218071be0 0x7f72180b12f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:01.856 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:01.855+0000 7f721fccc640 1 --2- 172.21.15.104:0/1654825546 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7218072bf0 0x7f72180b1e80 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:01.856 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:01.855+0000 7f721fccc640 1 -- 172.21.15.104:0/1654825546 >> 172.21.15.104:0/1654825546 conn(0x7f72180b7b10 msgr2=0x7f72180b7f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T14:08:01.856 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:01.855+0000 7f721fccc640 1 -- 172.21.15.104:0/1654825546 shutdown_connections 2024-03-28T14:08:01.856 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:01.855+0000 7f721fccc640 1 -- 172.21.15.104:0/1654825546 wait complete. 2024-03-28T14:08:01.859 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:01.855+0000 7f721fccc640 1 Processor -- start 2024-03-28T14:08:01.859 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:01.855+0000 7f721fccc640 1 -- start start 2024-03-28T14:08:01.859 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:01.855+0000 7f721fccc640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f7218071be0 0x7f7218140060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:01.859 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:01.855+0000 7f721fccc640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7218072bf0 0x7f72181e34a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:01.859 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:01.855+0000 7f721fccc640 1 --2- >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f72180b1830 0x7f72181e39e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:01.859 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:01.855+0000 7f721fccc640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f7218152830 con 0x7f7218072bf0 2024-03-28T14:08:01.859 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:01.855+0000 7f721fccc640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f72181526b0 con 0x7f7218071be0 2024-03-28T14:08:01.860 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:01.855+0000 7f721fccc640 1 -- --> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] -- mon_getmap magic: 0 -- 0x7f72181529b0 con 0x7f72180b1830 2024-03-28T14:08:01.860 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:01.855+0000 7f721d267640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7218072bf0 0x7f72181e34a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:01.860 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:01.855+0000 7f721d267640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7218072bf0 0x7f72181e34a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:42262/0 (socket says 172.21.15.104:42262) 2024-03-28T14:08:01.860 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:01.855+0000 7f721d267640 1 -- 172.21.15.104:0/1277937061 learned_addr learned my addr 172.21.15.104:0/1277937061 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-03-28T14:08:01.860 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:01.855+0000 7f721e269640 1 --2- 172.21.15.104:0/1277937061 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f72180b1830 0x7f72181e39e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:01.860 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:01.855+0000 7f721da68640 1 --2- 172.21.15.104:0/1277937061 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f7218071be0 0x7f7218140060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:01.860 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:01.855+0000 7f721d267640 1 -- 172.21.15.104:0/1277937061 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f72180b1830 msgr2=0x7f72181e39e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:01.860 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:01.855+0000 7f721d267640 1 --2- 172.21.15.104:0/1277937061 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f72180b1830 0x7f72181e39e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:01.860 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:01.855+0000 7f721d267640 1 -- 172.21.15.104:0/1277937061 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f7218071be0 msgr2=0x7f7218140060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:01.860 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:01.855+0000 7f721d267640 1 --2- 172.21.15.104:0/1277937061 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f7218071be0 0x7f7218140060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:01.861 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:01.855+0000 7f721d267640 1 -- 172.21.15.104:0/1277937061 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f72181412a0 con 0x7f7218072bf0 2024-03-28T14:08:01.861 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:01.855+0000 7f721d267640 1 --2- 172.21.15.104:0/1277937061 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7218072bf0 0x7f72181e34a0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f720800c7c0 tx=0x7f720800cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:08:01.861 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:01.855+0000 7f720e7fc640 1 -- 172.21.15.104:0/1277937061 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7208007a30 con 0x7f7218072bf0 2024-03-28T14:08:01.861 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:01.855+0000 7f720e7fc640 1 -- 172.21.15.104:0/1277937061 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7208007bd0 con 0x7f7218072bf0 2024-03-28T14:08:01.861 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:01.855+0000 7f721fccc640 1 -- 172.21.15.104:0/1277937061 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7218141580 con 0x7f7218072bf0 2024-03-28T14:08:01.861 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:01.855+0000 7f720e7fc640 1 -- 172.21.15.104:0/1277937061 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72080113f0 con 0x7f7218072bf0 2024-03-28T14:08:01.861 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:01.855+0000 7f721fccc640 1 -- 172.21.15.104:0/1277937061 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f72180a0d40 con 0x7f7218072bf0 2024-03-28T14:08:01.861 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:01.855+0000 7f721fccc640 1 -- 172.21.15.104:0/1277937061 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f72180687c0 con 0x7f7218072bf0 2024-03-28T14:08:01.862 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:01.859+0000 7f720e7fc640 1 -- 172.21.15.104:0/1277937061 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) ==== 87803+0+0 (secure 0 0 0) 0x7f7208011590 con 0x7f7218072bf0 2024-03-28T14:08:01.862 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:01.859+0000 7f720e7fc640 1 --2- 172.21.15.104:0/1277937061 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f71fc067fc0 0x7f71fc06a480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:01.862 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:01.859+0000 7f720e7fc640 1 -- 172.21.15.104:0/1277937061 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f7208086a60 con 0x7f7218072bf0 2024-03-28T14:08:01.862 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:01.859+0000 7f721da68640 1 --2- 172.21.15.104:0/1277937061 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f71fc067fc0 0x7f71fc06a480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:01.863 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:01.859+0000 7f721da68640 1 --2- 172.21.15.104:0/1277937061 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f71fc067fc0 0x7f71fc06a480 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f72100ad000 tx=0x7f72100a6000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:08:01.863 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:01.863+0000 7f720e7fc640 1 -- 172.21.15.104:0/1277937061 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179117 (secure 0 0 0) 0x7f72181412a0 con 0x7f7218072bf0 2024-03-28T14:08:02.055 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:02.051+0000 7f721fccc640 1 -- 172.21.15.104:0/1277937061 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) -- 0x7f721806c060 con 0x7f7218072bf0 2024-03-28T14:08:02.706 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:02.703+0000 7f720e7fc640 1 -- 172.21.15.104:0/1277937061 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v7) ==== 122+0+0 (secure 0 0 0) 0x7f7208015020 con 0x7f7218072bf0 2024-03-28T14:08:02.708 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:02.707+0000 7f721fccc640 1 -- 172.21.15.104:0/1277937061 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f71fc067fc0 msgr2=0x7f71fc06a480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:02.709 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:02.707+0000 7f721fccc640 1 --2- 172.21.15.104:0/1277937061 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f71fc067fc0 0x7f71fc06a480 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f72100ad000 tx=0x7f72100a6000 comp rx=0 tx=0).stop 2024-03-28T14:08:02.709 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:02.707+0000 7f721fccc640 1 -- 172.21.15.104:0/1277937061 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7218072bf0 msgr2=0x7f72181e34a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:02.709 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:02.707+0000 7f721fccc640 1 --2- 172.21.15.104:0/1277937061 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7218072bf0 0x7f72181e34a0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f720800c7c0 tx=0x7f720800cc80 comp rx=0 tx=0).stop 2024-03-28T14:08:02.709 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:02.707+0000 7f721fccc640 1 -- 172.21.15.104:0/1277937061 shutdown_connections 2024-03-28T14:08:02.709 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:02.707+0000 7f721fccc640 1 --2- 172.21.15.104:0/1277937061 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f72180b1830 0x7f72181e39e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:02.709 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:02.707+0000 7f721fccc640 1 --2- 172.21.15.104:0/1277937061 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f71fc067fc0 0x7f71fc06a480 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:02.709 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:02.707+0000 7f721fccc640 1 --2- 172.21.15.104:0/1277937061 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7218072bf0 0x7f72181e34a0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:02.709 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:02.707+0000 7f721fccc640 1 --2- 172.21.15.104:0/1277937061 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f7218071be0 0x7f7218140060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:02.709 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:02.707+0000 7f721fccc640 1 -- 172.21.15.104:0/1277937061 >> 172.21.15.104:0/1277937061 conn(0x7f72180b7b10 msgr2=0x7f721806e9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T14:08:02.709 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:02.707+0000 7f721fccc640 1 -- 172.21.15.104:0/1277937061 shutdown_connections 2024-03-28T14:08:02.709 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:02.707+0000 7f721fccc640 1 -- 172.21.15.104:0/1277937061 wait complete. 2024-03-28T14:08:02.727 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-03-28T14:08:02.727 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T14:08:02.771 INFO:tasks.ceph.mds.a.smithi104.stderr:2024-03-28T14:08:02.771+0000 7fc78ff0f640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-28T14:08:02.840 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:02.839+0000 7fc77f675640 1 Processor -- start 2024-03-28T14:08:02.840 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:02.839+0000 7fc77f675640 1 -- start start 2024-03-28T14:08:02.840 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:02.839+0000 7fc77f675640 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fc77802c5e0 0x7fc778148a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:02.840 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:02.839+0000 7fc77f675640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc7780b89a0 con 0x7fc7780b7c70 2024-03-28T14:08:02.841 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:02.839+0000 7fc77f675640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc7780b8c40 con 0x7fc77802c200 2024-03-28T14:08:02.841 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:02.839+0000 7fc77f675640 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 -- 0x7fc7780b8fb0 con 0x7fc77802c5e0 2024-03-28T14:08:02.841 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:02.839+0000 7fc77cc10640 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fc77802c5e0 0x7fc778148a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:02.841 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:02.839+0000 7fc77dc12640 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7fc7780b7c70 0x7fc77814fe20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:47294/0 (socket says 172.21.15.104:47294) 2024-03-28T14:08:02.841 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:02.839+0000 7fc77dc12640 1 -- 172.21.15.104:0/3785810263 learned_addr learned my addr 172.21.15.104:0/3785810263 (peer_addr_for_me v1:172.21.15.104:0/0) 2024-03-28T14:08:02.841 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:02.839+0000 7fc77cc10640 1 -- 172.21.15.104:0/3785810263 >> v1:172.21.15.171:6789/0 conn(0x7fc77802c200 legacy=0x7fc77813f390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T14:08:02.841 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:02.839+0000 7fc77cc10640 1 -- 172.21.15.104:0/3785810263 >> v1:172.21.15.104:6789/0 conn(0x7fc7780b7c70 legacy=0x7fc77814fe20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:02.841 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:02.839+0000 7fc77cc10640 1 -- 172.21.15.104:0/3785810263 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc77813faa0 con 0x7fc77802c5e0 2024-03-28T14:08:02.844 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:02.843+0000 7fc77cc10640 1 --2- 172.21.15.104:0/3785810263 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc77802c5e0 0x7fc778148a90 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fc7740097c0 tx=0x7fc77401c340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6bd5fc5eac0105ac server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:08:02.844 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:02.843+0000 7fc76bfff640 1 -- 172.21.15.104:0/3785810263 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7740b0070 con 0x7fc77802c5e0 2024-03-28T14:08:02.844 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:02.843+0000 7fc76bfff640 1 -- 172.21.15.104:0/3785810263 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc7740ab600 con 0x7fc77802c5e0 2024-03-28T14:08:02.844 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:02.843+0000 7fc76bfff640 1 -- 172.21.15.104:0/3785810263 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7740ab9d0 con 0x7fc77802c5e0 2024-03-28T14:08:02.844 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:02.843+0000 7fc77f675640 1 -- 172.21.15.104:0/3785810263 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc77802c5e0 msgr2=0x7fc778148a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:02.845 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:02.843+0000 7fc77f675640 1 --2- 172.21.15.104:0/3785810263 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc77802c5e0 0x7fc778148a90 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fc7740097c0 tx=0x7fc77401c340 comp rx=0 tx=0).stop 2024-03-28T14:08:02.845 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:02.843+0000 7fc77f675640 1 -- 172.21.15.104:0/3785810263 shutdown_connections 2024-03-28T14:08:02.845 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:02.843+0000 7fc77f675640 1 --2- 172.21.15.104:0/3785810263 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc77802c5e0 0x7fc778148a90 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:02.845 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:02.843+0000 7fc77f675640 1 -- 172.21.15.104:0/3785810263 >> 172.21.15.104:0/3785810263 conn(0x7fc77806db20 msgr2=0x7fc77806b5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T14:08:02.845 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:02.843+0000 7fc77f675640 1 -- 172.21.15.104:0/3785810263 shutdown_connections 2024-03-28T14:08:02.845 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:02.843+0000 7fc77f675640 1 -- 172.21.15.104:0/3785810263 wait complete. 2024-03-28T14:08:02.848 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:02.843+0000 7fc77f675640 1 Processor -- start 2024-03-28T14:08:02.848 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:02.843+0000 7fc77f675640 1 -- start start 2024-03-28T14:08:02.848 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:02.843+0000 7fc77f675640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc77802c200 0x7fc778071a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:02.848 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:02.843+0000 7fc77f675640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fc77802c5e0 0x7fc778132340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:02.848 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:02.843+0000 7fc77f675640 1 --2- >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fc7780b7c70 0x7fc778132880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:02.848 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:02.843+0000 7fc77f675640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7fc778153f70 con 0x7fc77802c200 2024-03-28T14:08:02.848 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:02.843+0000 7fc77f675640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7fc7780b15d0 con 0x7fc77802c5e0 2024-03-28T14:08:02.848 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:02.843+0000 7fc77f675640 1 -- --> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] -- mon_getmap magic: 0 -- 0x7fc7781540f0 con 0x7fc7780b7c70 2024-03-28T14:08:02.848 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:02.843+0000 7fc77d411640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc77802c200 0x7fc778071a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:02.848 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:02.843+0000 7fc77d411640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc77802c200 0x7fc778071a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:42278/0 (socket says 172.21.15.104:42278) 2024-03-28T14:08:02.848 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:02.843+0000 7fc77d411640 1 -- 172.21.15.104:0/449465429 learned_addr learned my addr 172.21.15.104:0/449465429 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-03-28T14:08:02.848 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:02.843+0000 7fc77dc12640 1 --2- 172.21.15.104:0/449465429 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fc7780b7c70 0x7fc778132880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:02.848 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:02.843+0000 7fc77cc10640 1 --2- 172.21.15.104:0/449465429 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fc77802c5e0 0x7fc778132340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:02.849 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:02.843+0000 7fc77d411640 1 -- 172.21.15.104:0/449465429 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fc7780b7c70 msgr2=0x7fc778132880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:02.849 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:02.843+0000 7fc77d411640 1 --2- 172.21.15.104:0/449465429 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fc7780b7c70 0x7fc778132880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:02.849 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:02.843+0000 7fc77d411640 1 -- 172.21.15.104:0/449465429 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fc77802c5e0 msgr2=0x7fc778132340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:02.849 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:02.843+0000 7fc77d411640 1 --2- 172.21.15.104:0/449465429 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fc77802c5e0 0x7fc778132340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:02.849 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:02.843+0000 7fc77d411640 1 -- 172.21.15.104:0/449465429 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc778131e40 con 0x7fc77802c200 2024-03-28T14:08:02.849 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:02.843+0000 7fc77cc10640 1 --2- 172.21.15.104:0/449465429 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fc77802c5e0 0x7fc778132340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T14:08:02.849 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:02.843+0000 7fc77d411640 1 --2- 172.21.15.104:0/449465429 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc77802c200 0x7fc778071a10 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fc76c00b410 tx=0x7fc76c00b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:08:02.849 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:02.843+0000 7fc769ffb640 1 -- 172.21.15.104:0/449465429 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc76c015030 con 0x7fc77802c200 2024-03-28T14:08:02.849 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:02.843+0000 7fc769ffb640 1 -- 172.21.15.104:0/449465429 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc76c004020 con 0x7fc77802c200 2024-03-28T14:08:02.849 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:02.843+0000 7fc77f675640 1 -- 172.21.15.104:0/449465429 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc778146660 con 0x7fc77802c200 2024-03-28T14:08:02.849 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:02.843+0000 7fc769ffb640 1 -- 172.21.15.104:0/449465429 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc76c004450 con 0x7fc77802c200 2024-03-28T14:08:02.849 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:02.843+0000 7fc77f675640 1 -- 172.21.15.104:0/449465429 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc7781473d0 con 0x7fc77802c200 2024-03-28T14:08:02.849 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:02.847+0000 7fc77f675640 1 -- 172.21.15.104:0/449465429 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc7780687c0 con 0x7fc77802c200 2024-03-28T14:08:02.850 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:02.847+0000 7fc769ffb640 1 -- 172.21.15.104:0/449465429 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) ==== 87803+0+0 (secure 0 0 0) 0x7fc76c018070 con 0x7fc77802c200 2024-03-28T14:08:02.850 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:02.847+0000 7fc769ffb640 1 --2- 172.21.15.104:0/449465429 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7fc75c067fc0 0x7fc75c06a480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:02.850 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:02.847+0000 7fc769ffb640 1 -- 172.21.15.104:0/449465429 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7fc76c085fa0 con 0x7fc77802c200 2024-03-28T14:08:02.850 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:02.847+0000 7fc77cc10640 1 --2- 172.21.15.104:0/449465429 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7fc75c067fc0 0x7fc75c06a480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:02.850 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:02.847+0000 7fc77cc10640 1 --2- 172.21.15.104:0/449465429 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7fc75c067fc0 0x7fc75c06a480 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fc7740ad000 tx=0x7fc774008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:08:02.853 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:02.851+0000 7fc769ffb640 1 -- 172.21.15.104:0/449465429 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179117 (secure 0 0 0) 0x7fc76c0567e0 con 0x7fc77802c200 2024-03-28T14:08:03.051 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:03.051+0000 7fc77f675640 1 -- 172.21.15.104:0/449465429 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fc7780b1010 con 0x7fc77802c200 2024-03-28T14:08:03.052 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:03.051+0000 7fc769ffb640 1 -- 172.21.15.104:0/449465429 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 7 v7) ==== 93+0+3533 (secure 0 0 0) 0x7fc76c0560e0 con 0x7fc77802c200 2024-03-28T14:08:03.052 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-03-28T14:08:03.052 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":7,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4229,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.171:6835/1775867467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6834","nonce":1775867467},{"type":"v1","addr":"172.21.15.171:6835","nonce":1775867467}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":7,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T14:07:59.027458+0000","modified":"2024-03-28T14:08:02.057923+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4245},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4245":{"gid":4245,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.104:6833/3184673325","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":3184673325},{"type":"v1","addr":"172.21.15.104:6833","nonce":3184673325}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4245,"qdb_cluster":[4245]},"id":1}]} 2024-03-28T14:08:03.053 INFO:teuthology.orchestra.run.smithi104.stderr:dumped fsmap epoch 7 2024-03-28T14:08:03.054 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:03.051+0000 7fc77f675640 1 -- 172.21.15.104:0/449465429 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7fc75c067fc0 msgr2=0x7fc75c06a480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:03.054 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:03.051+0000 7fc77f675640 1 --2- 172.21.15.104:0/449465429 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7fc75c067fc0 0x7fc75c06a480 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fc7740ad000 tx=0x7fc774008040 comp rx=0 tx=0).stop 2024-03-28T14:08:03.055 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:03.051+0000 7fc77f675640 1 -- 172.21.15.104:0/449465429 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc77802c200 msgr2=0x7fc778071a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:03.055 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:03.051+0000 7fc77f675640 1 --2- 172.21.15.104:0/449465429 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc77802c200 0x7fc778071a10 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fc76c00b410 tx=0x7fc76c00b8d0 comp rx=0 tx=0).stop 2024-03-28T14:08:03.055 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:03.051+0000 7fc77f675640 1 -- 172.21.15.104:0/449465429 shutdown_connections 2024-03-28T14:08:03.055 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:03.051+0000 7fc77f675640 1 --2- 172.21.15.104:0/449465429 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fc7780b7c70 0x7fc778132880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:03.055 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:03.051+0000 7fc77f675640 1 --2- 172.21.15.104:0/449465429 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fc77802c5e0 0x7fc778132340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:03.055 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:03.051+0000 7fc77f675640 1 --2- 172.21.15.104:0/449465429 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7fc75c067fc0 0x7fc75c06a480 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:03.055 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:03.051+0000 7fc77f675640 1 --2- 172.21.15.104:0/449465429 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc77802c200 0x7fc778071a10 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:03.055 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:03.051+0000 7fc77f675640 1 -- 172.21.15.104:0/449465429 >> 172.21.15.104:0/449465429 conn(0x7fc77806db20 msgr2=0x7fc7780b9b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T14:08:03.055 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:03.051+0000 7fc77f675640 1 -- 172.21.15.104:0/449465429 shutdown_connections 2024-03-28T14:08:03.055 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:03.051+0000 7fc77f675640 1 -- 172.21.15.104:0/449465429 wait complete. 2024-03-28T14:08:03.071 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-28T14:08:03.225 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:03.219+0000 7f026af7a640 1 Processor -- start 2024-03-28T14:08:03.225 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:03.219+0000 7f026af7a640 1 -- start start 2024-03-28T14:08:03.226 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:03.223+0000 7f026af7a640 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f02640b51e0 0x7f026409c450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:03.226 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:03.223+0000 7f026af7a640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f026409ae40 con 0x7f026406dd10 2024-03-28T14:08:03.226 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:03.223+0000 7f026af7a640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f026406e0f0 con 0x7f026409aa10 2024-03-28T14:08:03.226 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:03.223+0000 7f026af7a640 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 -- 0x7f02640b6370 con 0x7f02640b51e0 2024-03-28T14:08:03.226 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:03.223+0000 7f0263fff640 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f02640b51e0 0x7f026409c450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:03.226 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:03.223+0000 7f0268d16640 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f026406dd10 0x7f026413f6e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:47298/0 (socket says 172.21.15.104:47298) 2024-03-28T14:08:03.226 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:03.223+0000 7f0268d16640 1 -- 172.21.15.104:0/3657341213 learned_addr learned my addr 172.21.15.104:0/3657341213 (peer_addr_for_me v1:172.21.15.104:0/0) 2024-03-28T14:08:03.226 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:03.223+0000 7f0263fff640 1 -- 172.21.15.104:0/3657341213 >> v1:172.21.15.171:6789/0 conn(0x7f026409aa10 legacy=0x7f02641500e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T14:08:03.226 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:03.223+0000 7f0263fff640 1 -- 172.21.15.104:0/3657341213 >> v1:172.21.15.104:6789/0 conn(0x7f026406dd10 legacy=0x7f026413f6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:03.227 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:03.223+0000 7f02637fe640 1 -- 172.21.15.104:0/3657341213 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1524746699 0 0) 0x7f026409ae40 con 0x7f026406dd10 2024-03-28T14:08:03.227 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:03.223+0000 7f0263fff640 1 -- 172.21.15.104:0/3657341213 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f02640b5d20 con 0x7f02640b51e0 2024-03-28T14:08:03.228 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:03.227+0000 7f0263fff640 1 --2- 172.21.15.104:0/3657341213 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f02640b51e0 0x7f026409c450 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f02640b64f0 tx=0x7f025001ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=75a4877f32e72a46 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:08:03.228 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:03.227+0000 7f02637fe640 1 -- 172.21.15.104:0/3657341213 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02500b0070 con 0x7f02640b51e0 2024-03-28T14:08:03.228 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:03.227+0000 7f02637fe640 1 -- 172.21.15.104:0/3657341213 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f02500ab600 con 0x7f02640b51e0 2024-03-28T14:08:03.228 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:03.227+0000 7f02637fe640 1 -- 172.21.15.104:0/3657341213 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02500ab9d0 con 0x7f02640b51e0 2024-03-28T14:08:03.229 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:03.227+0000 7f026af7a640 1 -- 172.21.15.104:0/3657341213 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f02640b51e0 msgr2=0x7f026409c450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:03.229 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:03.227+0000 7f026af7a640 1 --2- 172.21.15.104:0/3657341213 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f02640b51e0 0x7f026409c450 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f02640b64f0 tx=0x7f025001ce30 comp rx=0 tx=0).stop 2024-03-28T14:08:03.229 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:03.227+0000 7f026af7a640 1 -- 172.21.15.104:0/3657341213 shutdown_connections 2024-03-28T14:08:03.229 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:03.227+0000 7f026af7a640 1 --2- 172.21.15.104:0/3657341213 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f02640b51e0 0x7f026409c450 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:03.229 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:03.227+0000 7f026af7a640 1 -- 172.21.15.104:0/3657341213 >> 172.21.15.104:0/3657341213 conn(0x7f02640ba390 msgr2=0x7f02640ba7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T14:08:03.230 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:03.227+0000 7f026af7a640 1 -- 172.21.15.104:0/3657341213 shutdown_connections 2024-03-28T14:08:03.230 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:03.227+0000 7f026af7a640 1 -- 172.21.15.104:0/3657341213 wait complete. 2024-03-28T14:08:03.232 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:03.227+0000 7f026af7a640 1 Processor -- start 2024-03-28T14:08:03.232 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:03.227+0000 7f026af7a640 1 -- start start 2024-03-28T14:08:03.233 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:03.227+0000 7f026af7a640 1 --2- >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f026406dd10 0x7f0264134b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:03.233 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:03.227+0000 7f026af7a640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f026409aa10 0x7f02641e3ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:03.233 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:03.227+0000 7f026af7a640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f02640b51e0 0x7f02641e41e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:03.233 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:03.227+0000 7f026af7a640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f02641545b0 con 0x7f026409aa10 2024-03-28T14:08:03.233 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:03.227+0000 7f026af7a640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f0264154430 con 0x7f02640b51e0 2024-03-28T14:08:03.233 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:03.227+0000 7f026af7a640 1 -- --> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] -- mon_getmap magic: 0 -- 0x7f0264154730 con 0x7f026406dd10 2024-03-28T14:08:03.233 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:03.227+0000 7f0263fff640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f026409aa10 0x7f02641e3ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:03.233 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:03.227+0000 7f0263fff640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f026409aa10 0x7f02641e3ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:42292/0 (socket says 172.21.15.104:42292) 2024-03-28T14:08:03.233 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:03.227+0000 7f0263fff640 1 -- 172.21.15.104:0/102143203 learned_addr learned my addr 172.21.15.104:0/102143203 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-03-28T14:08:03.233 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:03.227+0000 7f0268d16640 1 --2- 172.21.15.104:0/102143203 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f026406dd10 0x7f0264134b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:03.233 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:03.227+0000 7f0269517640 1 --2- 172.21.15.104:0/102143203 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f02640b51e0 0x7f02641e41e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:03.233 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:03.227+0000 7f0263fff640 1 -- 172.21.15.104:0/102143203 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f026406dd10 msgr2=0x7f0264134b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:03.234 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:03.227+0000 7f0263fff640 1 --2- 172.21.15.104:0/102143203 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f026406dd10 0x7f0264134b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:03.234 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:03.227+0000 7f0263fff640 1 -- 172.21.15.104:0/102143203 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f02640b51e0 msgr2=0x7f02641e41e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:03.234 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:03.227+0000 7f0263fff640 1 --2- 172.21.15.104:0/102143203 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f02640b51e0 0x7f02641e41e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:03.234 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:03.227+0000 7f0263fff640 1 -- 172.21.15.104:0/102143203 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0264071b00 con 0x7f026409aa10 2024-03-28T14:08:03.234 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:03.227+0000 7f0269517640 1 --2- 172.21.15.104:0/102143203 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f02640b51e0 0x7f02641e41e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T14:08:03.234 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:03.227+0000 7f0268d16640 1 --2- 172.21.15.104:0/102143203 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f026406dd10 0x7f0264134b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T14:08:03.234 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:03.231+0000 7f0263fff640 1 --2- 172.21.15.104:0/102143203 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f026409aa10 0x7f02641e3ca0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f02500ae040 tx=0x7f025001ac10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:08:03.234 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:03.231+0000 7f02617fa640 1 -- 172.21.15.104:0/102143203 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02500b0050 con 0x7f026409aa10 2024-03-28T14:08:03.234 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:03.231+0000 7f02617fa640 1 -- 172.21.15.104:0/102143203 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f02500bc070 con 0x7f026409aa10 2024-03-28T14:08:03.234 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:03.231+0000 7f026af7a640 1 -- 172.21.15.104:0/102143203 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0264071de0 con 0x7f026409aa10 2024-03-28T14:08:03.234 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:03.231+0000 7f02617fa640 1 -- 172.21.15.104:0/102143203 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02500b5830 con 0x7f026409aa10 2024-03-28T14:08:03.234 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:03.231+0000 7f026af7a640 1 -- 172.21.15.104:0/102143203 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0264070f50 con 0x7f026409aa10 2024-03-28T14:08:03.235 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:03.231+0000 7f026af7a640 1 -- 172.21.15.104:0/102143203 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f02640687c0 con 0x7f026409aa10 2024-03-28T14:08:03.235 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:03.231+0000 7f02617fa640 1 -- 172.21.15.104:0/102143203 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) ==== 87803+0+0 (secure 0 0 0) 0x7f02500b59d0 con 0x7f026409aa10 2024-03-28T14:08:03.235 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:03.231+0000 7f02617fa640 1 --2- 172.21.15.104:0/102143203 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f0248067fc0 0x7f024806a480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:03.236 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:03.231+0000 7f02617fa640 1 -- 172.21.15.104:0/102143203 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f0250120cd0 con 0x7f026409aa10 2024-03-28T14:08:03.236 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:03.231+0000 7f0268d16640 1 --2- 172.21.15.104:0/102143203 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f0248067fc0 0x7f024806a480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:03.236 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:03.231+0000 7f0268d16640 1 --2- 172.21.15.104:0/102143203 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f0248067fc0 0x7f024806a480 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f0258000f70 tx=0x7f0258009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:08:03.237 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:03.235+0000 7f02617fa640 1 -- 172.21.15.104:0/102143203 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179117 (secure 0 0 0) 0x7f0250016050 con 0x7f026409aa10 2024-03-28T14:08:03.387 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:03.387+0000 7f026af7a640 1 -- 172.21.15.104:0/102143203 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f02640b4fa0 con 0x7f026409aa10 2024-03-28T14:08:03.389 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:03.387+0000 7f02617fa640 1 -- 172.21.15.104:0/102143203 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) ==== 74+0+16629 (secure 0 0 0) 0x7f02500f1510 con 0x7f026409aa10 2024-03-28T14:08:03.389 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-03-28T14:08:03.389 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":21,"fsid":"428a90f2-3351-435f-8fcf-f7e66c78a13e","created":"2024-03-28T14:07:34.862458+0000","modified":"2024-03-28T14:08:00.671587+0000","last_up_change":"2024-03-28T14:07:46.028122+0000","last_in_change":"2024-03-28T14:07:37.157486+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-28T14:07:47.622988+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-28T14:07:57.147450+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-28T14:07:57.990975+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"1cb70540-4e42-402f-a446-aaec4ca83007","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6800","nonce":3745690918},{"type":"v1","addr":"172.21.15.104:6801","nonce":3745690918}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":3745690918},{"type":"v1","addr":"172.21.15.104:6803","nonce":3745690918}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":3745690918},{"type":"v1","addr":"172.21.15.104:6807","nonce":3745690918}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":3745690918},{"type":"v1","addr":"172.21.15.104:6805","nonce":3745690918}]},"public_addr":"172.21.15.104:6801/3745690918","cluster_addr":"172.21.15.104:6803/3745690918","heartbeat_back_addr":"172.21.15.104:6807/3745690918","heartbeat_front_addr":"172.21.15.104:6805/3745690918","state":["exists","up"]},{"osd":1,"uuid":"3ab7afdd-b980-418c-81f2-004e16de3dca","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":19438674},{"type":"v1","addr":"172.21.15.104:6810","nonce":19438674}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":19438674},{"type":"v1","addr":"172.21.15.104:6814","nonce":19438674}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":19438674},{"type":"v1","addr":"172.21.15.104:6823","nonce":19438674}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":19438674},{"type":"v1","addr":"172.21.15.104:6819","nonce":19438674}]},"public_addr":"172.21.15.104:6810/19438674","cluster_addr":"172.21.15.104:6814/19438674","heartbeat_back_addr":"172.21.15.104:6823/19438674","heartbeat_front_addr":"172.21.15.104:6819/19438674","state":["exists","up"]},{"osd":2,"uuid":"3e2651c8-ecae-4a9e-a053-27240a79eac2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":2352012626},{"type":"v1","addr":"172.21.15.104:6825","nonce":2352012626}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6826","nonce":2352012626},{"type":"v1","addr":"172.21.15.104:6827","nonce":2352012626}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":2352012626},{"type":"v1","addr":"172.21.15.104:6831","nonce":2352012626}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6828","nonce":2352012626},{"type":"v1","addr":"172.21.15.104:6829","nonce":2352012626}]},"public_addr":"172.21.15.104:6825/2352012626","cluster_addr":"172.21.15.104:6827/2352012626","heartbeat_back_addr":"172.21.15.104:6831/2352012626","heartbeat_front_addr":"172.21.15.104:6829/2352012626","state":["exists","up"]},{"osd":3,"uuid":"925c7e0a-38ef-47cd-b288-8bec8456fb17","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6809","nonce":3568347381},{"type":"v1","addr":"172.21.15.104:6811","nonce":3568347381}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6813","nonce":3568347381},{"type":"v1","addr":"172.21.15.104:6815","nonce":3568347381}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":3568347381},{"type":"v1","addr":"172.21.15.104:6822","nonce":3568347381}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6817","nonce":3568347381},{"type":"v1","addr":"172.21.15.104:6818","nonce":3568347381}]},"public_addr":"172.21.15.104:6811/3568347381","cluster_addr":"172.21.15.104:6815/3568347381","heartbeat_back_addr":"172.21.15.104:6822/3568347381","heartbeat_front_addr":"172.21.15.104:6818/3568347381","state":["exists","up"]},{"osd":4,"uuid":"4a8fac86-e595-4a0e-96e3-57edb3cc3f57","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6800","nonce":2107001077},{"type":"v1","addr":"172.21.15.171:6801","nonce":2107001077}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6802","nonce":2107001077},{"type":"v1","addr":"172.21.15.171:6803","nonce":2107001077}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6806","nonce":2107001077},{"type":"v1","addr":"172.21.15.171:6807","nonce":2107001077}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6804","nonce":2107001077},{"type":"v1","addr":"172.21.15.171:6805","nonce":2107001077}]},"public_addr":"172.21.15.171:6801/2107001077","cluster_addr":"172.21.15.171:6803/2107001077","heartbeat_back_addr":"172.21.15.171:6807/2107001077","heartbeat_front_addr":"172.21.15.171:6805/2107001077","state":["exists","up"]},{"osd":5,"uuid":"e48d50b0-beae-40ae-b15a-9a84dbfe88ab","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6808","nonce":2934826925},{"type":"v1","addr":"172.21.15.171:6810","nonce":2934826925}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6812","nonce":2934826925},{"type":"v1","addr":"172.21.15.171:6814","nonce":2934826925}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6819","nonce":2934826925},{"type":"v1","addr":"172.21.15.171:6821","nonce":2934826925}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6816","nonce":2934826925},{"type":"v1","addr":"172.21.15.171:6817","nonce":2934826925}]},"public_addr":"172.21.15.171:6810/2934826925","cluster_addr":"172.21.15.171:6814/2934826925","heartbeat_back_addr":"172.21.15.171:6821/2934826925","heartbeat_front_addr":"172.21.15.171:6817/2934826925","state":["exists","up"]},{"osd":6,"uuid":"5e7c3292-9936-4f35-b267-aed9d4e49323","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6824","nonce":436742919},{"type":"v1","addr":"172.21.15.171:6825","nonce":436742919}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6826","nonce":436742919},{"type":"v1","addr":"172.21.15.171:6827","nonce":436742919}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6830","nonce":436742919},{"type":"v1","addr":"172.21.15.171:6831","nonce":436742919}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6828","nonce":436742919},{"type":"v1","addr":"172.21.15.171:6829","nonce":436742919}]},"public_addr":"172.21.15.171:6825/436742919","cluster_addr":"172.21.15.171:6827/436742919","heartbeat_back_addr":"172.21.15.171:6831/436742919","heartbeat_front_addr":"172.21.15.171:6829/436742919","state":["exists","up"]},{"osd":7,"uuid":"339b8414-d7f0-4fb0-a446-efaf635cb7ff","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6809","nonce":2055631175},{"type":"v1","addr":"172.21.15.171:6811","nonce":2055631175}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6813","nonce":2055631175},{"type":"v1","addr":"172.21.15.171:6815","nonce":2055631175}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6822","nonce":2055631175},{"type":"v1","addr":"172.21.15.171:6823","nonce":2055631175}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6818","nonce":2055631175},{"type":"v1","addr":"172.21.15.171:6820","nonce":2055631175}]},"public_addr":"172.21.15.171:6811/2055631175","cluster_addr":"172.21.15.171:6815/2055631175","heartbeat_back_addr":"172.21.15.171:6823/2055631175","heartbeat_front_addr":"172.21.15.171:6820/2055631175","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T14:07:42.668881+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T14:07:42.708397+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T14:07:42.634191+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T14:07:42.733662+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T14:07:42.651541+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T14:07:42.783181+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T14:07:42.766310+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T14:07:42.670797+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-28T14:08:03.391 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:03.387+0000 7f026af7a640 1 -- 172.21.15.104:0/102143203 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f0248067fc0 msgr2=0x7f024806a480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:03.391 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:03.387+0000 7f026af7a640 1 --2- 172.21.15.104:0/102143203 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f0248067fc0 0x7f024806a480 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f0258000f70 tx=0x7f0258009040 comp rx=0 tx=0).stop 2024-03-28T14:08:03.391 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:03.387+0000 7f026af7a640 1 -- 172.21.15.104:0/102143203 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f026409aa10 msgr2=0x7f02641e3ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:03.391 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:03.387+0000 7f026af7a640 1 --2- 172.21.15.104:0/102143203 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f026409aa10 0x7f02641e3ca0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f02500ae040 tx=0x7f025001ac10 comp rx=0 tx=0).stop 2024-03-28T14:08:03.391 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:03.387+0000 7f026af7a640 1 -- 172.21.15.104:0/102143203 shutdown_connections 2024-03-28T14:08:03.391 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:03.387+0000 7f026af7a640 1 --2- 172.21.15.104:0/102143203 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f02640b51e0 0x7f02641e41e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:03.391 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:03.387+0000 7f026af7a640 1 --2- 172.21.15.104:0/102143203 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f0248067fc0 0x7f024806a480 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:03.391 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:03.387+0000 7f026af7a640 1 --2- 172.21.15.104:0/102143203 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f026409aa10 0x7f02641e3ca0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:03.391 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:03.387+0000 7f026af7a640 1 --2- 172.21.15.104:0/102143203 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f026406dd10 0x7f0264134b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:03.391 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:03.387+0000 7f026af7a640 1 -- 172.21.15.104:0/102143203 >> 172.21.15.104:0/102143203 conn(0x7f02640ba390 msgr2=0x7f026413ddf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T14:08:03.391 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:03.387+0000 7f026af7a640 1 -- 172.21.15.104:0/102143203 shutdown_connections 2024-03-28T14:08:03.392 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:03.387+0000 7f026af7a640 1 -- 172.21.15.104:0/102143203 wait complete. 2024-03-28T14:08:03.406 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-03-28T14:08:03.406 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-28T14:08:03.530 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:03.527+0000 7f0ceb37f640 1 Processor -- start 2024-03-28T14:08:03.530 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:03.527+0000 7f0ceb37f640 1 -- start start 2024-03-28T14:08:03.532 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:03.527+0000 7f0ceb37f640 1 --2- >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f0ce4071be0 0x7f0ce4150ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:03.532 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:03.527+0000 7f0ceb37f640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0ce4072030 con 0x7f0ce4072bf0 2024-03-28T14:08:03.533 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:03.527+0000 7f0ceb37f640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0ce4072fd0 con 0x7f0ce40b1e80 2024-03-28T14:08:03.533 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:03.527+0000 7f0ceb37f640 1 -- --> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] -- mon_getmap magic: 0 -- 0x7f0ce4073250 con 0x7f0ce4071be0 2024-03-28T14:08:03.533 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:03.527+0000 7f0ce911b640 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f0ce4072bf0 0x7f0ce40b12f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:47306/0 (socket says 172.21.15.104:47306) 2024-03-28T14:08:03.533 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:03.527+0000 7f0ce911b640 1 -- 172.21.15.104:0/295522755 learned_addr learned my addr 172.21.15.104:0/295522755 (peer_addr_for_me v1:172.21.15.104:0/0) 2024-03-28T14:08:03.533 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:03.527+0000 7f0ce3fff640 1 -- 172.21.15.104:0/295522755 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2953301341 0 0) 0x7f0cdc0014a0 con 0x7f0ce4072bf0 2024-03-28T14:08:03.533 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:03.527+0000 7f0ce3fff640 1 -- 172.21.15.104:0/295522755 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0ccc003530 con 0x7f0ce4072bf0 2024-03-28T14:08:03.533 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:03.527+0000 7f0ce991c640 1 --2- 172.21.15.104:0/295522755 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f0ce4071be0 0x7f0ce4150ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:03.533 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:03.531+0000 7f0ce3fff640 1 -- 172.21.15.104:0/295522755 <== mon.0 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1679566587 0 0) 0x7f0ccc003530 con 0x7f0ce4072bf0 2024-03-28T14:08:03.533 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:03.531+0000 7f0ce3fff640 1 -- 172.21.15.104:0/295522755 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f0ce4071be0 msgr2=0x7f0ce4150ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T14:08:03.533 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:03.531+0000 7f0ce3fff640 1 --2- 172.21.15.104:0/295522755 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f0ce4071be0 0x7f0ce4150ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:03.533 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:03.531+0000 7f0ce3fff640 1 -- 172.21.15.104:0/295522755 >> v1:172.21.15.171:6789/0 conn(0x7f0ce40b1e80 legacy=0x7f0ce4145ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:03.533 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:03.531+0000 7f0ce3fff640 1 -- 172.21.15.104:0/295522755 --> v1:172.21.15.104:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0ce40734a0 con 0x7f0ce4072bf0 2024-03-28T14:08:03.533 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:03.531+0000 7f0ce3fff640 1 -- 172.21.15.104:0/295522755 <== mon.0 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3745580849 0 0) 0x7f0cdc001bd0 con 0x7f0ce4072bf0 2024-03-28T14:08:03.533 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:03.531+0000 7f0ce3fff640 1 -- 172.21.15.104:0/295522755 >> v1:172.21.15.104:6789/0 conn(0x7f0ce4072bf0 legacy=0x7f0ce40b12f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:03.534 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:03.531+0000 7f0ce3fff640 1 --2- 172.21.15.104:0/295522755 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0ccc0039f0 0x7f0ccc003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:03.534 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:03.531+0000 7f0ce3fff640 1 --2- 172.21.15.104:0/295522755 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f0ce4071be0 0x7f0ccc004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:03.534 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:03.531+0000 7f0ce3fff640 1 --2- 172.21.15.104:0/295522755 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f0ccc004da0 0x7f0ccc005220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:03.534 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:03.531+0000 7f0ce3fff640 1 -- 172.21.15.104:0/295522755 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f0ce4073250 con 0x7f0ccc0039f0 2024-03-28T14:08:03.534 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:03.531+0000 7f0ce3fff640 1 -- 172.21.15.104:0/295522755 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f0ccc00bc50 con 0x7f0ccc004da0 2024-03-28T14:08:03.534 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:03.531+0000 7f0ce3fff640 1 -- 172.21.15.104:0/295522755 --> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] -- mon_getmap magic: 0 -- 0x7f0ccc00be50 con 0x7f0ce4071be0 2024-03-28T14:08:03.534 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:03.531+0000 7f0ce911b640 1 --2- 172.21.15.104:0/295522755 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0ccc0039f0 0x7f0ccc003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:03.534 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:03.531+0000 7f0ce891a640 1 --2- 172.21.15.104:0/295522755 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f0ce4071be0 0x7f0ccc004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:03.534 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:03.531+0000 7f0ce911b640 1 -- 172.21.15.104:0/295522755 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f0ce4071be0 msgr2=0x7f0ccc004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T14:08:03.534 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:03.531+0000 7f0ce991c640 1 --2- 172.21.15.104:0/295522755 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f0ccc004da0 0x7f0ccc005220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:03.534 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:03.531+0000 7f0ce911b640 1 --2- 172.21.15.104:0/295522755 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f0ce4071be0 0x7f0ccc004810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:03.534 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:03.531+0000 7f0ce911b640 1 -- 172.21.15.104:0/295522755 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f0ccc004da0 msgr2=0x7f0ccc005220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T14:08:03.534 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:03.531+0000 7f0ce911b640 1 --2- 172.21.15.104:0/295522755 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f0ccc004da0 0x7f0ccc005220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:03.535 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:03.531+0000 7f0ce911b640 1 -- 172.21.15.104:0/295522755 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0ccc003530 con 0x7f0ccc0039f0 2024-03-28T14:08:03.535 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:03.531+0000 7f0ce891a640 1 --2- 172.21.15.104:0/295522755 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f0ce4071be0 0x7f0ccc004810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T14:08:03.536 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:03.531+0000 7f0ce911b640 1 --2- 172.21.15.104:0/295522755 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0ccc0039f0 0x7f0ccc003e10 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f0cdc00aa00 tx=0x7f0cdc01d320 comp rx=0 tx=0).ready entity=mon.0 client_cookie=49d0ef28c0c82d85 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:08:03.536 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:03.531+0000 7f0ce3fff640 1 -- 172.21.15.104:0/295522755 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0cdc00a650 con 0x7f0ccc0039f0 2024-03-28T14:08:03.536 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:03.531+0000 7f0ce3fff640 1 -- 172.21.15.104:0/295522755 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0cdc0ad650 con 0x7f0ccc0039f0 2024-03-28T14:08:03.536 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:03.531+0000 7f0ce3fff640 1 -- 172.21.15.104:0/295522755 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0cdc0ada40 con 0x7f0ccc0039f0 2024-03-28T14:08:03.536 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:03.531+0000 7f0ceb37f640 1 -- 172.21.15.104:0/295522755 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0ccc0039f0 msgr2=0x7f0ccc003e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:03.536 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:03.531+0000 7f0ceb37f640 1 --2- 172.21.15.104:0/295522755 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0ccc0039f0 0x7f0ccc003e10 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f0cdc00aa00 tx=0x7f0cdc01d320 comp rx=0 tx=0).stop 2024-03-28T14:08:03.536 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:03.531+0000 7f0ce991c640 1 -- 172.21.15.104:0/295522755 reap_dead start 2024-03-28T14:08:03.536 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:03.531+0000 7f0ceb37f640 1 -- 172.21.15.104:0/295522755 shutdown_connections 2024-03-28T14:08:03.537 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:03.531+0000 7f0ceb37f640 1 -- 172.21.15.104:0/295522755 >> 172.21.15.104:0/295522755 conn(0x7f0ce40b7b10 msgr2=0x7f0ce40b7f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T14:08:03.537 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:03.531+0000 7f0ceb37f640 1 -- 172.21.15.104:0/295522755 shutdown_connections 2024-03-28T14:08:03.537 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:03.531+0000 7f0ceb37f640 1 -- 172.21.15.104:0/295522755 wait complete. 2024-03-28T14:08:03.537 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:03.535+0000 7f0ceb37f640 1 Processor -- start 2024-03-28T14:08:03.537 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:03.535+0000 7f0ceb37f640 1 -- start start 2024-03-28T14:08:03.539 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:03.535+0000 7f0ceb37f640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f0ccc0039f0 0x7f0ce4140760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:03.539 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:03.535+0000 7f0ceb37f640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0ce4141c20 0x7f0ce41deef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:03.539 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:03.535+0000 7f0ceb37f640 1 --2- >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f0ce41df430 0x7f0ce41391a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:03.539 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:03.535+0000 7f0ceb37f640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f0ce4146570 con 0x7f0ce4141c20 2024-03-28T14:08:03.539 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:03.535+0000 7f0ceb37f640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f0ce41463f0 con 0x7f0ccc0039f0 2024-03-28T14:08:03.539 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:03.535+0000 7f0ceb37f640 1 -- --> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] -- mon_getmap magic: 0 -- 0x7f0ce41516b0 con 0x7f0ce41df430 2024-03-28T14:08:03.539 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:03.535+0000 7f0ce891a640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0ce4141c20 0x7f0ce41deef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:03.539 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:03.535+0000 7f0ce891a640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0ce4141c20 0x7f0ce41deef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:42322/0 (socket says 172.21.15.104:42322) 2024-03-28T14:08:03.540 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:03.535+0000 7f0ce891a640 1 -- 172.21.15.104:0/3618230093 learned_addr learned my addr 172.21.15.104:0/3618230093 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-03-28T14:08:03.540 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:03.535+0000 7f0ce891a640 1 -- 172.21.15.104:0/3618230093 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f0ce41df430 msgr2=0x7f0ce41391a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:03.540 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:03.535+0000 7f0ce891a640 1 --2- 172.21.15.104:0/3618230093 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f0ce41df430 0x7f0ce41391a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:03.540 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:03.535+0000 7f0ce891a640 1 -- 172.21.15.104:0/3618230093 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f0ccc0039f0 msgr2=0x7f0ce4140760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:03.540 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:03.535+0000 7f0ce891a640 1 --2- 172.21.15.104:0/3618230093 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f0ccc0039f0 0x7f0ce4140760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:03.540 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:03.535+0000 7f0ce891a640 1 -- 172.21.15.104:0/3618230093 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0ce409ec20 con 0x7f0ce4141c20 2024-03-28T14:08:03.540 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:03.535+0000 7f0ce891a640 1 --2- 172.21.15.104:0/3618230093 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0ce4141c20 0x7f0ce41deef0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f0cd8000c10 tx=0x7f0cd800dc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:08:03.540 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:03.535+0000 7f0ce1ffb640 1 -- 172.21.15.104:0/3618230093 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0cd8019070 con 0x7f0ce4141c20 2024-03-28T14:08:03.540 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:03.535+0000 7f0ce1ffb640 1 -- 172.21.15.104:0/3618230093 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0cd8005270 con 0x7f0ce4141c20 2024-03-28T14:08:03.540 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:03.535+0000 7f0ceb37f640 1 -- 172.21.15.104:0/3618230093 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0ce41de750 con 0x7f0ce4141c20 2024-03-28T14:08:03.540 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:03.535+0000 7f0ce1ffb640 1 -- 172.21.15.104:0/3618230093 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0cd8007030 con 0x7f0ce4141c20 2024-03-28T14:08:03.540 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:03.535+0000 7f0ceb37f640 1 -- 172.21.15.104:0/3618230093 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0ce4142100 con 0x7f0ce4141c20 2024-03-28T14:08:03.540 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:03.535+0000 7f0ceb37f640 1 -- 172.21.15.104:0/3618230093 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0ce40687c0 con 0x7f0ce4141c20 2024-03-28T14:08:03.541 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:03.539+0000 7f0ce1ffb640 1 -- 172.21.15.104:0/3618230093 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) ==== 87803+0+0 (secure 0 0 0) 0x7f0cd801c070 con 0x7f0ce4141c20 2024-03-28T14:08:03.541 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:03.539+0000 7f0ce1ffb640 1 --2- 172.21.15.104:0/3618230093 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f0cd0067fc0 0x7f0cd006a480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:03.541 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:03.539+0000 7f0ce1ffb640 1 -- 172.21.15.104:0/3618230093 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f0cd8085d10 con 0x7f0ce4141c20 2024-03-28T14:08:03.542 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:03.539+0000 7f0ce911b640 1 --2- 172.21.15.104:0/3618230093 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f0cd0067fc0 0x7f0cd006a480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:03.542 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:03.539+0000 7f0ce911b640 1 --2- 172.21.15.104:0/3618230093 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f0cd0067fc0 0x7f0cd006a480 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f0cdc0ad000 tx=0x7f0ce4073670 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:08:03.543 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:03.543+0000 7f0ce1ffb640 1 -- 172.21.15.104:0/3618230093 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179117 (secure 0 0 0) 0x7f0ce41de750 con 0x7f0ce4141c20 2024-03-28T14:08:03.728 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:03.727+0000 7f0ceb37f640 1 -- 172.21.15.104:0/3618230093 --> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f0ce41ded20 con 0x7f0cd0067fc0 2024-03-28T14:08:03.747 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:03.747+0000 7f0ce1ffb640 1 -- 172.21.15.104:0/3618230093 <== mgr.4100 v2:172.21.15.171:6832/1522752412 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+251950 (secure 0 0 0) 0x7f0ce41ded20 con 0x7f0cd0067fc0 2024-03-28T14:08:03.750 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-03-28T14:08:03.752 INFO:teuthology.orchestra.run.smithi104.stderr:dumped all 2024-03-28T14:08:03.753 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:03.751+0000 7f0ceb37f640 1 -- 172.21.15.104:0/3618230093 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f0cd0067fc0 msgr2=0x7f0cd006a480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:03.754 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:03.751+0000 7f0ceb37f640 1 --2- 172.21.15.104:0/3618230093 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f0cd0067fc0 0x7f0cd006a480 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f0cdc0ad000 tx=0x7f0ce4073670 comp rx=0 tx=0).stop 2024-03-28T14:08:03.754 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:03.751+0000 7f0ceb37f640 1 -- 172.21.15.104:0/3618230093 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0ce4141c20 msgr2=0x7f0ce41deef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:03.754 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:03.751+0000 7f0ceb37f640 1 --2- 172.21.15.104:0/3618230093 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0ce4141c20 0x7f0ce41deef0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f0cd8000c10 tx=0x7f0cd800dc00 comp rx=0 tx=0).stop 2024-03-28T14:08:03.754 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:03.751+0000 7f0ceb37f640 1 -- 172.21.15.104:0/3618230093 shutdown_connections 2024-03-28T14:08:03.754 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:03.751+0000 7f0ceb37f640 1 --2- 172.21.15.104:0/3618230093 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f0ce41df430 0x7f0ce41391a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:03.754 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:03.751+0000 7f0ceb37f640 1 --2- 172.21.15.104:0/3618230093 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f0cd0067fc0 0x7f0cd006a480 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:03.754 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:03.751+0000 7f0ceb37f640 1 --2- 172.21.15.104:0/3618230093 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0ce4141c20 0x7f0ce41deef0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:03.754 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:03.751+0000 7f0ceb37f640 1 --2- 172.21.15.104:0/3618230093 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f0ccc0039f0 0x7f0ce4140760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:03.754 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:03.751+0000 7f0ceb37f640 1 -- 172.21.15.104:0/3618230093 >> 172.21.15.104:0/3618230093 conn(0x7f0ce40b7b10 msgr2=0x7f0ce413d580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T14:08:03.754 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:03.751+0000 7f0ceb37f640 1 -- 172.21.15.104:0/3618230093 shutdown_connections 2024-03-28T14:08:03.754 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:03.751+0000 7f0ceb37f640 1 -- 172.21.15.104:0/3618230093 wait complete. 2024-03-28T14:08:03.770 INFO:teuthology.orchestra.run.smithi104.stdout:{"pg_ready":true,"pg_map":{"version":23,"stamp":"2024-03-28T14:08:03.578345+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":146,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219108,"kb_used_data":2920,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755612,"statfs":{"total":773094113280,"available":772869746688,"internally_reserved":0,"allocated":2990080,"data_stored":1790396,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":221236785},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":45,"apply_latency_ms":45,"commit_latency_ns":45000000,"apply_latency_ns":45000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.934032"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-28T14:08:00.835047+0000","last_change":"2024-03-28T14:07:59.730372+0000","last_active":"2024-03-28T14:08:00.835047+0000","last_peered":"2024-03-28T14:08:00.835047+0000","last_clean":"2024-03-28T14:08:00.835047+0000","last_became_active":"2024-03-28T14:07:59.729644+0000","last_became_peered":"2024-03-28T14:07:59.729644+0000","last_unstale":"2024-03-28T14:08:00.835047+0000","last_undegraded":"2024-03-28T14:08:00.835047+0000","last_fullsized":"2024-03-28T14:08:00.835047+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_clean_scrub_stamp":"2024-03-28T14:07:58.641775+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:59:25.862480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-28T14:08:00.696778+0000","last_change":"2024-03-28T14:07:59.705187+0000","last_active":"2024-03-28T14:08:00.696778+0000","last_peered":"2024-03-28T14:08:00.696778+0000","last_clean":"2024-03-28T14:08:00.696778+0000","last_became_active":"2024-03-28T14:07:59.700386+0000","last_became_peered":"2024-03-28T14:07:59.700386+0000","last_unstale":"2024-03-28T14:08:00.696778+0000","last_undegraded":"2024-03-28T14:08:00.696778+0000","last_fullsized":"2024-03-28T14:08:00.696778+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_clean_scrub_stamp":"2024-03-28T14:07:58.641775+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T16:47:04.317583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-28T14:08:01.245347+0000","last_change":"2024-03-28T14:07:59.698753+0000","last_active":"2024-03-28T14:08:01.245347+0000","last_peered":"2024-03-28T14:08:01.245347+0000","last_clean":"2024-03-28T14:08:01.245347+0000","last_became_active":"2024-03-28T14:07:59.698467+0000","last_became_peered":"2024-03-28T14:07:59.698467+0000","last_unstale":"2024-03-28T14:08:01.245347+0000","last_undegraded":"2024-03-28T14:08:01.245347+0000","last_fullsized":"2024-03-28T14:08:01.245347+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_clean_scrub_stamp":"2024-03-28T14:07:58.641775+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:25:39.647896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-28T14:08:00.680934+0000","last_change":"2024-03-28T14:07:59.747773+0000","last_active":"2024-03-28T14:08:00.680934+0000","last_peered":"2024-03-28T14:08:00.680934+0000","last_clean":"2024-03-28T14:08:00.680934+0000","last_became_active":"2024-03-28T14:07:59.745640+0000","last_became_peered":"2024-03-28T14:07:59.745640+0000","last_unstale":"2024-03-28T14:08:00.680934+0000","last_undegraded":"2024-03-28T14:08:00.680934+0000","last_fullsized":"2024-03-28T14:08:00.680934+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_clean_scrub_stamp":"2024-03-28T14:07:58.641775+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:10:31.257059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-28T14:08:00.834115+0000","last_change":"2024-03-28T14:07:59.722923+0000","last_active":"2024-03-28T14:08:00.834115+0000","last_peered":"2024-03-28T14:08:00.834115+0000","last_clean":"2024-03-28T14:08:00.834115+0000","last_became_active":"2024-03-28T14:07:59.711460+0000","last_became_peered":"2024-03-28T14:07:59.711460+0000","last_unstale":"2024-03-28T14:08:00.834115+0000","last_undegraded":"2024-03-28T14:08:00.834115+0000","last_fullsized":"2024-03-28T14:08:00.834115+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_clean_scrub_stamp":"2024-03-28T14:07:58.641775+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T18:51:19.214005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-28T14:08:00.680087+0000","last_change":"2024-03-28T14:07:59.742906+0000","last_active":"2024-03-28T14:08:00.680087+0000","last_peered":"2024-03-28T14:08:00.680087+0000","last_clean":"2024-03-28T14:08:00.680087+0000","last_became_active":"2024-03-28T14:07:59.734609+0000","last_became_peered":"2024-03-28T14:07:59.734609+0000","last_unstale":"2024-03-28T14:08:00.680087+0000","last_undegraded":"2024-03-28T14:08:00.680087+0000","last_fullsized":"2024-03-28T14:08:00.680087+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_clean_scrub_stamp":"2024-03-28T14:07:58.641775+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:41:27.409736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-28T14:08:00.833773+0000","last_change":"2024-03-28T14:07:59.717632+0000","last_active":"2024-03-28T14:08:00.833773+0000","last_peered":"2024-03-28T14:08:00.833773+0000","last_clean":"2024-03-28T14:08:00.833773+0000","last_became_active":"2024-03-28T14:07:59.711894+0000","last_became_peered":"2024-03-28T14:07:59.711894+0000","last_unstale":"2024-03-28T14:08:00.833773+0000","last_undegraded":"2024-03-28T14:08:00.833773+0000","last_fullsized":"2024-03-28T14:08:00.833773+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_clean_scrub_stamp":"2024-03-28T14:07:58.641775+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T16:47:04.317583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-28T14:08:00.680228+0000","last_change":"2024-03-28T14:07:59.740843+0000","last_active":"2024-03-28T14:08:00.680228+0000","last_peered":"2024-03-28T14:08:00.680228+0000","last_clean":"2024-03-28T14:08:00.680228+0000","last_became_active":"2024-03-28T14:07:59.740606+0000","last_became_peered":"2024-03-28T14:07:59.740606+0000","last_unstale":"2024-03-28T14:08:00.680228+0000","last_undegraded":"2024-03-28T14:08:00.680228+0000","last_fullsized":"2024-03-28T14:08:00.680228+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_clean_scrub_stamp":"2024-03-28T14:07:58.641775+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T14:20:33.555524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-28T14:08:00.835049+0000","last_change":"2024-03-28T14:07:59.745208+0000","last_active":"2024-03-28T14:08:00.835049+0000","last_peered":"2024-03-28T14:08:00.835049+0000","last_clean":"2024-03-28T14:08:00.835049+0000","last_became_active":"2024-03-28T14:07:59.744704+0000","last_became_peered":"2024-03-28T14:07:59.744704+0000","last_unstale":"2024-03-28T14:08:00.835049+0000","last_undegraded":"2024-03-28T14:08:00.835049+0000","last_fullsized":"2024-03-28T14:08:00.835049+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_clean_scrub_stamp":"2024-03-28T14:07:58.641775+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:53:55.059948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-28T14:08:00.703336+0000","last_change":"2024-03-28T14:07:59.744713+0000","last_active":"2024-03-28T14:08:00.703336+0000","last_peered":"2024-03-28T14:08:00.703336+0000","last_clean":"2024-03-28T14:08:00.703336+0000","last_became_active":"2024-03-28T14:07:59.744134+0000","last_became_peered":"2024-03-28T14:07:59.744134+0000","last_unstale":"2024-03-28T14:08:00.703336+0000","last_undegraded":"2024-03-28T14:08:00.703336+0000","last_fullsized":"2024-03-28T14:08:00.703336+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_clean_scrub_stamp":"2024-03-28T14:07:58.641775+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:25:39.647896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-28T14:08:00.703286+0000","last_change":"2024-03-28T14:07:59.745769+0000","last_active":"2024-03-28T14:08:00.703286+0000","last_peered":"2024-03-28T14:08:00.703286+0000","last_clean":"2024-03-28T14:08:00.703286+0000","last_became_active":"2024-03-28T14:07:59.745151+0000","last_became_peered":"2024-03-28T14:07:59.745151+0000","last_unstale":"2024-03-28T14:08:00.703286+0000","last_undegraded":"2024-03-28T14:08:00.703286+0000","last_fullsized":"2024-03-28T14:08:00.703286+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_clean_scrub_stamp":"2024-03-28T14:07:58.641775+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:37:29.402787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-28T14:08:01.244022+0000","last_change":"2024-03-28T14:07:59.702599+0000","last_active":"2024-03-28T14:08:01.244022+0000","last_peered":"2024-03-28T14:08:01.244022+0000","last_clean":"2024-03-28T14:08:01.244022+0000","last_became_active":"2024-03-28T14:07:59.701506+0000","last_became_peered":"2024-03-28T14:07:59.701506+0000","last_unstale":"2024-03-28T14:08:01.244022+0000","last_undegraded":"2024-03-28T14:08:01.244022+0000","last_fullsized":"2024-03-28T14:08:01.244022+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_clean_scrub_stamp":"2024-03-28T14:07:58.641775+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T15:40:59.112394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-28T14:08:00.680979+0000","last_change":"2024-03-28T14:07:59.704862+0000","last_active":"2024-03-28T14:08:00.680979+0000","last_peered":"2024-03-28T14:08:00.680979+0000","last_clean":"2024-03-28T14:08:00.680979+0000","last_became_active":"2024-03-28T14:07:59.704461+0000","last_became_peered":"2024-03-28T14:07:59.704461+0000","last_unstale":"2024-03-28T14:08:00.680979+0000","last_undegraded":"2024-03-28T14:08:00.680979+0000","last_fullsized":"2024-03-28T14:08:00.680979+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_clean_scrub_stamp":"2024-03-28T14:07:58.641775+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:37:29.402787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-28T14:08:01.243740+0000","last_change":"2024-03-28T14:07:59.707399+0000","last_active":"2024-03-28T14:08:01.243740+0000","last_peered":"2024-03-28T14:08:01.243740+0000","last_clean":"2024-03-28T14:08:01.243740+0000","last_became_active":"2024-03-28T14:07:59.707057+0000","last_became_peered":"2024-03-28T14:07:59.707057+0000","last_unstale":"2024-03-28T14:08:01.243740+0000","last_undegraded":"2024-03-28T14:08:01.243740+0000","last_fullsized":"2024-03-28T14:08:01.243740+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_clean_scrub_stamp":"2024-03-28T14:07:58.641775+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:54:39.259029+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-28T14:08:00.696269+0000","last_change":"2024-03-28T14:07:59.684910+0000","last_active":"2024-03-28T14:08:00.696269+0000","last_peered":"2024-03-28T14:08:00.696269+0000","last_clean":"2024-03-28T14:08:00.696269+0000","last_became_active":"2024-03-28T14:07:59.684112+0000","last_became_peered":"2024-03-28T14:07:59.684112+0000","last_unstale":"2024-03-28T14:08:00.696269+0000","last_undegraded":"2024-03-28T14:08:00.696269+0000","last_fullsized":"2024-03-28T14:08:00.696269+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_clean_scrub_stamp":"2024-03-28T14:07:58.641775+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:37:29.402787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-28T14:08:00.680001+0000","last_change":"2024-03-28T14:07:59.740138+0000","last_active":"2024-03-28T14:08:00.680001+0000","last_peered":"2024-03-28T14:08:00.680001+0000","last_clean":"2024-03-28T14:08:00.680001+0000","last_became_active":"2024-03-28T14:07:59.739320+0000","last_became_peered":"2024-03-28T14:07:59.739320+0000","last_unstale":"2024-03-28T14:08:00.680001+0000","last_undegraded":"2024-03-28T14:08:00.680001+0000","last_fullsized":"2024-03-28T14:08:00.680001+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_clean_scrub_stamp":"2024-03-28T14:07:58.641775+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:25:39.647896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-28T14:08:00.681671+0000","last_change":"2024-03-28T14:07:59.744839+0000","last_active":"2024-03-28T14:08:00.681671+0000","last_peered":"2024-03-28T14:08:00.681671+0000","last_clean":"2024-03-28T14:08:00.681671+0000","last_became_active":"2024-03-28T14:07:59.739555+0000","last_became_peered":"2024-03-28T14:07:59.739555+0000","last_unstale":"2024-03-28T14:08:00.681671+0000","last_undegraded":"2024-03-28T14:08:00.681671+0000","last_fullsized":"2024-03-28T14:08:00.681671+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_clean_scrub_stamp":"2024-03-28T14:07:58.641775+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:53:55.059948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-28T14:08:01.242371+0000","last_change":"2024-03-28T14:07:59.707740+0000","last_active":"2024-03-28T14:08:01.242371+0000","last_peered":"2024-03-28T14:08:01.242371+0000","last_clean":"2024-03-28T14:08:01.242371+0000","last_became_active":"2024-03-28T14:07:59.702876+0000","last_became_peered":"2024-03-28T14:07:59.702876+0000","last_unstale":"2024-03-28T14:08:01.242371+0000","last_undegraded":"2024-03-28T14:08:01.242371+0000","last_fullsized":"2024-03-28T14:08:01.242371+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_clean_scrub_stamp":"2024-03-28T14:07:58.641775+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T15:40:01.667745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-28T14:08:00.834771+0000","last_change":"2024-03-28T14:07:59.723973+0000","last_active":"2024-03-28T14:08:00.834771+0000","last_peered":"2024-03-28T14:08:00.834771+0000","last_clean":"2024-03-28T14:08:00.834771+0000","last_became_active":"2024-03-28T14:07:59.712162+0000","last_became_peered":"2024-03-28T14:07:59.712162+0000","last_unstale":"2024-03-28T14:08:00.834771+0000","last_undegraded":"2024-03-28T14:08:00.834771+0000","last_fullsized":"2024-03-28T14:08:00.834771+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_clean_scrub_stamp":"2024-03-28T14:07:58.641775+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:51:23.322586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-28T14:08:00.703000+0000","last_change":"2024-03-28T14:07:59.704917+0000","last_active":"2024-03-28T14:08:00.703000+0000","last_peered":"2024-03-28T14:08:00.703000+0000","last_clean":"2024-03-28T14:08:00.703000+0000","last_became_active":"2024-03-28T14:07:59.702369+0000","last_became_peered":"2024-03-28T14:07:59.702369+0000","last_unstale":"2024-03-28T14:08:00.703000+0000","last_undegraded":"2024-03-28T14:08:00.703000+0000","last_fullsized":"2024-03-28T14:08:00.703000+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_clean_scrub_stamp":"2024-03-28T14:07:58.641775+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:24:30.197596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-28T14:08:01.244077+0000","last_change":"2024-03-28T14:07:59.727121+0000","last_active":"2024-03-28T14:08:01.244077+0000","last_peered":"2024-03-28T14:08:01.244077+0000","last_clean":"2024-03-28T14:08:01.244077+0000","last_became_active":"2024-03-28T14:07:59.726783+0000","last_became_peered":"2024-03-28T14:07:59.726783+0000","last_unstale":"2024-03-28T14:08:01.244077+0000","last_undegraded":"2024-03-28T14:08:01.244077+0000","last_fullsized":"2024-03-28T14:08:01.244077+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_clean_scrub_stamp":"2024-03-28T14:07:58.641775+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:10:27.248645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-28T14:08:00.680533+0000","last_change":"2024-03-28T14:07:59.742729+0000","last_active":"2024-03-28T14:08:00.680533+0000","last_peered":"2024-03-28T14:08:00.680533+0000","last_clean":"2024-03-28T14:08:00.680533+0000","last_became_active":"2024-03-28T14:07:59.739727+0000","last_became_peered":"2024-03-28T14:07:59.739727+0000","last_unstale":"2024-03-28T14:08:00.680533+0000","last_undegraded":"2024-03-28T14:08:00.680533+0000","last_fullsized":"2024-03-28T14:08:00.680533+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_clean_scrub_stamp":"2024-03-28T14:07:58.641775+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T17:46:06.253111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-28T14:08:00.681600+0000","last_change":"2024-03-28T14:07:59.704724+0000","last_active":"2024-03-28T14:08:00.681600+0000","last_peered":"2024-03-28T14:08:00.681600+0000","last_clean":"2024-03-28T14:08:00.681600+0000","last_became_active":"2024-03-28T14:07:59.704398+0000","last_became_peered":"2024-03-28T14:07:59.704398+0000","last_unstale":"2024-03-28T14:08:00.681600+0000","last_undegraded":"2024-03-28T14:08:00.681600+0000","last_fullsized":"2024-03-28T14:08:00.681600+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_clean_scrub_stamp":"2024-03-28T14:07:58.641775+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T18:51:19.214005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-28T14:08:00.680302+0000","last_change":"2024-03-28T14:07:59.739100+0000","last_active":"2024-03-28T14:08:00.680302+0000","last_peered":"2024-03-28T14:08:00.680302+0000","last_clean":"2024-03-28T14:08:00.680302+0000","last_became_active":"2024-03-28T14:07:59.733884+0000","last_became_peered":"2024-03-28T14:07:59.733884+0000","last_unstale":"2024-03-28T14:08:00.680302+0000","last_undegraded":"2024-03-28T14:08:00.680302+0000","last_fullsized":"2024-03-28T14:08:00.680302+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_clean_scrub_stamp":"2024-03-28T14:07:58.641775+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T16:47:04.317583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-28T14:08:00.681010+0000","last_change":"2024-03-28T14:07:59.716106+0000","last_active":"2024-03-28T14:08:00.681010+0000","last_peered":"2024-03-28T14:08:00.681010+0000","last_clean":"2024-03-28T14:08:00.681010+0000","last_became_active":"2024-03-28T14:07:59.715747+0000","last_became_peered":"2024-03-28T14:07:59.715747+0000","last_unstale":"2024-03-28T14:08:00.681010+0000","last_undegraded":"2024-03-28T14:08:00.681010+0000","last_fullsized":"2024-03-28T14:08:00.681010+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_clean_scrub_stamp":"2024-03-28T14:07:58.641775+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:25:39.647896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-28T14:08:00.703528+0000","last_change":"2024-03-28T14:07:59.710509+0000","last_active":"2024-03-28T14:08:00.703528+0000","last_peered":"2024-03-28T14:08:00.703528+0000","last_clean":"2024-03-28T14:08:00.703528+0000","last_became_active":"2024-03-28T14:07:59.709767+0000","last_became_peered":"2024-03-28T14:07:59.709767+0000","last_unstale":"2024-03-28T14:08:00.703528+0000","last_undegraded":"2024-03-28T14:08:00.703528+0000","last_fullsized":"2024-03-28T14:08:00.703528+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_clean_scrub_stamp":"2024-03-28T14:07:58.641775+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:41:27.409736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-28T14:08:00.696547+0000","last_change":"2024-03-28T14:07:59.704832+0000","last_active":"2024-03-28T14:08:00.696547+0000","last_peered":"2024-03-28T14:08:00.696547+0000","last_clean":"2024-03-28T14:08:00.696547+0000","last_became_active":"2024-03-28T14:07:59.704546+0000","last_became_peered":"2024-03-28T14:07:59.704546+0000","last_unstale":"2024-03-28T14:08:00.696547+0000","last_undegraded":"2024-03-28T14:08:00.696547+0000","last_fullsized":"2024-03-28T14:08:00.696547+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_clean_scrub_stamp":"2024-03-28T14:07:58.641775+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:41:27.409736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-28T14:08:00.681706+0000","last_change":"2024-03-28T14:07:59.702155+0000","last_active":"2024-03-28T14:08:00.681706+0000","last_peered":"2024-03-28T14:08:00.681706+0000","last_clean":"2024-03-28T14:08:00.681706+0000","last_became_active":"2024-03-28T14:07:59.701799+0000","last_became_peered":"2024-03-28T14:07:59.701799+0000","last_unstale":"2024-03-28T14:08:00.681706+0000","last_undegraded":"2024-03-28T14:08:00.681706+0000","last_fullsized":"2024-03-28T14:08:00.681706+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_clean_scrub_stamp":"2024-03-28T14:07:58.641775+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:10:31.257059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-28T14:08:01.244122+0000","last_change":"2024-03-28T14:07:59.706889+0000","last_active":"2024-03-28T14:08:01.244122+0000","last_peered":"2024-03-28T14:08:01.244122+0000","last_clean":"2024-03-28T14:08:01.244122+0000","last_became_active":"2024-03-28T14:07:59.706631+0000","last_became_peered":"2024-03-28T14:07:59.706631+0000","last_unstale":"2024-03-28T14:08:01.244122+0000","last_undegraded":"2024-03-28T14:08:01.244122+0000","last_fullsized":"2024-03-28T14:08:01.244122+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_clean_scrub_stamp":"2024-03-28T14:07:58.641775+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:57:11.908191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-28T14:08:00.680602+0000","last_change":"2024-03-28T14:07:59.722801+0000","last_active":"2024-03-28T14:08:00.680602+0000","last_peered":"2024-03-28T14:08:00.680602+0000","last_clean":"2024-03-28T14:08:00.680602+0000","last_became_active":"2024-03-28T14:07:59.722132+0000","last_became_peered":"2024-03-28T14:07:59.722132+0000","last_unstale":"2024-03-28T14:08:00.680602+0000","last_undegraded":"2024-03-28T14:08:00.680602+0000","last_fullsized":"2024-03-28T14:08:00.680602+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_clean_scrub_stamp":"2024-03-28T14:07:58.641775+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T18:51:19.214005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-28T14:08:01.242238+0000","last_change":"2024-03-28T14:07:59.740448+0000","last_active":"2024-03-28T14:08:01.242238+0000","last_peered":"2024-03-28T14:08:01.242238+0000","last_clean":"2024-03-28T14:08:01.242238+0000","last_became_active":"2024-03-28T14:07:59.740121+0000","last_became_peered":"2024-03-28T14:07:59.740121+0000","last_unstale":"2024-03-28T14:08:01.242238+0000","last_undegraded":"2024-03-28T14:08:01.242238+0000","last_fullsized":"2024-03-28T14:08:01.242238+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_clean_scrub_stamp":"2024-03-28T14:07:58.641775+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T15:40:59.112394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-28T14:08:00.702471+0000","last_change":"2024-03-28T14:07:59.704155+0000","last_active":"2024-03-28T14:08:00.702471+0000","last_peered":"2024-03-28T14:08:00.702471+0000","last_clean":"2024-03-28T14:08:00.702471+0000","last_became_active":"2024-03-28T14:07:59.702173+0000","last_became_peered":"2024-03-28T14:07:59.702173+0000","last_unstale":"2024-03-28T14:08:00.702471+0000","last_undegraded":"2024-03-28T14:08:00.702471+0000","last_fullsized":"2024-03-28T14:08:00.702471+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_clean_scrub_stamp":"2024-03-28T14:07:58.641775+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T14:20:33.555524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-28T14:08:01.243508+0000","last_change":"2024-03-28T14:07:59.682929+0000","last_active":"2024-03-28T14:08:01.243508+0000","last_peered":"2024-03-28T14:08:01.243508+0000","last_clean":"2024-03-28T14:08:01.243508+0000","last_became_active":"2024-03-28T14:07:59.682238+0000","last_became_peered":"2024-03-28T14:07:59.682238+0000","last_unstale":"2024-03-28T14:08:01.243508+0000","last_undegraded":"2024-03-28T14:08:01.243508+0000","last_fullsized":"2024-03-28T14:08:01.243508+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_clean_scrub_stamp":"2024-03-28T14:07:58.641775+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T17:35:01.163718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"20'1","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-28T14:08:00.681702+0000","last_change":"2024-03-28T14:07:58.672879+0000","last_active":"2024-03-28T14:08:00.681702+0000","last_peered":"2024-03-28T14:08:00.681702+0000","last_clean":"2024-03-28T14:08:00.681702+0000","last_became_active":"2024-03-28T14:07:58.672646+0000","last_became_peered":"2024-03-28T14:07:58.672646+0000","last_unstale":"2024-03-28T14:08:00.681702+0000","last_undegraded":"2024-03-28T14:08:00.681702+0000","last_fullsized":"2024-03-28T14:08:00.681702+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:07:57.631994+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:07:57.631994+0000","last_clean_scrub_stamp":"2024-03-28T14:07:57.631994+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:35:03.722392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-28T14:08:01.244389+0000","last_change":"2024-03-28T14:07:59.715972+0000","last_active":"2024-03-28T14:08:01.244389+0000","last_peered":"2024-03-28T14:08:01.244389+0000","last_clean":"2024-03-28T14:08:01.244389+0000","last_became_active":"2024-03-28T14:07:59.715621+0000","last_became_peered":"2024-03-28T14:07:59.715621+0000","last_unstale":"2024-03-28T14:08:01.244389+0000","last_undegraded":"2024-03-28T14:08:01.244389+0000","last_fullsized":"2024-03-28T14:08:01.244389+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_clean_scrub_stamp":"2024-03-28T14:07:58.641775+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:24:30.197596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1a","version":"20'1","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-28T14:08:00.702735+0000","last_change":"2024-03-28T14:07:58.688093+0000","last_active":"2024-03-28T14:08:00.702735+0000","last_peered":"2024-03-28T14:08:00.702735+0000","last_clean":"2024-03-28T14:08:00.702735+0000","last_became_active":"2024-03-28T14:07:58.687279+0000","last_became_peered":"2024-03-28T14:07:58.687279+0000","last_unstale":"2024-03-28T14:08:00.702735+0000","last_undegraded":"2024-03-28T14:08:00.702735+0000","last_fullsized":"2024-03-28T14:08:00.702735+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:07:57.631994+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:07:57.631994+0000","last_clean_scrub_stamp":"2024-03-28T14:07:57.631994+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T18:00:55.131543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-28T14:08:00.680142+0000","last_change":"2024-03-28T14:07:58.678668+0000","last_active":"2024-03-28T14:08:00.680142+0000","last_peered":"2024-03-28T14:08:00.680142+0000","last_clean":"2024-03-28T14:08:00.680142+0000","last_became_active":"2024-03-28T14:07:58.678292+0000","last_became_peered":"2024-03-28T14:07:58.678292+0000","last_unstale":"2024-03-28T14:08:00.680142+0000","last_undegraded":"2024-03-28T14:08:00.680142+0000","last_fullsized":"2024-03-28T14:08:00.680142+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:07:57.631994+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:07:57.631994+0000","last_clean_scrub_stamp":"2024-03-28T14:07:57.631994+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T18:00:55.131543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-28T14:08:00.680364+0000","last_change":"2024-03-28T14:07:59.703432+0000","last_active":"2024-03-28T14:08:00.680364+0000","last_peered":"2024-03-28T14:08:00.680364+0000","last_clean":"2024-03-28T14:08:00.680364+0000","last_became_active":"2024-03-28T14:07:59.702896+0000","last_became_peered":"2024-03-28T14:07:59.702896+0000","last_unstale":"2024-03-28T14:08:00.680364+0000","last_undegraded":"2024-03-28T14:08:00.680364+0000","last_fullsized":"2024-03-28T14:08:00.680364+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_clean_scrub_stamp":"2024-03-28T14:07:58.641775+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:10:27.248645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-28T14:08:00.680262+0000","last_change":"2024-03-28T14:07:58.703631+0000","last_active":"2024-03-28T14:08:00.680262+0000","last_peered":"2024-03-28T14:08:00.680262+0000","last_clean":"2024-03-28T14:08:00.680262+0000","last_became_active":"2024-03-28T14:07:58.702897+0000","last_became_peered":"2024-03-28T14:07:58.702897+0000","last_unstale":"2024-03-28T14:08:00.680262+0000","last_undegraded":"2024-03-28T14:08:00.680262+0000","last_fullsized":"2024-03-28T14:08:00.680262+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:07:57.631994+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:07:57.631994+0000","last_clean_scrub_stamp":"2024-03-28T14:07:57.631994+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:35:03.722392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-28T14:08:00.696158+0000","last_change":"2024-03-28T14:07:59.706310+0000","last_active":"2024-03-28T14:08:00.696158+0000","last_peered":"2024-03-28T14:08:00.696158+0000","last_clean":"2024-03-28T14:08:00.696158+0000","last_became_active":"2024-03-28T14:07:59.705967+0000","last_became_peered":"2024-03-28T14:07:59.705967+0000","last_unstale":"2024-03-28T14:08:00.696158+0000","last_undegraded":"2024-03-28T14:08:00.696158+0000","last_fullsized":"2024-03-28T14:08:00.696158+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_clean_scrub_stamp":"2024-03-28T14:07:58.641775+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:24:30.197596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-28T14:08:00.703408+0000","last_change":"2024-03-28T14:07:58.702042+0000","last_active":"2024-03-28T14:08:00.703408+0000","last_peered":"2024-03-28T14:08:00.703408+0000","last_clean":"2024-03-28T14:08:00.703408+0000","last_became_active":"2024-03-28T14:07:58.701395+0000","last_became_peered":"2024-03-28T14:07:58.701395+0000","last_unstale":"2024-03-28T14:08:00.703408+0000","last_undegraded":"2024-03-28T14:08:00.703408+0000","last_fullsized":"2024-03-28T14:08:00.703408+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:07:57.631994+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:07:57.631994+0000","last_clean_scrub_stamp":"2024-03-28T14:07:57.631994+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T17:35:00.153938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-28T14:08:00.681271+0000","last_change":"2024-03-28T14:07:59.740904+0000","last_active":"2024-03-28T14:08:00.681271+0000","last_peered":"2024-03-28T14:08:00.681271+0000","last_clean":"2024-03-28T14:08:00.681271+0000","last_became_active":"2024-03-28T14:07:59.740209+0000","last_became_peered":"2024-03-28T14:07:59.740209+0000","last_unstale":"2024-03-28T14:08:00.681271+0000","last_undegraded":"2024-03-28T14:08:00.681271+0000","last_fullsized":"2024-03-28T14:08:00.681271+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_clean_scrub_stamp":"2024-03-28T14:07:58.641775+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:02:48.017536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-28T14:08:00.681223+0000","last_change":"2024-03-28T14:07:58.699895+0000","last_active":"2024-03-28T14:08:00.681223+0000","last_peered":"2024-03-28T14:08:00.681223+0000","last_clean":"2024-03-28T14:08:00.681223+0000","last_became_active":"2024-03-28T14:07:58.698823+0000","last_became_peered":"2024-03-28T14:07:58.698823+0000","last_unstale":"2024-03-28T14:08:00.681223+0000","last_undegraded":"2024-03-28T14:08:00.681223+0000","last_fullsized":"2024-03-28T14:08:00.681223+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:07:57.631994+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:07:57.631994+0000","last_clean_scrub_stamp":"2024-03-28T14:07:57.631994+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:09:38.264121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-28T14:08:01.245669+0000","last_change":"2024-03-28T14:07:59.744137+0000","last_active":"2024-03-28T14:08:01.245669+0000","last_peered":"2024-03-28T14:08:01.245669+0000","last_clean":"2024-03-28T14:08:01.245669+0000","last_became_active":"2024-03-28T14:07:59.743854+0000","last_became_peered":"2024-03-28T14:07:59.743854+0000","last_unstale":"2024-03-28T14:08:01.245669+0000","last_undegraded":"2024-03-28T14:08:01.245669+0000","last_fullsized":"2024-03-28T14:08:01.245669+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_clean_scrub_stamp":"2024-03-28T14:07:58.641775+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:41:27.409736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"20'3","reported_seq":17,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-28T14:08:00.696618+0000","last_change":"2024-03-28T14:07:58.703242+0000","last_active":"2024-03-28T14:08:00.696618+0000","last_peered":"2024-03-28T14:08:00.696618+0000","last_clean":"2024-03-28T14:08:00.696618+0000","last_became_active":"2024-03-28T14:07:58.702970+0000","last_became_peered":"2024-03-28T14:07:58.702970+0000","last_unstale":"2024-03-28T14:08:00.696618+0000","last_undegraded":"2024-03-28T14:08:00.696618+0000","last_fullsized":"2024-03-28T14:08:00.696618+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:07:57.631994+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:07:57.631994+0000","last_clean_scrub_stamp":"2024-03-28T14:07:57.631994+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T15:40:58.102613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-28T14:08:01.244046+0000","last_change":"2024-03-28T14:07:59.708551+0000","last_active":"2024-03-28T14:08:01.244046+0000","last_peered":"2024-03-28T14:08:01.244046+0000","last_clean":"2024-03-28T14:08:01.244046+0000","last_became_active":"2024-03-28T14:07:59.706665+0000","last_became_peered":"2024-03-28T14:07:59.706665+0000","last_unstale":"2024-03-28T14:08:01.244046+0000","last_undegraded":"2024-03-28T14:08:01.244046+0000","last_fullsized":"2024-03-28T14:08:01.244046+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_clean_scrub_stamp":"2024-03-28T14:07:58.641775+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T15:40:01.667745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.14","version":"20'6","reported_seq":20,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-28T14:08:00.702870+0000","last_change":"2024-03-28T14:07:58.680343+0000","last_active":"2024-03-28T14:08:00.702870+0000","last_peered":"2024-03-28T14:08:00.702870+0000","last_clean":"2024-03-28T14:08:00.702870+0000","last_became_active":"2024-03-28T14:07:58.679621+0000","last_became_peered":"2024-03-28T14:07:58.679621+0000","last_unstale":"2024-03-28T14:08:00.702870+0000","last_undegraded":"2024-03-28T14:08:00.702870+0000","last_fullsized":"2024-03-28T14:08:00.702870+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:07:57.631994+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:07:57.631994+0000","last_clean_scrub_stamp":"2024-03-28T14:07:57.631994+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:30:19.088624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-28T14:08:00.834351+0000","last_change":"2024-03-28T14:07:59.714064+0000","last_active":"2024-03-28T14:08:00.834351+0000","last_peered":"2024-03-28T14:08:00.834351+0000","last_clean":"2024-03-28T14:08:00.834351+0000","last_became_active":"2024-03-28T14:07:59.710619+0000","last_became_peered":"2024-03-28T14:07:59.710619+0000","last_unstale":"2024-03-28T14:08:00.834351+0000","last_undegraded":"2024-03-28T14:08:00.834351+0000","last_fullsized":"2024-03-28T14:08:00.834351+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_clean_scrub_stamp":"2024-03-28T14:07:58.641775+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:34:47.572488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"20'2","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-28T14:08:01.244515+0000","last_change":"2024-03-28T14:07:58.684715+0000","last_active":"2024-03-28T14:08:01.244515+0000","last_peered":"2024-03-28T14:08:01.244515+0000","last_clean":"2024-03-28T14:08:01.244515+0000","last_became_active":"2024-03-28T14:07:58.684314+0000","last_became_peered":"2024-03-28T14:07:58.684314+0000","last_unstale":"2024-03-28T14:08:01.244515+0000","last_undegraded":"2024-03-28T14:08:01.244515+0000","last_fullsized":"2024-03-28T14:08:01.244515+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:07:57.631994+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:07:57.631994+0000","last_clean_scrub_stamp":"2024-03-28T14:07:57.631994+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T17:34:09.030115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-28T14:08:01.243877+0000","last_change":"2024-03-28T14:07:59.730071+0000","last_active":"2024-03-28T14:08:01.243877+0000","last_peered":"2024-03-28T14:08:01.243877+0000","last_clean":"2024-03-28T14:08:01.243877+0000","last_became_active":"2024-03-28T14:07:59.729794+0000","last_became_peered":"2024-03-28T14:07:59.729794+0000","last_unstale":"2024-03-28T14:08:01.243877+0000","last_undegraded":"2024-03-28T14:08:01.243877+0000","last_fullsized":"2024-03-28T14:08:01.243877+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_clean_scrub_stamp":"2024-03-28T14:07:58.641775+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:38:43.102577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-28T14:08:00.680825+0000","last_change":"2024-03-28T14:07:58.704245+0000","last_active":"2024-03-28T14:08:00.680825+0000","last_peered":"2024-03-28T14:08:00.680825+0000","last_clean":"2024-03-28T14:08:00.680825+0000","last_became_active":"2024-03-28T14:07:58.703603+0000","last_became_peered":"2024-03-28T14:07:58.703603+0000","last_unstale":"2024-03-28T14:08:00.680825+0000","last_undegraded":"2024-03-28T14:08:00.680825+0000","last_fullsized":"2024-03-28T14:08:00.680825+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:07:57.631994+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:07:57.631994+0000","last_clean_scrub_stamp":"2024-03-28T14:07:57.631994+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T17:35:00.153938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-28T14:08:00.834163+0000","last_change":"2024-03-28T14:07:59.718822+0000","last_active":"2024-03-28T14:08:00.834163+0000","last_peered":"2024-03-28T14:08:00.834163+0000","last_clean":"2024-03-28T14:08:00.834163+0000","last_became_active":"2024-03-28T14:07:59.718058+0000","last_became_peered":"2024-03-28T14:07:59.718058+0000","last_unstale":"2024-03-28T14:08:00.834163+0000","last_undegraded":"2024-03-28T14:08:00.834163+0000","last_fullsized":"2024-03-28T14:08:00.834163+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_clean_scrub_stamp":"2024-03-28T14:07:58.641775+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:10:31.257059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-28T14:08:00.702420+0000","last_change":"2024-03-28T14:07:58.689870+0000","last_active":"2024-03-28T14:08:00.702420+0000","last_peered":"2024-03-28T14:08:00.702420+0000","last_clean":"2024-03-28T14:08:00.702420+0000","last_became_active":"2024-03-28T14:07:58.689121+0000","last_became_peered":"2024-03-28T14:07:58.689121+0000","last_unstale":"2024-03-28T14:08:00.702420+0000","last_undegraded":"2024-03-28T14:08:00.702420+0000","last_fullsized":"2024-03-28T14:08:00.702420+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:07:57.631994+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:07:57.631994+0000","last_clean_scrub_stamp":"2024-03-28T14:07:57.631994+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:35:03.722392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-28T14:08:00.702531+0000","last_change":"2024-03-28T14:07:59.712485+0000","last_active":"2024-03-28T14:08:00.702531+0000","last_peered":"2024-03-28T14:08:00.702531+0000","last_clean":"2024-03-28T14:08:00.702531+0000","last_became_active":"2024-03-28T14:07:59.704056+0000","last_became_peered":"2024-03-28T14:07:59.704056+0000","last_unstale":"2024-03-28T14:08:00.702531+0000","last_undegraded":"2024-03-28T14:08:00.702531+0000","last_fullsized":"2024-03-28T14:08:00.702531+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_clean_scrub_stamp":"2024-03-28T14:07:58.641775+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:10:27.248645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"20'8","reported_seq":22,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-28T14:08:01.242042+0000","last_change":"2024-03-28T14:07:58.667252+0000","last_active":"2024-03-28T14:08:01.242042+0000","last_peered":"2024-03-28T14:08:01.242042+0000","last_clean":"2024-03-28T14:08:01.242042+0000","last_became_active":"2024-03-28T14:07:58.666837+0000","last_became_peered":"2024-03-28T14:07:58.666837+0000","last_unstale":"2024-03-28T14:08:01.242042+0000","last_undegraded":"2024-03-28T14:08:01.242042+0000","last_fullsized":"2024-03-28T14:08:01.242042+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:07:57.631994+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:07:57.631994+0000","last_clean_scrub_stamp":"2024-03-28T14:07:57.631994+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:30:19.088624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-28T14:08:00.833905+0000","last_change":"2024-03-28T14:07:59.719433+0000","last_active":"2024-03-28T14:08:00.833905+0000","last_peered":"2024-03-28T14:08:00.833905+0000","last_clean":"2024-03-28T14:08:00.833905+0000","last_became_active":"2024-03-28T14:07:59.718644+0000","last_became_peered":"2024-03-28T14:07:59.718644+0000","last_unstale":"2024-03-28T14:08:00.833905+0000","last_undegraded":"2024-03-28T14:08:00.833905+0000","last_fullsized":"2024-03-28T14:08:00.833905+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_clean_scrub_stamp":"2024-03-28T14:07:58.641775+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T17:46:06.253111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"20'1","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-28T14:08:01.243993+0000","last_change":"2024-03-28T14:07:58.674598+0000","last_active":"2024-03-28T14:08:01.243993+0000","last_peered":"2024-03-28T14:08:01.243993+0000","last_clean":"2024-03-28T14:08:01.243993+0000","last_became_active":"2024-03-28T14:07:58.673894+0000","last_became_peered":"2024-03-28T14:07:58.673894+0000","last_unstale":"2024-03-28T14:08:01.243993+0000","last_undegraded":"2024-03-28T14:08:01.243993+0000","last_fullsized":"2024-03-28T14:08:01.243993+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:07:57.631994+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:07:57.631994+0000","last_clean_scrub_stamp":"2024-03-28T14:07:57.631994+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:32:35.921987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-28T14:08:00.834711+0000","last_change":"2024-03-28T14:07:59.725643+0000","last_active":"2024-03-28T14:08:00.834711+0000","last_peered":"2024-03-28T14:08:00.834711+0000","last_clean":"2024-03-28T14:08:00.834711+0000","last_became_active":"2024-03-28T14:07:59.725019+0000","last_became_peered":"2024-03-28T14:07:59.725019+0000","last_unstale":"2024-03-28T14:08:00.834711+0000","last_undegraded":"2024-03-28T14:08:00.834711+0000","last_fullsized":"2024-03-28T14:08:00.834711+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_clean_scrub_stamp":"2024-03-28T14:07:58.641775+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:02:48.017536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"20'2","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-28T14:08:01.242483+0000","last_change":"2024-03-28T14:07:58.663444+0000","last_active":"2024-03-28T14:08:01.242483+0000","last_peered":"2024-03-28T14:08:01.242483+0000","last_clean":"2024-03-28T14:08:01.242483+0000","last_became_active":"2024-03-28T14:07:58.663001+0000","last_became_peered":"2024-03-28T14:07:58.663001+0000","last_unstale":"2024-03-28T14:08:01.242483+0000","last_undegraded":"2024-03-28T14:08:01.242483+0000","last_fullsized":"2024-03-28T14:08:01.242483+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:07:57.631994+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:07:57.631994+0000","last_clean_scrub_stamp":"2024-03-28T14:07:57.631994+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:35:03.722392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-28T14:08:00.835294+0000","last_change":"2024-03-28T14:07:59.731950+0000","last_active":"2024-03-28T14:08:00.835294+0000","last_peered":"2024-03-28T14:08:00.835294+0000","last_clean":"2024-03-28T14:08:00.835294+0000","last_became_active":"2024-03-28T14:07:59.731208+0000","last_became_peered":"2024-03-28T14:07:59.731208+0000","last_unstale":"2024-03-28T14:08:00.835294+0000","last_undegraded":"2024-03-28T14:08:00.835294+0000","last_fullsized":"2024-03-28T14:08:00.835294+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_clean_scrub_stamp":"2024-03-28T14:07:58.641775+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T14:17:34.720686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"20'1","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-28T14:08:00.696647+0000","last_change":"2024-03-28T14:07:58.684273+0000","last_active":"2024-03-28T14:08:00.696647+0000","last_peered":"2024-03-28T14:08:00.696647+0000","last_clean":"2024-03-28T14:08:00.696647+0000","last_became_active":"2024-03-28T14:07:58.683886+0000","last_became_peered":"2024-03-28T14:07:58.683886+0000","last_unstale":"2024-03-28T14:08:00.696647+0000","last_undegraded":"2024-03-28T14:08:00.696647+0000","last_fullsized":"2024-03-28T14:08:00.696647+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:07:57.631994+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:07:57.631994+0000","last_clean_scrub_stamp":"2024-03-28T14:07:57.631994+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T15:40:00.657964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-28T14:08:00.681505+0000","last_change":"2024-03-28T14:07:59.739696+0000","last_active":"2024-03-28T14:08:00.681505+0000","last_peered":"2024-03-28T14:08:00.681505+0000","last_clean":"2024-03-28T14:08:00.681505+0000","last_became_active":"2024-03-28T14:07:59.734736+0000","last_became_peered":"2024-03-28T14:07:59.734736+0000","last_unstale":"2024-03-28T14:08:00.681505+0000","last_undegraded":"2024-03-28T14:08:00.681505+0000","last_fullsized":"2024-03-28T14:08:00.681505+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_clean_scrub_stamp":"2024-03-28T14:07:58.641775+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:51:23.322586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-28T14:08:01.242313+0000","last_change":"2024-03-28T14:07:58.704650+0000","last_active":"2024-03-28T14:08:01.242313+0000","last_peered":"2024-03-28T14:08:01.242313+0000","last_clean":"2024-03-28T14:08:01.242313+0000","last_became_active":"2024-03-28T14:07:58.704305+0000","last_became_peered":"2024-03-28T14:07:58.704305+0000","last_unstale":"2024-03-28T14:08:01.242313+0000","last_undegraded":"2024-03-28T14:08:01.242313+0000","last_fullsized":"2024-03-28T14:08:01.242313+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:07:57.631994+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:07:57.631994+0000","last_clean_scrub_stamp":"2024-03-28T14:07:57.631994+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T17:35:00.153938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-28T14:08:00.834345+0000","last_change":"2024-03-28T14:07:59.745242+0000","last_active":"2024-03-28T14:08:00.834345+0000","last_peered":"2024-03-28T14:08:00.834345+0000","last_clean":"2024-03-28T14:08:00.834345+0000","last_became_active":"2024-03-28T14:07:59.744734+0000","last_became_peered":"2024-03-28T14:07:59.744734+0000","last_unstale":"2024-03-28T14:08:00.834345+0000","last_undegraded":"2024-03-28T14:08:00.834345+0000","last_fullsized":"2024-03-28T14:08:00.834345+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_clean_scrub_stamp":"2024-03-28T14:07:58.641775+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:49:58.161511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":170,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-28T14:08:00.833744+0000","last_change":"2024-03-28T14:07:49.234448+0000","last_active":"2024-03-28T14:08:00.833744+0000","last_peered":"2024-03-28T14:08:00.833744+0000","last_clean":"2024-03-28T14:08:00.833744+0000","last_became_active":"2024-03-28T14:07:49.233895+0000","last_became_peered":"2024-03-28T14:07:49.233895+0000","last_unstale":"2024-03-28T14:08:00.833744+0000","last_undegraded":"2024-03-28T14:08:00.833744+0000","last_fullsized":"2024-03-28T14:08:00.833744+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:07:48.069157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:07:48.069157+0000","last_clean_scrub_stamp":"2024-03-28T14:07:48.069157+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:11:39.977557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-28T14:08:00.680569+0000","last_change":"2024-03-28T14:07:58.699326+0000","last_active":"2024-03-28T14:08:00.680569+0000","last_peered":"2024-03-28T14:08:00.680569+0000","last_clean":"2024-03-28T14:08:00.680569+0000","last_became_active":"2024-03-28T14:07:58.698258+0000","last_became_peered":"2024-03-28T14:07:58.698258+0000","last_unstale":"2024-03-28T14:08:00.680569+0000","last_undegraded":"2024-03-28T14:08:00.680569+0000","last_fullsized":"2024-03-28T14:08:00.680569+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:07:57.631994+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:07:57.631994+0000","last_clean_scrub_stamp":"2024-03-28T14:07:57.631994+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T18:00:55.131543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-28T14:08:00.681381+0000","last_change":"2024-03-28T14:07:59.744015+0000","last_active":"2024-03-28T14:08:00.681381+0000","last_peered":"2024-03-28T14:08:00.681381+0000","last_clean":"2024-03-28T14:08:00.681381+0000","last_became_active":"2024-03-28T14:07:59.743723+0000","last_became_peered":"2024-03-28T14:07:59.743723+0000","last_unstale":"2024-03-28T14:08:00.681381+0000","last_undegraded":"2024-03-28T14:08:00.681381+0000","last_fullsized":"2024-03-28T14:08:00.681381+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_clean_scrub_stamp":"2024-03-28T14:07:58.641775+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T16:47:04.317583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-28T14:08:00.833592+0000","last_change":"2024-03-28T14:07:58.688278+0000","last_active":"2024-03-28T14:08:00.833592+0000","last_peered":"2024-03-28T14:08:00.833592+0000","last_clean":"2024-03-28T14:08:00.833592+0000","last_became_active":"2024-03-28T14:07:58.687442+0000","last_became_peered":"2024-03-28T14:07:58.687442+0000","last_unstale":"2024-03-28T14:08:00.833592+0000","last_undegraded":"2024-03-28T14:08:00.833592+0000","last_fullsized":"2024-03-28T14:08:00.833592+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:07:57.631994+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:07:57.631994+0000","last_clean_scrub_stamp":"2024-03-28T14:07:57.631994+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T17:35:00.153938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-28T14:08:01.243711+0000","last_change":"2024-03-28T14:07:59.715990+0000","last_active":"2024-03-28T14:08:01.243711+0000","last_peered":"2024-03-28T14:08:01.243711+0000","last_clean":"2024-03-28T14:08:01.243711+0000","last_became_active":"2024-03-28T14:07:59.715627+0000","last_became_peered":"2024-03-28T14:07:59.715627+0000","last_unstale":"2024-03-28T14:08:01.243711+0000","last_undegraded":"2024-03-28T14:08:01.243711+0000","last_fullsized":"2024-03-28T14:08:01.243711+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_clean_scrub_stamp":"2024-03-28T14:07:58.641775+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T18:46:53.473825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-28T14:08:01.242294+0000","last_change":"2024-03-28T14:07:58.663475+0000","last_active":"2024-03-28T14:08:01.242294+0000","last_peered":"2024-03-28T14:08:01.242294+0000","last_clean":"2024-03-28T14:08:01.242294+0000","last_became_active":"2024-03-28T14:07:58.662967+0000","last_became_peered":"2024-03-28T14:07:58.662967+0000","last_unstale":"2024-03-28T14:08:01.242294+0000","last_undegraded":"2024-03-28T14:08:01.242294+0000","last_fullsized":"2024-03-28T14:08:01.242294+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:07:57.631994+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:07:57.631994+0000","last_clean_scrub_stamp":"2024-03-28T14:07:57.631994+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T18:00:55.131543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-28T14:08:00.696122+0000","last_change":"2024-03-28T14:07:59.693645+0000","last_active":"2024-03-28T14:08:00.696122+0000","last_peered":"2024-03-28T14:08:00.696122+0000","last_clean":"2024-03-28T14:08:00.696122+0000","last_became_active":"2024-03-28T14:07:59.693346+0000","last_became_peered":"2024-03-28T14:07:59.693346+0000","last_unstale":"2024-03-28T14:08:00.696122+0000","last_undegraded":"2024-03-28T14:08:00.696122+0000","last_fullsized":"2024-03-28T14:08:00.696122+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_clean_scrub_stamp":"2024-03-28T14:07:58.641775+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:57:11.908191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-28T14:08:00.680856+0000","last_change":"2024-03-28T14:07:58.702868+0000","last_active":"2024-03-28T14:08:00.680856+0000","last_peered":"2024-03-28T14:08:00.680856+0000","last_clean":"2024-03-28T14:08:00.680856+0000","last_became_active":"2024-03-28T14:07:58.700099+0000","last_became_peered":"2024-03-28T14:07:58.700099+0000","last_unstale":"2024-03-28T14:08:00.680856+0000","last_undegraded":"2024-03-28T14:08:00.680856+0000","last_fullsized":"2024-03-28T14:08:00.680856+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:07:57.631994+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:07:57.631994+0000","last_clean_scrub_stamp":"2024-03-28T14:07:57.631994+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:30:19.088624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-28T14:08:01.243699+0000","last_change":"2024-03-28T14:07:59.707739+0000","last_active":"2024-03-28T14:08:01.243699+0000","last_peered":"2024-03-28T14:08:01.243699+0000","last_clean":"2024-03-28T14:08:01.243699+0000","last_became_active":"2024-03-28T14:07:59.705252+0000","last_became_peered":"2024-03-28T14:07:59.705252+0000","last_unstale":"2024-03-28T14:08:01.243699+0000","last_undegraded":"2024-03-28T14:08:01.243699+0000","last_fullsized":"2024-03-28T14:08:01.243699+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_clean_scrub_stamp":"2024-03-28T14:07:58.641775+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:09:39.273902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-28T14:08:00.696496+0000","last_change":"2024-03-28T14:07:58.669940+0000","last_active":"2024-03-28T14:08:00.696496+0000","last_peered":"2024-03-28T14:08:00.696496+0000","last_clean":"2024-03-28T14:08:00.696496+0000","last_became_active":"2024-03-28T14:07:58.669568+0000","last_became_peered":"2024-03-28T14:07:58.669568+0000","last_unstale":"2024-03-28T14:08:00.696496+0000","last_undegraded":"2024-03-28T14:08:00.696496+0000","last_fullsized":"2024-03-28T14:08:00.696496+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:07:57.631994+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:07:57.631994+0000","last_clean_scrub_stamp":"2024-03-28T14:07:57.631994+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:09:38.264121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-28T14:08:00.681619+0000","last_change":"2024-03-28T14:07:59.742899+0000","last_active":"2024-03-28T14:08:00.681619+0000","last_peered":"2024-03-28T14:08:00.681619+0000","last_clean":"2024-03-28T14:08:00.681619+0000","last_became_active":"2024-03-28T14:07:59.740075+0000","last_became_peered":"2024-03-28T14:07:59.740075+0000","last_unstale":"2024-03-28T14:08:00.681619+0000","last_undegraded":"2024-03-28T14:08:00.681619+0000","last_fullsized":"2024-03-28T14:08:00.681619+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_clean_scrub_stamp":"2024-03-28T14:07:58.641775+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:59:25.862480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-28T14:08:00.834925+0000","last_change":"2024-03-28T14:07:58.687874+0000","last_active":"2024-03-28T14:08:00.834925+0000","last_peered":"2024-03-28T14:08:00.834925+0000","last_clean":"2024-03-28T14:08:00.834925+0000","last_became_active":"2024-03-28T14:07:58.686656+0000","last_became_peered":"2024-03-28T14:07:58.686656+0000","last_unstale":"2024-03-28T14:08:00.834925+0000","last_undegraded":"2024-03-28T14:08:00.834925+0000","last_fullsized":"2024-03-28T14:08:00.834925+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:07:57.631994+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:07:57.631994+0000","last_clean_scrub_stamp":"2024-03-28T14:07:57.631994+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T15:40:00.657964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-28T14:08:00.696528+0000","last_change":"2024-03-28T14:07:59.700465+0000","last_active":"2024-03-28T14:08:00.696528+0000","last_peered":"2024-03-28T14:08:00.696528+0000","last_clean":"2024-03-28T14:08:00.696528+0000","last_became_active":"2024-03-28T14:07:59.699705+0000","last_became_peered":"2024-03-28T14:07:59.699705+0000","last_unstale":"2024-03-28T14:08:00.696528+0000","last_undegraded":"2024-03-28T14:08:00.696528+0000","last_fullsized":"2024-03-28T14:08:00.696528+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_clean_scrub_stamp":"2024-03-28T14:07:58.641775+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:25:39.647896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"21'1","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-28T14:08:00.700925+0000","last_change":"2024-03-28T14:07:58.681340+0000","last_active":"2024-03-28T14:08:00.700925+0000","last_peered":"2024-03-28T14:08:00.700925+0000","last_clean":"2024-03-28T14:08:00.700925+0000","last_became_active":"2024-03-28T14:07:58.681006+0000","last_became_peered":"2024-03-28T14:07:58.681006+0000","last_unstale":"2024-03-28T14:08:00.700925+0000","last_undegraded":"2024-03-28T14:08:00.700925+0000","last_fullsized":"2024-03-28T14:08:00.700925+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:07:57.631994+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:07:57.631994+0000","last_clean_scrub_stamp":"2024-03-28T14:07:57.631994+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:04:40.185372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-28T14:08:00.681402+0000","last_change":"2024-03-28T14:07:59.727008+0000","last_active":"2024-03-28T14:08:00.681402+0000","last_peered":"2024-03-28T14:08:00.681402+0000","last_clean":"2024-03-28T14:08:00.681402+0000","last_became_active":"2024-03-28T14:07:59.726702+0000","last_became_peered":"2024-03-28T14:07:59.726702+0000","last_unstale":"2024-03-28T14:08:00.681402+0000","last_undegraded":"2024-03-28T14:08:00.681402+0000","last_fullsized":"2024-03-28T14:08:00.681402+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_clean_scrub_stamp":"2024-03-28T14:07:58.641775+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T17:46:06.253111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"20'2","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-28T14:08:00.703440+0000","last_change":"2024-03-28T14:07:58.685500+0000","last_active":"2024-03-28T14:08:00.703440+0000","last_peered":"2024-03-28T14:08:00.703440+0000","last_clean":"2024-03-28T14:08:00.703440+0000","last_became_active":"2024-03-28T14:07:58.684781+0000","last_became_peered":"2024-03-28T14:07:58.684781+0000","last_unstale":"2024-03-28T14:08:00.703440+0000","last_undegraded":"2024-03-28T14:08:00.703440+0000","last_fullsized":"2024-03-28T14:08:00.703440+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:07:57.631994+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:07:57.631994+0000","last_clean_scrub_stamp":"2024-03-28T14:07:57.631994+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:09:38.264121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-28T14:08:01.244918+0000","last_change":"2024-03-28T14:07:59.707407+0000","last_active":"2024-03-28T14:08:01.244918+0000","last_peered":"2024-03-28T14:08:01.244918+0000","last_clean":"2024-03-28T14:08:01.244918+0000","last_became_active":"2024-03-28T14:07:59.707103+0000","last_became_peered":"2024-03-28T14:07:59.707103+0000","last_unstale":"2024-03-28T14:08:01.244918+0000","last_undegraded":"2024-03-28T14:08:01.244918+0000","last_fullsized":"2024-03-28T14:08:01.244918+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_clean_scrub_stamp":"2024-03-28T14:07:58.641775+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:37:29.402787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-28T14:08:00.833570+0000","last_change":"2024-03-28T14:07:58.682456+0000","last_active":"2024-03-28T14:08:00.833570+0000","last_peered":"2024-03-28T14:08:00.833570+0000","last_clean":"2024-03-28T14:08:00.833570+0000","last_became_active":"2024-03-28T14:07:58.678987+0000","last_became_peered":"2024-03-28T14:07:58.678987+0000","last_unstale":"2024-03-28T14:08:00.833570+0000","last_undegraded":"2024-03-28T14:08:00.833570+0000","last_fullsized":"2024-03-28T14:08:00.833570+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:07:57.631994+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:07:57.631994+0000","last_clean_scrub_stamp":"2024-03-28T14:07:57.631994+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:30:19.088624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-28T14:08:01.243764+0000","last_change":"2024-03-28T14:07:59.697567+0000","last_active":"2024-03-28T14:08:01.243764+0000","last_peered":"2024-03-28T14:08:01.243764+0000","last_clean":"2024-03-28T14:08:01.243764+0000","last_became_active":"2024-03-28T14:07:59.693975+0000","last_became_peered":"2024-03-28T14:07:59.693975+0000","last_unstale":"2024-03-28T14:08:01.243764+0000","last_undegraded":"2024-03-28T14:08:01.243764+0000","last_fullsized":"2024-03-28T14:08:01.243764+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_clean_scrub_stamp":"2024-03-28T14:07:58.641775+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:04:41.195153+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-28T14:08:00.696246+0000","last_change":"2024-03-28T14:07:58.680918+0000","last_active":"2024-03-28T14:08:00.696246+0000","last_peered":"2024-03-28T14:08:00.696246+0000","last_clean":"2024-03-28T14:08:00.696246+0000","last_became_active":"2024-03-28T14:07:58.680548+0000","last_became_peered":"2024-03-28T14:07:58.680548+0000","last_unstale":"2024-03-28T14:08:00.696246+0000","last_undegraded":"2024-03-28T14:08:00.696246+0000","last_fullsized":"2024-03-28T14:08:00.696246+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:07:57.631994+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:07:57.631994+0000","last_clean_scrub_stamp":"2024-03-28T14:07:57.631994+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:30:19.088624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-28T14:08:00.680594+0000","last_change":"2024-03-28T14:07:59.702172+0000","last_active":"2024-03-28T14:08:00.680594+0000","last_peered":"2024-03-28T14:08:00.680594+0000","last_clean":"2024-03-28T14:08:00.680594+0000","last_became_active":"2024-03-28T14:07:59.701709+0000","last_became_peered":"2024-03-28T14:07:59.701709+0000","last_unstale":"2024-03-28T14:08:00.680594+0000","last_undegraded":"2024-03-28T14:08:00.680594+0000","last_fullsized":"2024-03-28T14:08:00.680594+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_clean_scrub_stamp":"2024-03-28T14:07:58.641775+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:57:11.908191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-28T14:08:00.696422+0000","last_change":"2024-03-28T14:07:58.672681+0000","last_active":"2024-03-28T14:08:00.696422+0000","last_peered":"2024-03-28T14:08:00.696422+0000","last_clean":"2024-03-28T14:08:00.696422+0000","last_became_active":"2024-03-28T14:07:58.672094+0000","last_became_peered":"2024-03-28T14:07:58.672094+0000","last_unstale":"2024-03-28T14:08:00.696422+0000","last_undegraded":"2024-03-28T14:08:00.696422+0000","last_fullsized":"2024-03-28T14:08:00.696422+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:07:57.631994+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:07:57.631994+0000","last_clean_scrub_stamp":"2024-03-28T14:07:57.631994+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T17:35:00.153938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-28T14:08:00.681203+0000","last_change":"2024-03-28T14:07:59.705251+0000","last_active":"2024-03-28T14:08:00.681203+0000","last_peered":"2024-03-28T14:08:00.681203+0000","last_clean":"2024-03-28T14:08:00.681203+0000","last_became_active":"2024-03-28T14:07:59.703366+0000","last_became_peered":"2024-03-28T14:07:59.703366+0000","last_unstale":"2024-03-28T14:08:00.681203+0000","last_undegraded":"2024-03-28T14:08:00.681203+0000","last_fullsized":"2024-03-28T14:08:00.681203+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_clean_scrub_stamp":"2024-03-28T14:07:58.641775+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:41:27.409736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"20'2","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-28T14:08:00.834672+0000","last_change":"2024-03-28T14:07:58.688811+0000","last_active":"2024-03-28T14:08:00.834672+0000","last_peered":"2024-03-28T14:08:00.834672+0000","last_clean":"2024-03-28T14:08:00.834672+0000","last_became_active":"2024-03-28T14:07:58.688054+0000","last_became_peered":"2024-03-28T14:07:58.688054+0000","last_unstale":"2024-03-28T14:08:00.834672+0000","last_undegraded":"2024-03-28T14:08:00.834672+0000","last_fullsized":"2024-03-28T14:08:00.834672+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:07:57.631994+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:07:57.631994+0000","last_clean_scrub_stamp":"2024-03-28T14:07:57.631994+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:04:40.185372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-28T14:08:00.702555+0000","last_change":"2024-03-28T14:07:59.709224+0000","last_active":"2024-03-28T14:08:00.702555+0000","last_peered":"2024-03-28T14:08:00.702555+0000","last_clean":"2024-03-28T14:08:00.702555+0000","last_became_active":"2024-03-28T14:07:59.704202+0000","last_became_peered":"2024-03-28T14:07:59.704202+0000","last_unstale":"2024-03-28T14:08:00.702555+0000","last_undegraded":"2024-03-28T14:08:00.702555+0000","last_fullsized":"2024-03-28T14:08:00.702555+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_clean_scrub_stamp":"2024-03-28T14:07:58.641775+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:57:11.908191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-28T14:08:00.681258+0000","last_change":"2024-03-28T14:07:59.739995+0000","last_active":"2024-03-28T14:08:00.681258+0000","last_peered":"2024-03-28T14:08:00.681258+0000","last_clean":"2024-03-28T14:08:00.681258+0000","last_became_active":"2024-03-28T14:07:59.739369+0000","last_became_peered":"2024-03-28T14:07:59.739369+0000","last_unstale":"2024-03-28T14:08:00.681258+0000","last_undegraded":"2024-03-28T14:08:00.681258+0000","last_fullsized":"2024-03-28T14:08:00.681258+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_clean_scrub_stamp":"2024-03-28T14:07:58.641775+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:34:47.572488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"20'2","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-28T14:08:00.834148+0000","last_change":"2024-03-28T14:07:58.705499+0000","last_active":"2024-03-28T14:08:00.834148+0000","last_peered":"2024-03-28T14:08:00.834148+0000","last_clean":"2024-03-28T14:08:00.834148+0000","last_became_active":"2024-03-28T14:07:58.704848+0000","last_became_peered":"2024-03-28T14:07:58.704848+0000","last_unstale":"2024-03-28T14:08:00.834148+0000","last_undegraded":"2024-03-28T14:08:00.834148+0000","last_fullsized":"2024-03-28T14:08:00.834148+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:07:57.631994+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:07:57.631994+0000","last_clean_scrub_stamp":"2024-03-28T14:07:57.631994+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:09:38.264121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-28T14:08:00.680957+0000","last_change":"2024-03-28T14:07:59.740931+0000","last_active":"2024-03-28T14:08:00.680957+0000","last_peered":"2024-03-28T14:08:00.680957+0000","last_clean":"2024-03-28T14:08:00.680957+0000","last_became_active":"2024-03-28T14:07:59.740255+0000","last_became_peered":"2024-03-28T14:07:59.740255+0000","last_unstale":"2024-03-28T14:08:00.680957+0000","last_undegraded":"2024-03-28T14:08:00.680957+0000","last_fullsized":"2024-03-28T14:08:00.680957+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_clean_scrub_stamp":"2024-03-28T14:07:58.641775+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:49:58.161511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"20'2","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-28T14:08:00.834791+0000","last_change":"2024-03-28T14:07:58.693440+0000","last_active":"2024-03-28T14:08:00.834791+0000","last_peered":"2024-03-28T14:08:00.834791+0000","last_clean":"2024-03-28T14:08:00.834791+0000","last_became_active":"2024-03-28T14:07:58.692096+0000","last_became_peered":"2024-03-28T14:07:58.692096+0000","last_unstale":"2024-03-28T14:08:00.834791+0000","last_undegraded":"2024-03-28T14:08:00.834791+0000","last_fullsized":"2024-03-28T14:08:00.834791+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:07:57.631994+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:07:57.631994+0000","last_clean_scrub_stamp":"2024-03-28T14:07:57.631994+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T15:40:58.102613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-28T14:08:01.244001+0000","last_change":"2024-03-28T14:07:59.744398+0000","last_active":"2024-03-28T14:08:01.244001+0000","last_peered":"2024-03-28T14:08:01.244001+0000","last_clean":"2024-03-28T14:08:01.244001+0000","last_became_active":"2024-03-28T14:07:59.744127+0000","last_became_peered":"2024-03-28T14:07:59.744127+0000","last_unstale":"2024-03-28T14:08:01.244001+0000","last_undegraded":"2024-03-28T14:08:01.244001+0000","last_fullsized":"2024-03-28T14:08:01.244001+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_clean_scrub_stamp":"2024-03-28T14:07:58.641775+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T14:20:33.555524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"20'1","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-28T14:08:00.681112+0000","last_change":"2024-03-28T14:07:58.671168+0000","last_active":"2024-03-28T14:08:00.681112+0000","last_peered":"2024-03-28T14:08:00.681112+0000","last_clean":"2024-03-28T14:08:00.681112+0000","last_became_active":"2024-03-28T14:07:58.670691+0000","last_became_peered":"2024-03-28T14:07:58.670691+0000","last_unstale":"2024-03-28T14:08:00.681112+0000","last_undegraded":"2024-03-28T14:08:00.681112+0000","last_fullsized":"2024-03-28T14:08:00.681112+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:07:57.631994+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:07:57.631994+0000","last_clean_scrub_stamp":"2024-03-28T14:07:57.631994+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T17:34:09.030115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-28T14:08:00.680644+0000","last_change":"2024-03-28T14:07:59.727344+0000","last_active":"2024-03-28T14:08:00.680644+0000","last_peered":"2024-03-28T14:08:00.680644+0000","last_clean":"2024-03-28T14:08:00.680644+0000","last_became_active":"2024-03-28T14:07:59.727023+0000","last_became_peered":"2024-03-28T14:07:59.727023+0000","last_unstale":"2024-03-28T14:08:00.680644+0000","last_undegraded":"2024-03-28T14:08:00.680644+0000","last_fullsized":"2024-03-28T14:08:00.680644+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_clean_scrub_stamp":"2024-03-28T14:07:58.641775+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:24:30.197596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"20'4","reported_seq":18,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-28T14:08:01.245810+0000","last_change":"2024-03-28T14:07:58.676958+0000","last_active":"2024-03-28T14:08:01.245810+0000","last_peered":"2024-03-28T14:08:01.245810+0000","last_clean":"2024-03-28T14:08:01.245810+0000","last_became_active":"2024-03-28T14:07:58.676611+0000","last_became_peered":"2024-03-28T14:07:58.676611+0000","last_unstale":"2024-03-28T14:08:01.245810+0000","last_undegraded":"2024-03-28T14:08:01.245810+0000","last_fullsized":"2024-03-28T14:08:01.245810+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:07:57.631994+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:07:57.631994+0000","last_clean_scrub_stamp":"2024-03-28T14:07:57.631994+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:59:04.674426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574854,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27820,"kb_used_data":796,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344020,"statfs":{"total":96636764160,"available":96608276480,"internally_reserved":0,"allocated":815104,"data_stored":666555,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574855,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27248,"kb_used_data":224,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344592,"statfs":{"total":96636764160,"available":96608862208,"internally_reserved":0,"allocated":229376,"data_stored":77217,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574853,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27240,"kb_used_data":216,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344600,"statfs":{"total":96636764160,"available":96608870400,"internally_reserved":0,"allocated":221184,"data_stored":75695,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":11,"apply_latency_ms":11,"commit_latency_ns":11000000,"apply_latency_ns":11000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574854,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27248,"kb_used_data":224,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344592,"statfs":{"total":96636764160,"available":96608862208,"internally_reserved":0,"allocated":229376,"data_stored":75731,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574856,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27248,"kb_used_data":224,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344592,"statfs":{"total":96636764160,"available":96608862208,"internally_reserved":0,"allocated":229376,"data_stored":77217,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574854,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27236,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344604,"statfs":{"total":96636764160,"available":96608874496,"internally_reserved":0,"allocated":217088,"data_stored":75605,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574852,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27244,"kb_used_data":220,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344596,"statfs":{"total":96636764160,"available":96608866304,"internally_reserved":0,"allocated":225280,"data_stored":75765,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574855,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27824,"kb_used_data":804,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344016,"statfs":{"total":96636764160,"available":96608272384,"internally_reserved":0,"allocated":823296,"data_stored":666611,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-28T14:08:03.775 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-28T14:08:03.920 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:03.919+0000 7f99625c7640 1 Processor -- start 2024-03-28T14:08:03.920 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:03.919+0000 7f99625c7640 1 -- start start 2024-03-28T14:08:03.921 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:03.919+0000 7f99625c7640 1 --2- >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f995c0afd00 0x7f995c148410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:03.921 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:03.919+0000 7f99625c7640 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f995c148950 0x7f995c152260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:03.921 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:03.919+0000 7f99625c7640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f995c02c840 con 0x7f995c02c460 2024-03-28T14:08:03.921 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:03.919+0000 7f99625c7640 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 -- 0x7f995c0af5b0 con 0x7f995c148950 2024-03-28T14:08:03.922 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:03.919+0000 7f99625c7640 1 -- --> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] -- mon_getmap magic: 0 -- 0x7f995c0af870 con 0x7f995c0afd00 2024-03-28T14:08:03.922 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:03.919+0000 7f9960b64640 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f995c148950 0x7f995c152260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:03.922 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:03.919+0000 7f9960b64640 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f995c148950 0x7f995c152260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:42334/0 (socket says 172.21.15.104:42334) 2024-03-28T14:08:03.922 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:03.919+0000 7f9960b64640 1 -- 172.21.15.104:0/3795247595 learned_addr learned my addr 172.21.15.104:0/3795247595 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-03-28T14:08:03.922 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:03.919+0000 7f995b7fe640 1 --2- 172.21.15.104:0/3795247595 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f995c0afd00 0x7f995c148410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:03.922 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:03.919+0000 7f9960b64640 1 -- 172.21.15.104:0/3795247595 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f995c0afd00 msgr2=0x7f995c148410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T14:08:03.922 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:03.919+0000 7f9960b64640 1 --2- 172.21.15.104:0/3795247595 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f995c0afd00 0x7f995c148410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:03.922 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:03.919+0000 7f9960b64640 1 -- 172.21.15.104:0/3795247595 >> v1:172.21.15.171:6789/0 conn(0x7f995c02c460 legacy=0x7f995c06cf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T14:08:03.922 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:03.919+0000 7f9960b64640 1 -- 172.21.15.104:0/3795247595 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f995c06e3a0 con 0x7f995c148950 2024-03-28T14:08:03.923 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:03.923+0000 7f9960b64640 1 --2- 172.21.15.104:0/3795247595 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f995c148950 0x7f995c152260 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f9954009e50 tx=0x7f995401c9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f43b75dc8683b3e5 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:08:03.923 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:03.923+0000 7f995affd640 1 -- 172.21.15.104:0/3795247595 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99540a56c0 con 0x7f995c148950 2024-03-28T14:08:03.923 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:03.923+0000 7f995affd640 1 -- 172.21.15.104:0/3795247595 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f99540a5860 con 0x7f995c148950 2024-03-28T14:08:03.923 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:03.923+0000 7f995affd640 1 -- 172.21.15.104:0/3795247595 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99540a5c30 con 0x7f995c148950 2024-03-28T14:08:03.924 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:03.923+0000 7f99625c7640 1 -- 172.21.15.104:0/3795247595 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f995c148950 msgr2=0x7f995c152260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:03.924 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:03.923+0000 7f99625c7640 1 --2- 172.21.15.104:0/3795247595 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f995c148950 0x7f995c152260 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f9954009e50 tx=0x7f995401c9f0 comp rx=0 tx=0).stop 2024-03-28T14:08:03.924 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:03.923+0000 7f99625c7640 1 -- 172.21.15.104:0/3795247595 shutdown_connections 2024-03-28T14:08:03.924 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:03.923+0000 7f99625c7640 1 --2- 172.21.15.104:0/3795247595 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f995c148950 0x7f995c152260 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:03.924 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:03.923+0000 7f99625c7640 1 --2- 172.21.15.104:0/3795247595 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f995c0afd00 0x7f995c148410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:03.925 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:03.923+0000 7f99625c7640 1 -- 172.21.15.104:0/3795247595 >> 172.21.15.104:0/3795247595 conn(0x7f995c0b6080 msgr2=0x7f995c0b6460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T14:08:03.925 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:03.923+0000 7f99625c7640 1 -- 172.21.15.104:0/3795247595 shutdown_connections 2024-03-28T14:08:03.925 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:03.923+0000 7f99625c7640 1 -- 172.21.15.104:0/3795247595 wait complete. 2024-03-28T14:08:03.926 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:03.923+0000 7f99625c7640 1 Processor -- start 2024-03-28T14:08:03.927 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:03.923+0000 7f99625c7640 1 -- start start 2024-03-28T14:08:03.927 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:03.923+0000 7f99625c7640 1 --2- >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f995c02c460 0x7f995c099810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:03.927 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:03.923+0000 7f99625c7640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f995c0afd00 0x7f995c147630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:03.927 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:03.923+0000 7f99625c7640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f995c148950 0x7f995c09a1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:03.927 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:03.923+0000 7f99625c7640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f995c09eb70 con 0x7f995c148950 2024-03-28T14:08:03.927 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:03.923+0000 7f99625c7640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f995c09e9f0 con 0x7f995c0afd00 2024-03-28T14:08:03.927 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:03.923+0000 7f99625c7640 1 -- --> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] -- mon_getmap magic: 0 -- 0x7f995c154fc0 con 0x7f995c02c460 2024-03-28T14:08:03.927 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:03.923+0000 7f9960b64640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f995c148950 0x7f995c09a1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:03.927 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:03.923+0000 7f9960b64640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f995c148950 0x7f995c09a1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:42342/0 (socket says 172.21.15.104:42342) 2024-03-28T14:08:03.927 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:03.923+0000 7f9960b64640 1 -- 172.21.15.104:0/1685897928 learned_addr learned my addr 172.21.15.104:0/1685897928 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-03-28T14:08:03.928 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:03.923+0000 7f995bfff640 1 --2- 172.21.15.104:0/1685897928 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f995c02c460 0x7f995c099810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:03.928 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:03.923+0000 7f9960b64640 1 -- 172.21.15.104:0/1685897928 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f995c02c460 msgr2=0x7f995c099810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:03.928 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:03.923+0000 7f9960b64640 1 --2- 172.21.15.104:0/1685897928 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f995c02c460 0x7f995c099810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:03.928 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:03.923+0000 7f9960b64640 1 -- 172.21.15.104:0/1685897928 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f995c0afd00 msgr2=0x7f995c147630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:03.928 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:03.923+0000 7f995b7fe640 1 --2- 172.21.15.104:0/1685897928 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f995c0afd00 0x7f995c147630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:03.928 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:03.923+0000 7f9960b64640 1 --2- 172.21.15.104:0/1685897928 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f995c0afd00 0x7f995c147630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:03.928 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:03.923+0000 7f9960b64640 1 -- 172.21.15.104:0/1685897928 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f995c09a900 con 0x7f995c148950 2024-03-28T14:08:03.928 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:03.923+0000 7f9960b64640 1 --2- 172.21.15.104:0/1685897928 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f995c148950 0x7f995c09a1b0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f99540af040 tx=0x7f9954019790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:08:03.928 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:03.923+0000 7f9958ff9640 1 -- 172.21.15.104:0/1685897928 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99540bc020 con 0x7f995c148950 2024-03-28T14:08:03.928 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:03.923+0000 7f9958ff9640 1 -- 172.21.15.104:0/1685897928 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f99540b8670 con 0x7f995c148950 2024-03-28T14:08:03.928 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:03.923+0000 7f99625c7640 1 -- 172.21.15.104:0/1685897928 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f995c149ee0 con 0x7f995c148950 2024-03-28T14:08:03.928 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:03.923+0000 7f9958ff9640 1 -- 172.21.15.104:0/1685897928 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99540b8a60 con 0x7f995c148950 2024-03-28T14:08:03.929 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:03.923+0000 7f99625c7640 1 -- 172.21.15.104:0/1685897928 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f995c14afa0 con 0x7f995c148950 2024-03-28T14:08:03.930 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:03.927+0000 7f99625c7640 1 -- 172.21.15.104:0/1685897928 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f995c14b1d0 con 0x7f995c148950 2024-03-28T14:08:03.930 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:03.927+0000 7f9958ff9640 1 -- 172.21.15.104:0/1685897928 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) ==== 87803+0+0 (secure 0 0 0) 0x7f99540bf020 con 0x7f995c148950 2024-03-28T14:08:03.930 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:03.927+0000 7f9958ff9640 1 --2- 172.21.15.104:0/1685897928 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f9948067fc0 0x7f994806a480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:03.930 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:03.927+0000 7f9958ff9640 1 -- 172.21.15.104:0/1685897928 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f9954121920 con 0x7f995c148950 2024-03-28T14:08:03.930 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:03.927+0000 7f995bfff640 1 --2- 172.21.15.104:0/1685897928 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f9948067fc0 0x7f994806a480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:03.930 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:03.927+0000 7f995bfff640 1 --2- 172.21.15.104:0/1685897928 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f9948067fc0 0x7f994806a480 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f994c003d60 tx=0x7f994c009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:08:03.932 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:03.931+0000 7f9958ff9640 1 -- 172.21.15.104:0/1685897928 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179117 (secure 0 0 0) 0x7f99540c6030 con 0x7f995c148950 2024-03-28T14:08:04.087 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:04.083+0000 7f99625c7640 1 -- 172.21.15.104:0/1685897928 --> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f995c0687c0 con 0x7f9948067fc0 2024-03-28T14:08:04.104 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:04.103+0000 7f9958ff9640 1 -- 172.21.15.104:0/1685897928 <== mgr.4100 v2:172.21.15.171:6832/1522752412 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+251950 (secure 0 0 0) 0x7f995c0687c0 con 0x7f9948067fc0 2024-03-28T14:08:04.107 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-03-28T14:08:04.109 INFO:teuthology.orchestra.run.smithi104.stderr:dumped all 2024-03-28T14:08:04.110 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:04.107+0000 7f99625c7640 1 -- 172.21.15.104:0/1685897928 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f9948067fc0 msgr2=0x7f994806a480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:04.110 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:04.107+0000 7f99625c7640 1 --2- 172.21.15.104:0/1685897928 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f9948067fc0 0x7f994806a480 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f994c003d60 tx=0x7f994c009040 comp rx=0 tx=0).stop 2024-03-28T14:08:04.110 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:04.107+0000 7f99625c7640 1 -- 172.21.15.104:0/1685897928 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f995c148950 msgr2=0x7f995c09a1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:04.110 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:04.107+0000 7f99625c7640 1 --2- 172.21.15.104:0/1685897928 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f995c148950 0x7f995c09a1b0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f99540af040 tx=0x7f9954019790 comp rx=0 tx=0).stop 2024-03-28T14:08:04.110 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:04.107+0000 7f99625c7640 1 -- 172.21.15.104:0/1685897928 shutdown_connections 2024-03-28T14:08:04.110 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:04.107+0000 7f99625c7640 1 --2- 172.21.15.104:0/1685897928 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f9948067fc0 0x7f994806a480 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:04.110 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:04.107+0000 7f99625c7640 1 --2- 172.21.15.104:0/1685897928 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f995c148950 0x7f995c09a1b0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:04.111 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:04.107+0000 7f99625c7640 1 --2- 172.21.15.104:0/1685897928 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f995c0afd00 0x7f995c147630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:04.111 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:04.107+0000 7f99625c7640 1 --2- 172.21.15.104:0/1685897928 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f995c02c460 0x7f995c099810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:04.111 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:04.107+0000 7f99625c7640 1 -- 172.21.15.104:0/1685897928 >> 172.21.15.104:0/1685897928 conn(0x7f995c0b6080 msgr2=0x7f995c0b8830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T14:08:04.111 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:04.107+0000 7f99625c7640 1 -- 172.21.15.104:0/1685897928 shutdown_connections 2024-03-28T14:08:04.111 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:04.107+0000 7f99625c7640 1 -- 172.21.15.104:0/1685897928 wait complete. 2024-03-28T14:08:04.130 INFO:teuthology.orchestra.run.smithi104.stdout:{"pg_ready":true,"pg_map":{"version":23,"stamp":"2024-03-28T14:08:03.578345+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":146,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219108,"kb_used_data":2920,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755612,"statfs":{"total":773094113280,"available":772869746688,"internally_reserved":0,"allocated":2990080,"data_stored":1790396,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":221236785},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":45,"apply_latency_ms":45,"commit_latency_ns":45000000,"apply_latency_ns":45000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.934032"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-28T14:08:00.835047+0000","last_change":"2024-03-28T14:07:59.730372+0000","last_active":"2024-03-28T14:08:00.835047+0000","last_peered":"2024-03-28T14:08:00.835047+0000","last_clean":"2024-03-28T14:08:00.835047+0000","last_became_active":"2024-03-28T14:07:59.729644+0000","last_became_peered":"2024-03-28T14:07:59.729644+0000","last_unstale":"2024-03-28T14:08:00.835047+0000","last_undegraded":"2024-03-28T14:08:00.835047+0000","last_fullsized":"2024-03-28T14:08:00.835047+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_clean_scrub_stamp":"2024-03-28T14:07:58.641775+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:59:25.862480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-28T14:08:00.696778+0000","last_change":"2024-03-28T14:07:59.705187+0000","last_active":"2024-03-28T14:08:00.696778+0000","last_peered":"2024-03-28T14:08:00.696778+0000","last_clean":"2024-03-28T14:08:00.696778+0000","last_became_active":"2024-03-28T14:07:59.700386+0000","last_became_peered":"2024-03-28T14:07:59.700386+0000","last_unstale":"2024-03-28T14:08:00.696778+0000","last_undegraded":"2024-03-28T14:08:00.696778+0000","last_fullsized":"2024-03-28T14:08:00.696778+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_clean_scrub_stamp":"2024-03-28T14:07:58.641775+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T16:47:04.317583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-28T14:08:01.245347+0000","last_change":"2024-03-28T14:07:59.698753+0000","last_active":"2024-03-28T14:08:01.245347+0000","last_peered":"2024-03-28T14:08:01.245347+0000","last_clean":"2024-03-28T14:08:01.245347+0000","last_became_active":"2024-03-28T14:07:59.698467+0000","last_became_peered":"2024-03-28T14:07:59.698467+0000","last_unstale":"2024-03-28T14:08:01.245347+0000","last_undegraded":"2024-03-28T14:08:01.245347+0000","last_fullsized":"2024-03-28T14:08:01.245347+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_clean_scrub_stamp":"2024-03-28T14:07:58.641775+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:25:39.647896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-28T14:08:00.680934+0000","last_change":"2024-03-28T14:07:59.747773+0000","last_active":"2024-03-28T14:08:00.680934+0000","last_peered":"2024-03-28T14:08:00.680934+0000","last_clean":"2024-03-28T14:08:00.680934+0000","last_became_active":"2024-03-28T14:07:59.745640+0000","last_became_peered":"2024-03-28T14:07:59.745640+0000","last_unstale":"2024-03-28T14:08:00.680934+0000","last_undegraded":"2024-03-28T14:08:00.680934+0000","last_fullsized":"2024-03-28T14:08:00.680934+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_clean_scrub_stamp":"2024-03-28T14:07:58.641775+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:10:31.257059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-28T14:08:00.834115+0000","last_change":"2024-03-28T14:07:59.722923+0000","last_active":"2024-03-28T14:08:00.834115+0000","last_peered":"2024-03-28T14:08:00.834115+0000","last_clean":"2024-03-28T14:08:00.834115+0000","last_became_active":"2024-03-28T14:07:59.711460+0000","last_became_peered":"2024-03-28T14:07:59.711460+0000","last_unstale":"2024-03-28T14:08:00.834115+0000","last_undegraded":"2024-03-28T14:08:00.834115+0000","last_fullsized":"2024-03-28T14:08:00.834115+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_clean_scrub_stamp":"2024-03-28T14:07:58.641775+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T18:51:19.214005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-28T14:08:00.680087+0000","last_change":"2024-03-28T14:07:59.742906+0000","last_active":"2024-03-28T14:08:00.680087+0000","last_peered":"2024-03-28T14:08:00.680087+0000","last_clean":"2024-03-28T14:08:00.680087+0000","last_became_active":"2024-03-28T14:07:59.734609+0000","last_became_peered":"2024-03-28T14:07:59.734609+0000","last_unstale":"2024-03-28T14:08:00.680087+0000","last_undegraded":"2024-03-28T14:08:00.680087+0000","last_fullsized":"2024-03-28T14:08:00.680087+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_clean_scrub_stamp":"2024-03-28T14:07:58.641775+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:41:27.409736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-28T14:08:00.833773+0000","last_change":"2024-03-28T14:07:59.717632+0000","last_active":"2024-03-28T14:08:00.833773+0000","last_peered":"2024-03-28T14:08:00.833773+0000","last_clean":"2024-03-28T14:08:00.833773+0000","last_became_active":"2024-03-28T14:07:59.711894+0000","last_became_peered":"2024-03-28T14:07:59.711894+0000","last_unstale":"2024-03-28T14:08:00.833773+0000","last_undegraded":"2024-03-28T14:08:00.833773+0000","last_fullsized":"2024-03-28T14:08:00.833773+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_clean_scrub_stamp":"2024-03-28T14:07:58.641775+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T16:47:04.317583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-28T14:08:00.680228+0000","last_change":"2024-03-28T14:07:59.740843+0000","last_active":"2024-03-28T14:08:00.680228+0000","last_peered":"2024-03-28T14:08:00.680228+0000","last_clean":"2024-03-28T14:08:00.680228+0000","last_became_active":"2024-03-28T14:07:59.740606+0000","last_became_peered":"2024-03-28T14:07:59.740606+0000","last_unstale":"2024-03-28T14:08:00.680228+0000","last_undegraded":"2024-03-28T14:08:00.680228+0000","last_fullsized":"2024-03-28T14:08:00.680228+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_clean_scrub_stamp":"2024-03-28T14:07:58.641775+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T14:20:33.555524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-28T14:08:00.835049+0000","last_change":"2024-03-28T14:07:59.745208+0000","last_active":"2024-03-28T14:08:00.835049+0000","last_peered":"2024-03-28T14:08:00.835049+0000","last_clean":"2024-03-28T14:08:00.835049+0000","last_became_active":"2024-03-28T14:07:59.744704+0000","last_became_peered":"2024-03-28T14:07:59.744704+0000","last_unstale":"2024-03-28T14:08:00.835049+0000","last_undegraded":"2024-03-28T14:08:00.835049+0000","last_fullsized":"2024-03-28T14:08:00.835049+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_clean_scrub_stamp":"2024-03-28T14:07:58.641775+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:53:55.059948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-28T14:08:00.703336+0000","last_change":"2024-03-28T14:07:59.744713+0000","last_active":"2024-03-28T14:08:00.703336+0000","last_peered":"2024-03-28T14:08:00.703336+0000","last_clean":"2024-03-28T14:08:00.703336+0000","last_became_active":"2024-03-28T14:07:59.744134+0000","last_became_peered":"2024-03-28T14:07:59.744134+0000","last_unstale":"2024-03-28T14:08:00.703336+0000","last_undegraded":"2024-03-28T14:08:00.703336+0000","last_fullsized":"2024-03-28T14:08:00.703336+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_clean_scrub_stamp":"2024-03-28T14:07:58.641775+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:25:39.647896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-28T14:08:00.703286+0000","last_change":"2024-03-28T14:07:59.745769+0000","last_active":"2024-03-28T14:08:00.703286+0000","last_peered":"2024-03-28T14:08:00.703286+0000","last_clean":"2024-03-28T14:08:00.703286+0000","last_became_active":"2024-03-28T14:07:59.745151+0000","last_became_peered":"2024-03-28T14:07:59.745151+0000","last_unstale":"2024-03-28T14:08:00.703286+0000","last_undegraded":"2024-03-28T14:08:00.703286+0000","last_fullsized":"2024-03-28T14:08:00.703286+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_clean_scrub_stamp":"2024-03-28T14:07:58.641775+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:37:29.402787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-28T14:08:01.244022+0000","last_change":"2024-03-28T14:07:59.702599+0000","last_active":"2024-03-28T14:08:01.244022+0000","last_peered":"2024-03-28T14:08:01.244022+0000","last_clean":"2024-03-28T14:08:01.244022+0000","last_became_active":"2024-03-28T14:07:59.701506+0000","last_became_peered":"2024-03-28T14:07:59.701506+0000","last_unstale":"2024-03-28T14:08:01.244022+0000","last_undegraded":"2024-03-28T14:08:01.244022+0000","last_fullsized":"2024-03-28T14:08:01.244022+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_clean_scrub_stamp":"2024-03-28T14:07:58.641775+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T15:40:59.112394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-28T14:08:00.680979+0000","last_change":"2024-03-28T14:07:59.704862+0000","last_active":"2024-03-28T14:08:00.680979+0000","last_peered":"2024-03-28T14:08:00.680979+0000","last_clean":"2024-03-28T14:08:00.680979+0000","last_became_active":"2024-03-28T14:07:59.704461+0000","last_became_peered":"2024-03-28T14:07:59.704461+0000","last_unstale":"2024-03-28T14:08:00.680979+0000","last_undegraded":"2024-03-28T14:08:00.680979+0000","last_fullsized":"2024-03-28T14:08:00.680979+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_clean_scrub_stamp":"2024-03-28T14:07:58.641775+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:37:29.402787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-28T14:08:01.243740+0000","last_change":"2024-03-28T14:07:59.707399+0000","last_active":"2024-03-28T14:08:01.243740+0000","last_peered":"2024-03-28T14:08:01.243740+0000","last_clean":"2024-03-28T14:08:01.243740+0000","last_became_active":"2024-03-28T14:07:59.707057+0000","last_became_peered":"2024-03-28T14:07:59.707057+0000","last_unstale":"2024-03-28T14:08:01.243740+0000","last_undegraded":"2024-03-28T14:08:01.243740+0000","last_fullsized":"2024-03-28T14:08:01.243740+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_clean_scrub_stamp":"2024-03-28T14:07:58.641775+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:54:39.259029+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-28T14:08:00.696269+0000","last_change":"2024-03-28T14:07:59.684910+0000","last_active":"2024-03-28T14:08:00.696269+0000","last_peered":"2024-03-28T14:08:00.696269+0000","last_clean":"2024-03-28T14:08:00.696269+0000","last_became_active":"2024-03-28T14:07:59.684112+0000","last_became_peered":"2024-03-28T14:07:59.684112+0000","last_unstale":"2024-03-28T14:08:00.696269+0000","last_undegraded":"2024-03-28T14:08:00.696269+0000","last_fullsized":"2024-03-28T14:08:00.696269+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_clean_scrub_stamp":"2024-03-28T14:07:58.641775+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:37:29.402787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-28T14:08:00.680001+0000","last_change":"2024-03-28T14:07:59.740138+0000","last_active":"2024-03-28T14:08:00.680001+0000","last_peered":"2024-03-28T14:08:00.680001+0000","last_clean":"2024-03-28T14:08:00.680001+0000","last_became_active":"2024-03-28T14:07:59.739320+0000","last_became_peered":"2024-03-28T14:07:59.739320+0000","last_unstale":"2024-03-28T14:08:00.680001+0000","last_undegraded":"2024-03-28T14:08:00.680001+0000","last_fullsized":"2024-03-28T14:08:00.680001+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_clean_scrub_stamp":"2024-03-28T14:07:58.641775+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:25:39.647896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-28T14:08:00.681671+0000","last_change":"2024-03-28T14:07:59.744839+0000","last_active":"2024-03-28T14:08:00.681671+0000","last_peered":"2024-03-28T14:08:00.681671+0000","last_clean":"2024-03-28T14:08:00.681671+0000","last_became_active":"2024-03-28T14:07:59.739555+0000","last_became_peered":"2024-03-28T14:07:59.739555+0000","last_unstale":"2024-03-28T14:08:00.681671+0000","last_undegraded":"2024-03-28T14:08:00.681671+0000","last_fullsized":"2024-03-28T14:08:00.681671+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_clean_scrub_stamp":"2024-03-28T14:07:58.641775+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:53:55.059948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-28T14:08:01.242371+0000","last_change":"2024-03-28T14:07:59.707740+0000","last_active":"2024-03-28T14:08:01.242371+0000","last_peered":"2024-03-28T14:08:01.242371+0000","last_clean":"2024-03-28T14:08:01.242371+0000","last_became_active":"2024-03-28T14:07:59.702876+0000","last_became_peered":"2024-03-28T14:07:59.702876+0000","last_unstale":"2024-03-28T14:08:01.242371+0000","last_undegraded":"2024-03-28T14:08:01.242371+0000","last_fullsized":"2024-03-28T14:08:01.242371+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_clean_scrub_stamp":"2024-03-28T14:07:58.641775+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T15:40:01.667745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-28T14:08:00.834771+0000","last_change":"2024-03-28T14:07:59.723973+0000","last_active":"2024-03-28T14:08:00.834771+0000","last_peered":"2024-03-28T14:08:00.834771+0000","last_clean":"2024-03-28T14:08:00.834771+0000","last_became_active":"2024-03-28T14:07:59.712162+0000","last_became_peered":"2024-03-28T14:07:59.712162+0000","last_unstale":"2024-03-28T14:08:00.834771+0000","last_undegraded":"2024-03-28T14:08:00.834771+0000","last_fullsized":"2024-03-28T14:08:00.834771+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_clean_scrub_stamp":"2024-03-28T14:07:58.641775+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:51:23.322586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-28T14:08:00.703000+0000","last_change":"2024-03-28T14:07:59.704917+0000","last_active":"2024-03-28T14:08:00.703000+0000","last_peered":"2024-03-28T14:08:00.703000+0000","last_clean":"2024-03-28T14:08:00.703000+0000","last_became_active":"2024-03-28T14:07:59.702369+0000","last_became_peered":"2024-03-28T14:07:59.702369+0000","last_unstale":"2024-03-28T14:08:00.703000+0000","last_undegraded":"2024-03-28T14:08:00.703000+0000","last_fullsized":"2024-03-28T14:08:00.703000+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_clean_scrub_stamp":"2024-03-28T14:07:58.641775+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:24:30.197596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-28T14:08:01.244077+0000","last_change":"2024-03-28T14:07:59.727121+0000","last_active":"2024-03-28T14:08:01.244077+0000","last_peered":"2024-03-28T14:08:01.244077+0000","last_clean":"2024-03-28T14:08:01.244077+0000","last_became_active":"2024-03-28T14:07:59.726783+0000","last_became_peered":"2024-03-28T14:07:59.726783+0000","last_unstale":"2024-03-28T14:08:01.244077+0000","last_undegraded":"2024-03-28T14:08:01.244077+0000","last_fullsized":"2024-03-28T14:08:01.244077+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_clean_scrub_stamp":"2024-03-28T14:07:58.641775+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:10:27.248645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-28T14:08:00.680533+0000","last_change":"2024-03-28T14:07:59.742729+0000","last_active":"2024-03-28T14:08:00.680533+0000","last_peered":"2024-03-28T14:08:00.680533+0000","last_clean":"2024-03-28T14:08:00.680533+0000","last_became_active":"2024-03-28T14:07:59.739727+0000","last_became_peered":"2024-03-28T14:07:59.739727+0000","last_unstale":"2024-03-28T14:08:00.680533+0000","last_undegraded":"2024-03-28T14:08:00.680533+0000","last_fullsized":"2024-03-28T14:08:00.680533+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_clean_scrub_stamp":"2024-03-28T14:07:58.641775+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T17:46:06.253111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-28T14:08:00.681600+0000","last_change":"2024-03-28T14:07:59.704724+0000","last_active":"2024-03-28T14:08:00.681600+0000","last_peered":"2024-03-28T14:08:00.681600+0000","last_clean":"2024-03-28T14:08:00.681600+0000","last_became_active":"2024-03-28T14:07:59.704398+0000","last_became_peered":"2024-03-28T14:07:59.704398+0000","last_unstale":"2024-03-28T14:08:00.681600+0000","last_undegraded":"2024-03-28T14:08:00.681600+0000","last_fullsized":"2024-03-28T14:08:00.681600+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_clean_scrub_stamp":"2024-03-28T14:07:58.641775+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T18:51:19.214005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-28T14:08:00.680302+0000","last_change":"2024-03-28T14:07:59.739100+0000","last_active":"2024-03-28T14:08:00.680302+0000","last_peered":"2024-03-28T14:08:00.680302+0000","last_clean":"2024-03-28T14:08:00.680302+0000","last_became_active":"2024-03-28T14:07:59.733884+0000","last_became_peered":"2024-03-28T14:07:59.733884+0000","last_unstale":"2024-03-28T14:08:00.680302+0000","last_undegraded":"2024-03-28T14:08:00.680302+0000","last_fullsized":"2024-03-28T14:08:00.680302+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_clean_scrub_stamp":"2024-03-28T14:07:58.641775+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T16:47:04.317583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-28T14:08:00.681010+0000","last_change":"2024-03-28T14:07:59.716106+0000","last_active":"2024-03-28T14:08:00.681010+0000","last_peered":"2024-03-28T14:08:00.681010+0000","last_clean":"2024-03-28T14:08:00.681010+0000","last_became_active":"2024-03-28T14:07:59.715747+0000","last_became_peered":"2024-03-28T14:07:59.715747+0000","last_unstale":"2024-03-28T14:08:00.681010+0000","last_undegraded":"2024-03-28T14:08:00.681010+0000","last_fullsized":"2024-03-28T14:08:00.681010+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_clean_scrub_stamp":"2024-03-28T14:07:58.641775+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:25:39.647896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-28T14:08:00.703528+0000","last_change":"2024-03-28T14:07:59.710509+0000","last_active":"2024-03-28T14:08:00.703528+0000","last_peered":"2024-03-28T14:08:00.703528+0000","last_clean":"2024-03-28T14:08:00.703528+0000","last_became_active":"2024-03-28T14:07:59.709767+0000","last_became_peered":"2024-03-28T14:07:59.709767+0000","last_unstale":"2024-03-28T14:08:00.703528+0000","last_undegraded":"2024-03-28T14:08:00.703528+0000","last_fullsized":"2024-03-28T14:08:00.703528+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_clean_scrub_stamp":"2024-03-28T14:07:58.641775+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:41:27.409736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-28T14:08:00.696547+0000","last_change":"2024-03-28T14:07:59.704832+0000","last_active":"2024-03-28T14:08:00.696547+0000","last_peered":"2024-03-28T14:08:00.696547+0000","last_clean":"2024-03-28T14:08:00.696547+0000","last_became_active":"2024-03-28T14:07:59.704546+0000","last_became_peered":"2024-03-28T14:07:59.704546+0000","last_unstale":"2024-03-28T14:08:00.696547+0000","last_undegraded":"2024-03-28T14:08:00.696547+0000","last_fullsized":"2024-03-28T14:08:00.696547+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_clean_scrub_stamp":"2024-03-28T14:07:58.641775+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:41:27.409736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-28T14:08:00.681706+0000","last_change":"2024-03-28T14:07:59.702155+0000","last_active":"2024-03-28T14:08:00.681706+0000","last_peered":"2024-03-28T14:08:00.681706+0000","last_clean":"2024-03-28T14:08:00.681706+0000","last_became_active":"2024-03-28T14:07:59.701799+0000","last_became_peered":"2024-03-28T14:07:59.701799+0000","last_unstale":"2024-03-28T14:08:00.681706+0000","last_undegraded":"2024-03-28T14:08:00.681706+0000","last_fullsized":"2024-03-28T14:08:00.681706+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_clean_scrub_stamp":"2024-03-28T14:07:58.641775+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:10:31.257059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-28T14:08:01.244122+0000","last_change":"2024-03-28T14:07:59.706889+0000","last_active":"2024-03-28T14:08:01.244122+0000","last_peered":"2024-03-28T14:08:01.244122+0000","last_clean":"2024-03-28T14:08:01.244122+0000","last_became_active":"2024-03-28T14:07:59.706631+0000","last_became_peered":"2024-03-28T14:07:59.706631+0000","last_unstale":"2024-03-28T14:08:01.244122+0000","last_undegraded":"2024-03-28T14:08:01.244122+0000","last_fullsized":"2024-03-28T14:08:01.244122+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_clean_scrub_stamp":"2024-03-28T14:07:58.641775+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:57:11.908191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-28T14:08:00.680602+0000","last_change":"2024-03-28T14:07:59.722801+0000","last_active":"2024-03-28T14:08:00.680602+0000","last_peered":"2024-03-28T14:08:00.680602+0000","last_clean":"2024-03-28T14:08:00.680602+0000","last_became_active":"2024-03-28T14:07:59.722132+0000","last_became_peered":"2024-03-28T14:07:59.722132+0000","last_unstale":"2024-03-28T14:08:00.680602+0000","last_undegraded":"2024-03-28T14:08:00.680602+0000","last_fullsized":"2024-03-28T14:08:00.680602+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_clean_scrub_stamp":"2024-03-28T14:07:58.641775+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T18:51:19.214005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-28T14:08:01.242238+0000","last_change":"2024-03-28T14:07:59.740448+0000","last_active":"2024-03-28T14:08:01.242238+0000","last_peered":"2024-03-28T14:08:01.242238+0000","last_clean":"2024-03-28T14:08:01.242238+0000","last_became_active":"2024-03-28T14:07:59.740121+0000","last_became_peered":"2024-03-28T14:07:59.740121+0000","last_unstale":"2024-03-28T14:08:01.242238+0000","last_undegraded":"2024-03-28T14:08:01.242238+0000","last_fullsized":"2024-03-28T14:08:01.242238+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_clean_scrub_stamp":"2024-03-28T14:07:58.641775+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T15:40:59.112394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-28T14:08:00.702471+0000","last_change":"2024-03-28T14:07:59.704155+0000","last_active":"2024-03-28T14:08:00.702471+0000","last_peered":"2024-03-28T14:08:00.702471+0000","last_clean":"2024-03-28T14:08:00.702471+0000","last_became_active":"2024-03-28T14:07:59.702173+0000","last_became_peered":"2024-03-28T14:07:59.702173+0000","last_unstale":"2024-03-28T14:08:00.702471+0000","last_undegraded":"2024-03-28T14:08:00.702471+0000","last_fullsized":"2024-03-28T14:08:00.702471+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_clean_scrub_stamp":"2024-03-28T14:07:58.641775+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T14:20:33.555524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-28T14:08:01.243508+0000","last_change":"2024-03-28T14:07:59.682929+0000","last_active":"2024-03-28T14:08:01.243508+0000","last_peered":"2024-03-28T14:08:01.243508+0000","last_clean":"2024-03-28T14:08:01.243508+0000","last_became_active":"2024-03-28T14:07:59.682238+0000","last_became_peered":"2024-03-28T14:07:59.682238+0000","last_unstale":"2024-03-28T14:08:01.243508+0000","last_undegraded":"2024-03-28T14:08:01.243508+0000","last_fullsized":"2024-03-28T14:08:01.243508+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_clean_scrub_stamp":"2024-03-28T14:07:58.641775+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T17:35:01.163718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"20'1","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-28T14:08:00.681702+0000","last_change":"2024-03-28T14:07:58.672879+0000","last_active":"2024-03-28T14:08:00.681702+0000","last_peered":"2024-03-28T14:08:00.681702+0000","last_clean":"2024-03-28T14:08:00.681702+0000","last_became_active":"2024-03-28T14:07:58.672646+0000","last_became_peered":"2024-03-28T14:07:58.672646+0000","last_unstale":"2024-03-28T14:08:00.681702+0000","last_undegraded":"2024-03-28T14:08:00.681702+0000","last_fullsized":"2024-03-28T14:08:00.681702+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:07:57.631994+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:07:57.631994+0000","last_clean_scrub_stamp":"2024-03-28T14:07:57.631994+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:35:03.722392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-28T14:08:01.244389+0000","last_change":"2024-03-28T14:07:59.715972+0000","last_active":"2024-03-28T14:08:01.244389+0000","last_peered":"2024-03-28T14:08:01.244389+0000","last_clean":"2024-03-28T14:08:01.244389+0000","last_became_active":"2024-03-28T14:07:59.715621+0000","last_became_peered":"2024-03-28T14:07:59.715621+0000","last_unstale":"2024-03-28T14:08:01.244389+0000","last_undegraded":"2024-03-28T14:08:01.244389+0000","last_fullsized":"2024-03-28T14:08:01.244389+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_clean_scrub_stamp":"2024-03-28T14:07:58.641775+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:24:30.197596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1a","version":"20'1","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-28T14:08:00.702735+0000","last_change":"2024-03-28T14:07:58.688093+0000","last_active":"2024-03-28T14:08:00.702735+0000","last_peered":"2024-03-28T14:08:00.702735+0000","last_clean":"2024-03-28T14:08:00.702735+0000","last_became_active":"2024-03-28T14:07:58.687279+0000","last_became_peered":"2024-03-28T14:07:58.687279+0000","last_unstale":"2024-03-28T14:08:00.702735+0000","last_undegraded":"2024-03-28T14:08:00.702735+0000","last_fullsized":"2024-03-28T14:08:00.702735+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:07:57.631994+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:07:57.631994+0000","last_clean_scrub_stamp":"2024-03-28T14:07:57.631994+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T18:00:55.131543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-28T14:08:00.680142+0000","last_change":"2024-03-28T14:07:58.678668+0000","last_active":"2024-03-28T14:08:00.680142+0000","last_peered":"2024-03-28T14:08:00.680142+0000","last_clean":"2024-03-28T14:08:00.680142+0000","last_became_active":"2024-03-28T14:07:58.678292+0000","last_became_peered":"2024-03-28T14:07:58.678292+0000","last_unstale":"2024-03-28T14:08:00.680142+0000","last_undegraded":"2024-03-28T14:08:00.680142+0000","last_fullsized":"2024-03-28T14:08:00.680142+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:07:57.631994+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:07:57.631994+0000","last_clean_scrub_stamp":"2024-03-28T14:07:57.631994+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T18:00:55.131543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-28T14:08:00.680364+0000","last_change":"2024-03-28T14:07:59.703432+0000","last_active":"2024-03-28T14:08:00.680364+0000","last_peered":"2024-03-28T14:08:00.680364+0000","last_clean":"2024-03-28T14:08:00.680364+0000","last_became_active":"2024-03-28T14:07:59.702896+0000","last_became_peered":"2024-03-28T14:07:59.702896+0000","last_unstale":"2024-03-28T14:08:00.680364+0000","last_undegraded":"2024-03-28T14:08:00.680364+0000","last_fullsized":"2024-03-28T14:08:00.680364+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_clean_scrub_stamp":"2024-03-28T14:07:58.641775+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:10:27.248645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-28T14:08:00.680262+0000","last_change":"2024-03-28T14:07:58.703631+0000","last_active":"2024-03-28T14:08:00.680262+0000","last_peered":"2024-03-28T14:08:00.680262+0000","last_clean":"2024-03-28T14:08:00.680262+0000","last_became_active":"2024-03-28T14:07:58.702897+0000","last_became_peered":"2024-03-28T14:07:58.702897+0000","last_unstale":"2024-03-28T14:08:00.680262+0000","last_undegraded":"2024-03-28T14:08:00.680262+0000","last_fullsized":"2024-03-28T14:08:00.680262+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:07:57.631994+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:07:57.631994+0000","last_clean_scrub_stamp":"2024-03-28T14:07:57.631994+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:35:03.722392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-28T14:08:00.696158+0000","last_change":"2024-03-28T14:07:59.706310+0000","last_active":"2024-03-28T14:08:00.696158+0000","last_peered":"2024-03-28T14:08:00.696158+0000","last_clean":"2024-03-28T14:08:00.696158+0000","last_became_active":"2024-03-28T14:07:59.705967+0000","last_became_peered":"2024-03-28T14:07:59.705967+0000","last_unstale":"2024-03-28T14:08:00.696158+0000","last_undegraded":"2024-03-28T14:08:00.696158+0000","last_fullsized":"2024-03-28T14:08:00.696158+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_clean_scrub_stamp":"2024-03-28T14:07:58.641775+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:24:30.197596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-28T14:08:00.703408+0000","last_change":"2024-03-28T14:07:58.702042+0000","last_active":"2024-03-28T14:08:00.703408+0000","last_peered":"2024-03-28T14:08:00.703408+0000","last_clean":"2024-03-28T14:08:00.703408+0000","last_became_active":"2024-03-28T14:07:58.701395+0000","last_became_peered":"2024-03-28T14:07:58.701395+0000","last_unstale":"2024-03-28T14:08:00.703408+0000","last_undegraded":"2024-03-28T14:08:00.703408+0000","last_fullsized":"2024-03-28T14:08:00.703408+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:07:57.631994+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:07:57.631994+0000","last_clean_scrub_stamp":"2024-03-28T14:07:57.631994+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T17:35:00.153938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-28T14:08:00.681271+0000","last_change":"2024-03-28T14:07:59.740904+0000","last_active":"2024-03-28T14:08:00.681271+0000","last_peered":"2024-03-28T14:08:00.681271+0000","last_clean":"2024-03-28T14:08:00.681271+0000","last_became_active":"2024-03-28T14:07:59.740209+0000","last_became_peered":"2024-03-28T14:07:59.740209+0000","last_unstale":"2024-03-28T14:08:00.681271+0000","last_undegraded":"2024-03-28T14:08:00.681271+0000","last_fullsized":"2024-03-28T14:08:00.681271+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_clean_scrub_stamp":"2024-03-28T14:07:58.641775+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:02:48.017536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-28T14:08:00.681223+0000","last_change":"2024-03-28T14:07:58.699895+0000","last_active":"2024-03-28T14:08:00.681223+0000","last_peered":"2024-03-28T14:08:00.681223+0000","last_clean":"2024-03-28T14:08:00.681223+0000","last_became_active":"2024-03-28T14:07:58.698823+0000","last_became_peered":"2024-03-28T14:07:58.698823+0000","last_unstale":"2024-03-28T14:08:00.681223+0000","last_undegraded":"2024-03-28T14:08:00.681223+0000","last_fullsized":"2024-03-28T14:08:00.681223+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:07:57.631994+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:07:57.631994+0000","last_clean_scrub_stamp":"2024-03-28T14:07:57.631994+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:09:38.264121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-28T14:08:01.245669+0000","last_change":"2024-03-28T14:07:59.744137+0000","last_active":"2024-03-28T14:08:01.245669+0000","last_peered":"2024-03-28T14:08:01.245669+0000","last_clean":"2024-03-28T14:08:01.245669+0000","last_became_active":"2024-03-28T14:07:59.743854+0000","last_became_peered":"2024-03-28T14:07:59.743854+0000","last_unstale":"2024-03-28T14:08:01.245669+0000","last_undegraded":"2024-03-28T14:08:01.245669+0000","last_fullsized":"2024-03-28T14:08:01.245669+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_clean_scrub_stamp":"2024-03-28T14:07:58.641775+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:41:27.409736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"20'3","reported_seq":17,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-28T14:08:00.696618+0000","last_change":"2024-03-28T14:07:58.703242+0000","last_active":"2024-03-28T14:08:00.696618+0000","last_peered":"2024-03-28T14:08:00.696618+0000","last_clean":"2024-03-28T14:08:00.696618+0000","last_became_active":"2024-03-28T14:07:58.702970+0000","last_became_peered":"2024-03-28T14:07:58.702970+0000","last_unstale":"2024-03-28T14:08:00.696618+0000","last_undegraded":"2024-03-28T14:08:00.696618+0000","last_fullsized":"2024-03-28T14:08:00.696618+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:07:57.631994+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:07:57.631994+0000","last_clean_scrub_stamp":"2024-03-28T14:07:57.631994+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T15:40:58.102613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-28T14:08:01.244046+0000","last_change":"2024-03-28T14:07:59.708551+0000","last_active":"2024-03-28T14:08:01.244046+0000","last_peered":"2024-03-28T14:08:01.244046+0000","last_clean":"2024-03-28T14:08:01.244046+0000","last_became_active":"2024-03-28T14:07:59.706665+0000","last_became_peered":"2024-03-28T14:07:59.706665+0000","last_unstale":"2024-03-28T14:08:01.244046+0000","last_undegraded":"2024-03-28T14:08:01.244046+0000","last_fullsized":"2024-03-28T14:08:01.244046+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_clean_scrub_stamp":"2024-03-28T14:07:58.641775+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T15:40:01.667745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.14","version":"20'6","reported_seq":20,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-28T14:08:00.702870+0000","last_change":"2024-03-28T14:07:58.680343+0000","last_active":"2024-03-28T14:08:00.702870+0000","last_peered":"2024-03-28T14:08:00.702870+0000","last_clean":"2024-03-28T14:08:00.702870+0000","last_became_active":"2024-03-28T14:07:58.679621+0000","last_became_peered":"2024-03-28T14:07:58.679621+0000","last_unstale":"2024-03-28T14:08:00.702870+0000","last_undegraded":"2024-03-28T14:08:00.702870+0000","last_fullsized":"2024-03-28T14:08:00.702870+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:07:57.631994+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:07:57.631994+0000","last_clean_scrub_stamp":"2024-03-28T14:07:57.631994+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:30:19.088624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-28T14:08:00.834351+0000","last_change":"2024-03-28T14:07:59.714064+0000","last_active":"2024-03-28T14:08:00.834351+0000","last_peered":"2024-03-28T14:08:00.834351+0000","last_clean":"2024-03-28T14:08:00.834351+0000","last_became_active":"2024-03-28T14:07:59.710619+0000","last_became_peered":"2024-03-28T14:07:59.710619+0000","last_unstale":"2024-03-28T14:08:00.834351+0000","last_undegraded":"2024-03-28T14:08:00.834351+0000","last_fullsized":"2024-03-28T14:08:00.834351+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_clean_scrub_stamp":"2024-03-28T14:07:58.641775+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:34:47.572488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"20'2","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-28T14:08:01.244515+0000","last_change":"2024-03-28T14:07:58.684715+0000","last_active":"2024-03-28T14:08:01.244515+0000","last_peered":"2024-03-28T14:08:01.244515+0000","last_clean":"2024-03-28T14:08:01.244515+0000","last_became_active":"2024-03-28T14:07:58.684314+0000","last_became_peered":"2024-03-28T14:07:58.684314+0000","last_unstale":"2024-03-28T14:08:01.244515+0000","last_undegraded":"2024-03-28T14:08:01.244515+0000","last_fullsized":"2024-03-28T14:08:01.244515+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:07:57.631994+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:07:57.631994+0000","last_clean_scrub_stamp":"2024-03-28T14:07:57.631994+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T17:34:09.030115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-28T14:08:01.243877+0000","last_change":"2024-03-28T14:07:59.730071+0000","last_active":"2024-03-28T14:08:01.243877+0000","last_peered":"2024-03-28T14:08:01.243877+0000","last_clean":"2024-03-28T14:08:01.243877+0000","last_became_active":"2024-03-28T14:07:59.729794+0000","last_became_peered":"2024-03-28T14:07:59.729794+0000","last_unstale":"2024-03-28T14:08:01.243877+0000","last_undegraded":"2024-03-28T14:08:01.243877+0000","last_fullsized":"2024-03-28T14:08:01.243877+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_clean_scrub_stamp":"2024-03-28T14:07:58.641775+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:38:43.102577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-28T14:08:00.680825+0000","last_change":"2024-03-28T14:07:58.704245+0000","last_active":"2024-03-28T14:08:00.680825+0000","last_peered":"2024-03-28T14:08:00.680825+0000","last_clean":"2024-03-28T14:08:00.680825+0000","last_became_active":"2024-03-28T14:07:58.703603+0000","last_became_peered":"2024-03-28T14:07:58.703603+0000","last_unstale":"2024-03-28T14:08:00.680825+0000","last_undegraded":"2024-03-28T14:08:00.680825+0000","last_fullsized":"2024-03-28T14:08:00.680825+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:07:57.631994+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:07:57.631994+0000","last_clean_scrub_stamp":"2024-03-28T14:07:57.631994+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T17:35:00.153938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-28T14:08:00.834163+0000","last_change":"2024-03-28T14:07:59.718822+0000","last_active":"2024-03-28T14:08:00.834163+0000","last_peered":"2024-03-28T14:08:00.834163+0000","last_clean":"2024-03-28T14:08:00.834163+0000","last_became_active":"2024-03-28T14:07:59.718058+0000","last_became_peered":"2024-03-28T14:07:59.718058+0000","last_unstale":"2024-03-28T14:08:00.834163+0000","last_undegraded":"2024-03-28T14:08:00.834163+0000","last_fullsized":"2024-03-28T14:08:00.834163+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_clean_scrub_stamp":"2024-03-28T14:07:58.641775+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:10:31.257059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-28T14:08:00.702420+0000","last_change":"2024-03-28T14:07:58.689870+0000","last_active":"2024-03-28T14:08:00.702420+0000","last_peered":"2024-03-28T14:08:00.702420+0000","last_clean":"2024-03-28T14:08:00.702420+0000","last_became_active":"2024-03-28T14:07:58.689121+0000","last_became_peered":"2024-03-28T14:07:58.689121+0000","last_unstale":"2024-03-28T14:08:00.702420+0000","last_undegraded":"2024-03-28T14:08:00.702420+0000","last_fullsized":"2024-03-28T14:08:00.702420+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:07:57.631994+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:07:57.631994+0000","last_clean_scrub_stamp":"2024-03-28T14:07:57.631994+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:35:03.722392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-28T14:08:00.702531+0000","last_change":"2024-03-28T14:07:59.712485+0000","last_active":"2024-03-28T14:08:00.702531+0000","last_peered":"2024-03-28T14:08:00.702531+0000","last_clean":"2024-03-28T14:08:00.702531+0000","last_became_active":"2024-03-28T14:07:59.704056+0000","last_became_peered":"2024-03-28T14:07:59.704056+0000","last_unstale":"2024-03-28T14:08:00.702531+0000","last_undegraded":"2024-03-28T14:08:00.702531+0000","last_fullsized":"2024-03-28T14:08:00.702531+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_clean_scrub_stamp":"2024-03-28T14:07:58.641775+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:10:27.248645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"20'8","reported_seq":22,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-28T14:08:01.242042+0000","last_change":"2024-03-28T14:07:58.667252+0000","last_active":"2024-03-28T14:08:01.242042+0000","last_peered":"2024-03-28T14:08:01.242042+0000","last_clean":"2024-03-28T14:08:01.242042+0000","last_became_active":"2024-03-28T14:07:58.666837+0000","last_became_peered":"2024-03-28T14:07:58.666837+0000","last_unstale":"2024-03-28T14:08:01.242042+0000","last_undegraded":"2024-03-28T14:08:01.242042+0000","last_fullsized":"2024-03-28T14:08:01.242042+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:07:57.631994+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:07:57.631994+0000","last_clean_scrub_stamp":"2024-03-28T14:07:57.631994+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:30:19.088624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-28T14:08:00.833905+0000","last_change":"2024-03-28T14:07:59.719433+0000","last_active":"2024-03-28T14:08:00.833905+0000","last_peered":"2024-03-28T14:08:00.833905+0000","last_clean":"2024-03-28T14:08:00.833905+0000","last_became_active":"2024-03-28T14:07:59.718644+0000","last_became_peered":"2024-03-28T14:07:59.718644+0000","last_unstale":"2024-03-28T14:08:00.833905+0000","last_undegraded":"2024-03-28T14:08:00.833905+0000","last_fullsized":"2024-03-28T14:08:00.833905+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_clean_scrub_stamp":"2024-03-28T14:07:58.641775+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T17:46:06.253111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"20'1","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-28T14:08:01.243993+0000","last_change":"2024-03-28T14:07:58.674598+0000","last_active":"2024-03-28T14:08:01.243993+0000","last_peered":"2024-03-28T14:08:01.243993+0000","last_clean":"2024-03-28T14:08:01.243993+0000","last_became_active":"2024-03-28T14:07:58.673894+0000","last_became_peered":"2024-03-28T14:07:58.673894+0000","last_unstale":"2024-03-28T14:08:01.243993+0000","last_undegraded":"2024-03-28T14:08:01.243993+0000","last_fullsized":"2024-03-28T14:08:01.243993+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:07:57.631994+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:07:57.631994+0000","last_clean_scrub_stamp":"2024-03-28T14:07:57.631994+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:32:35.921987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-28T14:08:00.834711+0000","last_change":"2024-03-28T14:07:59.725643+0000","last_active":"2024-03-28T14:08:00.834711+0000","last_peered":"2024-03-28T14:08:00.834711+0000","last_clean":"2024-03-28T14:08:00.834711+0000","last_became_active":"2024-03-28T14:07:59.725019+0000","last_became_peered":"2024-03-28T14:07:59.725019+0000","last_unstale":"2024-03-28T14:08:00.834711+0000","last_undegraded":"2024-03-28T14:08:00.834711+0000","last_fullsized":"2024-03-28T14:08:00.834711+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_clean_scrub_stamp":"2024-03-28T14:07:58.641775+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:02:48.017536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"20'2","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-28T14:08:01.242483+0000","last_change":"2024-03-28T14:07:58.663444+0000","last_active":"2024-03-28T14:08:01.242483+0000","last_peered":"2024-03-28T14:08:01.242483+0000","last_clean":"2024-03-28T14:08:01.242483+0000","last_became_active":"2024-03-28T14:07:58.663001+0000","last_became_peered":"2024-03-28T14:07:58.663001+0000","last_unstale":"2024-03-28T14:08:01.242483+0000","last_undegraded":"2024-03-28T14:08:01.242483+0000","last_fullsized":"2024-03-28T14:08:01.242483+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:07:57.631994+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:07:57.631994+0000","last_clean_scrub_stamp":"2024-03-28T14:07:57.631994+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:35:03.722392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-28T14:08:00.835294+0000","last_change":"2024-03-28T14:07:59.731950+0000","last_active":"2024-03-28T14:08:00.835294+0000","last_peered":"2024-03-28T14:08:00.835294+0000","last_clean":"2024-03-28T14:08:00.835294+0000","last_became_active":"2024-03-28T14:07:59.731208+0000","last_became_peered":"2024-03-28T14:07:59.731208+0000","last_unstale":"2024-03-28T14:08:00.835294+0000","last_undegraded":"2024-03-28T14:08:00.835294+0000","last_fullsized":"2024-03-28T14:08:00.835294+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_clean_scrub_stamp":"2024-03-28T14:07:58.641775+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T14:17:34.720686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"20'1","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-28T14:08:00.696647+0000","last_change":"2024-03-28T14:07:58.684273+0000","last_active":"2024-03-28T14:08:00.696647+0000","last_peered":"2024-03-28T14:08:00.696647+0000","last_clean":"2024-03-28T14:08:00.696647+0000","last_became_active":"2024-03-28T14:07:58.683886+0000","last_became_peered":"2024-03-28T14:07:58.683886+0000","last_unstale":"2024-03-28T14:08:00.696647+0000","last_undegraded":"2024-03-28T14:08:00.696647+0000","last_fullsized":"2024-03-28T14:08:00.696647+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:07:57.631994+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:07:57.631994+0000","last_clean_scrub_stamp":"2024-03-28T14:07:57.631994+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T15:40:00.657964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-28T14:08:00.681505+0000","last_change":"2024-03-28T14:07:59.739696+0000","last_active":"2024-03-28T14:08:00.681505+0000","last_peered":"2024-03-28T14:08:00.681505+0000","last_clean":"2024-03-28T14:08:00.681505+0000","last_became_active":"2024-03-28T14:07:59.734736+0000","last_became_peered":"2024-03-28T14:07:59.734736+0000","last_unstale":"2024-03-28T14:08:00.681505+0000","last_undegraded":"2024-03-28T14:08:00.681505+0000","last_fullsized":"2024-03-28T14:08:00.681505+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_clean_scrub_stamp":"2024-03-28T14:07:58.641775+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:51:23.322586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-28T14:08:01.242313+0000","last_change":"2024-03-28T14:07:58.704650+0000","last_active":"2024-03-28T14:08:01.242313+0000","last_peered":"2024-03-28T14:08:01.242313+0000","last_clean":"2024-03-28T14:08:01.242313+0000","last_became_active":"2024-03-28T14:07:58.704305+0000","last_became_peered":"2024-03-28T14:07:58.704305+0000","last_unstale":"2024-03-28T14:08:01.242313+0000","last_undegraded":"2024-03-28T14:08:01.242313+0000","last_fullsized":"2024-03-28T14:08:01.242313+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:07:57.631994+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:07:57.631994+0000","last_clean_scrub_stamp":"2024-03-28T14:07:57.631994+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T17:35:00.153938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-28T14:08:00.834345+0000","last_change":"2024-03-28T14:07:59.745242+0000","last_active":"2024-03-28T14:08:00.834345+0000","last_peered":"2024-03-28T14:08:00.834345+0000","last_clean":"2024-03-28T14:08:00.834345+0000","last_became_active":"2024-03-28T14:07:59.744734+0000","last_became_peered":"2024-03-28T14:07:59.744734+0000","last_unstale":"2024-03-28T14:08:00.834345+0000","last_undegraded":"2024-03-28T14:08:00.834345+0000","last_fullsized":"2024-03-28T14:08:00.834345+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_clean_scrub_stamp":"2024-03-28T14:07:58.641775+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:49:58.161511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":170,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-28T14:08:00.833744+0000","last_change":"2024-03-28T14:07:49.234448+0000","last_active":"2024-03-28T14:08:00.833744+0000","last_peered":"2024-03-28T14:08:00.833744+0000","last_clean":"2024-03-28T14:08:00.833744+0000","last_became_active":"2024-03-28T14:07:49.233895+0000","last_became_peered":"2024-03-28T14:07:49.233895+0000","last_unstale":"2024-03-28T14:08:00.833744+0000","last_undegraded":"2024-03-28T14:08:00.833744+0000","last_fullsized":"2024-03-28T14:08:00.833744+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:07:48.069157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:07:48.069157+0000","last_clean_scrub_stamp":"2024-03-28T14:07:48.069157+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:11:39.977557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-28T14:08:00.680569+0000","last_change":"2024-03-28T14:07:58.699326+0000","last_active":"2024-03-28T14:08:00.680569+0000","last_peered":"2024-03-28T14:08:00.680569+0000","last_clean":"2024-03-28T14:08:00.680569+0000","last_became_active":"2024-03-28T14:07:58.698258+0000","last_became_peered":"2024-03-28T14:07:58.698258+0000","last_unstale":"2024-03-28T14:08:00.680569+0000","last_undegraded":"2024-03-28T14:08:00.680569+0000","last_fullsized":"2024-03-28T14:08:00.680569+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:07:57.631994+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:07:57.631994+0000","last_clean_scrub_stamp":"2024-03-28T14:07:57.631994+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T18:00:55.131543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-28T14:08:00.681381+0000","last_change":"2024-03-28T14:07:59.744015+0000","last_active":"2024-03-28T14:08:00.681381+0000","last_peered":"2024-03-28T14:08:00.681381+0000","last_clean":"2024-03-28T14:08:00.681381+0000","last_became_active":"2024-03-28T14:07:59.743723+0000","last_became_peered":"2024-03-28T14:07:59.743723+0000","last_unstale":"2024-03-28T14:08:00.681381+0000","last_undegraded":"2024-03-28T14:08:00.681381+0000","last_fullsized":"2024-03-28T14:08:00.681381+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_clean_scrub_stamp":"2024-03-28T14:07:58.641775+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T16:47:04.317583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-28T14:08:00.833592+0000","last_change":"2024-03-28T14:07:58.688278+0000","last_active":"2024-03-28T14:08:00.833592+0000","last_peered":"2024-03-28T14:08:00.833592+0000","last_clean":"2024-03-28T14:08:00.833592+0000","last_became_active":"2024-03-28T14:07:58.687442+0000","last_became_peered":"2024-03-28T14:07:58.687442+0000","last_unstale":"2024-03-28T14:08:00.833592+0000","last_undegraded":"2024-03-28T14:08:00.833592+0000","last_fullsized":"2024-03-28T14:08:00.833592+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:07:57.631994+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:07:57.631994+0000","last_clean_scrub_stamp":"2024-03-28T14:07:57.631994+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T17:35:00.153938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-28T14:08:01.243711+0000","last_change":"2024-03-28T14:07:59.715990+0000","last_active":"2024-03-28T14:08:01.243711+0000","last_peered":"2024-03-28T14:08:01.243711+0000","last_clean":"2024-03-28T14:08:01.243711+0000","last_became_active":"2024-03-28T14:07:59.715627+0000","last_became_peered":"2024-03-28T14:07:59.715627+0000","last_unstale":"2024-03-28T14:08:01.243711+0000","last_undegraded":"2024-03-28T14:08:01.243711+0000","last_fullsized":"2024-03-28T14:08:01.243711+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_clean_scrub_stamp":"2024-03-28T14:07:58.641775+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T18:46:53.473825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-28T14:08:01.242294+0000","last_change":"2024-03-28T14:07:58.663475+0000","last_active":"2024-03-28T14:08:01.242294+0000","last_peered":"2024-03-28T14:08:01.242294+0000","last_clean":"2024-03-28T14:08:01.242294+0000","last_became_active":"2024-03-28T14:07:58.662967+0000","last_became_peered":"2024-03-28T14:07:58.662967+0000","last_unstale":"2024-03-28T14:08:01.242294+0000","last_undegraded":"2024-03-28T14:08:01.242294+0000","last_fullsized":"2024-03-28T14:08:01.242294+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:07:57.631994+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:07:57.631994+0000","last_clean_scrub_stamp":"2024-03-28T14:07:57.631994+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T18:00:55.131543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-28T14:08:00.696122+0000","last_change":"2024-03-28T14:07:59.693645+0000","last_active":"2024-03-28T14:08:00.696122+0000","last_peered":"2024-03-28T14:08:00.696122+0000","last_clean":"2024-03-28T14:08:00.696122+0000","last_became_active":"2024-03-28T14:07:59.693346+0000","last_became_peered":"2024-03-28T14:07:59.693346+0000","last_unstale":"2024-03-28T14:08:00.696122+0000","last_undegraded":"2024-03-28T14:08:00.696122+0000","last_fullsized":"2024-03-28T14:08:00.696122+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_clean_scrub_stamp":"2024-03-28T14:07:58.641775+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:57:11.908191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-28T14:08:00.680856+0000","last_change":"2024-03-28T14:07:58.702868+0000","last_active":"2024-03-28T14:08:00.680856+0000","last_peered":"2024-03-28T14:08:00.680856+0000","last_clean":"2024-03-28T14:08:00.680856+0000","last_became_active":"2024-03-28T14:07:58.700099+0000","last_became_peered":"2024-03-28T14:07:58.700099+0000","last_unstale":"2024-03-28T14:08:00.680856+0000","last_undegraded":"2024-03-28T14:08:00.680856+0000","last_fullsized":"2024-03-28T14:08:00.680856+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:07:57.631994+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:07:57.631994+0000","last_clean_scrub_stamp":"2024-03-28T14:07:57.631994+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:30:19.088624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-28T14:08:01.243699+0000","last_change":"2024-03-28T14:07:59.707739+0000","last_active":"2024-03-28T14:08:01.243699+0000","last_peered":"2024-03-28T14:08:01.243699+0000","last_clean":"2024-03-28T14:08:01.243699+0000","last_became_active":"2024-03-28T14:07:59.705252+0000","last_became_peered":"2024-03-28T14:07:59.705252+0000","last_unstale":"2024-03-28T14:08:01.243699+0000","last_undegraded":"2024-03-28T14:08:01.243699+0000","last_fullsized":"2024-03-28T14:08:01.243699+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_clean_scrub_stamp":"2024-03-28T14:07:58.641775+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:09:39.273902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-28T14:08:00.696496+0000","last_change":"2024-03-28T14:07:58.669940+0000","last_active":"2024-03-28T14:08:00.696496+0000","last_peered":"2024-03-28T14:08:00.696496+0000","last_clean":"2024-03-28T14:08:00.696496+0000","last_became_active":"2024-03-28T14:07:58.669568+0000","last_became_peered":"2024-03-28T14:07:58.669568+0000","last_unstale":"2024-03-28T14:08:00.696496+0000","last_undegraded":"2024-03-28T14:08:00.696496+0000","last_fullsized":"2024-03-28T14:08:00.696496+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:07:57.631994+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:07:57.631994+0000","last_clean_scrub_stamp":"2024-03-28T14:07:57.631994+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:09:38.264121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-28T14:08:00.681619+0000","last_change":"2024-03-28T14:07:59.742899+0000","last_active":"2024-03-28T14:08:00.681619+0000","last_peered":"2024-03-28T14:08:00.681619+0000","last_clean":"2024-03-28T14:08:00.681619+0000","last_became_active":"2024-03-28T14:07:59.740075+0000","last_became_peered":"2024-03-28T14:07:59.740075+0000","last_unstale":"2024-03-28T14:08:00.681619+0000","last_undegraded":"2024-03-28T14:08:00.681619+0000","last_fullsized":"2024-03-28T14:08:00.681619+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_clean_scrub_stamp":"2024-03-28T14:07:58.641775+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:59:25.862480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-28T14:08:00.834925+0000","last_change":"2024-03-28T14:07:58.687874+0000","last_active":"2024-03-28T14:08:00.834925+0000","last_peered":"2024-03-28T14:08:00.834925+0000","last_clean":"2024-03-28T14:08:00.834925+0000","last_became_active":"2024-03-28T14:07:58.686656+0000","last_became_peered":"2024-03-28T14:07:58.686656+0000","last_unstale":"2024-03-28T14:08:00.834925+0000","last_undegraded":"2024-03-28T14:08:00.834925+0000","last_fullsized":"2024-03-28T14:08:00.834925+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:07:57.631994+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:07:57.631994+0000","last_clean_scrub_stamp":"2024-03-28T14:07:57.631994+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T15:40:00.657964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-28T14:08:00.696528+0000","last_change":"2024-03-28T14:07:59.700465+0000","last_active":"2024-03-28T14:08:00.696528+0000","last_peered":"2024-03-28T14:08:00.696528+0000","last_clean":"2024-03-28T14:08:00.696528+0000","last_became_active":"2024-03-28T14:07:59.699705+0000","last_became_peered":"2024-03-28T14:07:59.699705+0000","last_unstale":"2024-03-28T14:08:00.696528+0000","last_undegraded":"2024-03-28T14:08:00.696528+0000","last_fullsized":"2024-03-28T14:08:00.696528+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_clean_scrub_stamp":"2024-03-28T14:07:58.641775+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:25:39.647896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"21'1","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-28T14:08:00.700925+0000","last_change":"2024-03-28T14:07:58.681340+0000","last_active":"2024-03-28T14:08:00.700925+0000","last_peered":"2024-03-28T14:08:00.700925+0000","last_clean":"2024-03-28T14:08:00.700925+0000","last_became_active":"2024-03-28T14:07:58.681006+0000","last_became_peered":"2024-03-28T14:07:58.681006+0000","last_unstale":"2024-03-28T14:08:00.700925+0000","last_undegraded":"2024-03-28T14:08:00.700925+0000","last_fullsized":"2024-03-28T14:08:00.700925+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:07:57.631994+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:07:57.631994+0000","last_clean_scrub_stamp":"2024-03-28T14:07:57.631994+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:04:40.185372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-28T14:08:00.681402+0000","last_change":"2024-03-28T14:07:59.727008+0000","last_active":"2024-03-28T14:08:00.681402+0000","last_peered":"2024-03-28T14:08:00.681402+0000","last_clean":"2024-03-28T14:08:00.681402+0000","last_became_active":"2024-03-28T14:07:59.726702+0000","last_became_peered":"2024-03-28T14:07:59.726702+0000","last_unstale":"2024-03-28T14:08:00.681402+0000","last_undegraded":"2024-03-28T14:08:00.681402+0000","last_fullsized":"2024-03-28T14:08:00.681402+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_clean_scrub_stamp":"2024-03-28T14:07:58.641775+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T17:46:06.253111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"20'2","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-28T14:08:00.703440+0000","last_change":"2024-03-28T14:07:58.685500+0000","last_active":"2024-03-28T14:08:00.703440+0000","last_peered":"2024-03-28T14:08:00.703440+0000","last_clean":"2024-03-28T14:08:00.703440+0000","last_became_active":"2024-03-28T14:07:58.684781+0000","last_became_peered":"2024-03-28T14:07:58.684781+0000","last_unstale":"2024-03-28T14:08:00.703440+0000","last_undegraded":"2024-03-28T14:08:00.703440+0000","last_fullsized":"2024-03-28T14:08:00.703440+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:07:57.631994+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:07:57.631994+0000","last_clean_scrub_stamp":"2024-03-28T14:07:57.631994+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:09:38.264121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-28T14:08:01.244918+0000","last_change":"2024-03-28T14:07:59.707407+0000","last_active":"2024-03-28T14:08:01.244918+0000","last_peered":"2024-03-28T14:08:01.244918+0000","last_clean":"2024-03-28T14:08:01.244918+0000","last_became_active":"2024-03-28T14:07:59.707103+0000","last_became_peered":"2024-03-28T14:07:59.707103+0000","last_unstale":"2024-03-28T14:08:01.244918+0000","last_undegraded":"2024-03-28T14:08:01.244918+0000","last_fullsized":"2024-03-28T14:08:01.244918+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_clean_scrub_stamp":"2024-03-28T14:07:58.641775+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:37:29.402787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-28T14:08:00.833570+0000","last_change":"2024-03-28T14:07:58.682456+0000","last_active":"2024-03-28T14:08:00.833570+0000","last_peered":"2024-03-28T14:08:00.833570+0000","last_clean":"2024-03-28T14:08:00.833570+0000","last_became_active":"2024-03-28T14:07:58.678987+0000","last_became_peered":"2024-03-28T14:07:58.678987+0000","last_unstale":"2024-03-28T14:08:00.833570+0000","last_undegraded":"2024-03-28T14:08:00.833570+0000","last_fullsized":"2024-03-28T14:08:00.833570+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:07:57.631994+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:07:57.631994+0000","last_clean_scrub_stamp":"2024-03-28T14:07:57.631994+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:30:19.088624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-28T14:08:01.243764+0000","last_change":"2024-03-28T14:07:59.697567+0000","last_active":"2024-03-28T14:08:01.243764+0000","last_peered":"2024-03-28T14:08:01.243764+0000","last_clean":"2024-03-28T14:08:01.243764+0000","last_became_active":"2024-03-28T14:07:59.693975+0000","last_became_peered":"2024-03-28T14:07:59.693975+0000","last_unstale":"2024-03-28T14:08:01.243764+0000","last_undegraded":"2024-03-28T14:08:01.243764+0000","last_fullsized":"2024-03-28T14:08:01.243764+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_clean_scrub_stamp":"2024-03-28T14:07:58.641775+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:04:41.195153+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-28T14:08:00.696246+0000","last_change":"2024-03-28T14:07:58.680918+0000","last_active":"2024-03-28T14:08:00.696246+0000","last_peered":"2024-03-28T14:08:00.696246+0000","last_clean":"2024-03-28T14:08:00.696246+0000","last_became_active":"2024-03-28T14:07:58.680548+0000","last_became_peered":"2024-03-28T14:07:58.680548+0000","last_unstale":"2024-03-28T14:08:00.696246+0000","last_undegraded":"2024-03-28T14:08:00.696246+0000","last_fullsized":"2024-03-28T14:08:00.696246+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:07:57.631994+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:07:57.631994+0000","last_clean_scrub_stamp":"2024-03-28T14:07:57.631994+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:30:19.088624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-28T14:08:00.680594+0000","last_change":"2024-03-28T14:07:59.702172+0000","last_active":"2024-03-28T14:08:00.680594+0000","last_peered":"2024-03-28T14:08:00.680594+0000","last_clean":"2024-03-28T14:08:00.680594+0000","last_became_active":"2024-03-28T14:07:59.701709+0000","last_became_peered":"2024-03-28T14:07:59.701709+0000","last_unstale":"2024-03-28T14:08:00.680594+0000","last_undegraded":"2024-03-28T14:08:00.680594+0000","last_fullsized":"2024-03-28T14:08:00.680594+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_clean_scrub_stamp":"2024-03-28T14:07:58.641775+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:57:11.908191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-28T14:08:00.696422+0000","last_change":"2024-03-28T14:07:58.672681+0000","last_active":"2024-03-28T14:08:00.696422+0000","last_peered":"2024-03-28T14:08:00.696422+0000","last_clean":"2024-03-28T14:08:00.696422+0000","last_became_active":"2024-03-28T14:07:58.672094+0000","last_became_peered":"2024-03-28T14:07:58.672094+0000","last_unstale":"2024-03-28T14:08:00.696422+0000","last_undegraded":"2024-03-28T14:08:00.696422+0000","last_fullsized":"2024-03-28T14:08:00.696422+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:07:57.631994+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:07:57.631994+0000","last_clean_scrub_stamp":"2024-03-28T14:07:57.631994+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T17:35:00.153938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-28T14:08:00.681203+0000","last_change":"2024-03-28T14:07:59.705251+0000","last_active":"2024-03-28T14:08:00.681203+0000","last_peered":"2024-03-28T14:08:00.681203+0000","last_clean":"2024-03-28T14:08:00.681203+0000","last_became_active":"2024-03-28T14:07:59.703366+0000","last_became_peered":"2024-03-28T14:07:59.703366+0000","last_unstale":"2024-03-28T14:08:00.681203+0000","last_undegraded":"2024-03-28T14:08:00.681203+0000","last_fullsized":"2024-03-28T14:08:00.681203+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_clean_scrub_stamp":"2024-03-28T14:07:58.641775+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:41:27.409736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"20'2","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-28T14:08:00.834672+0000","last_change":"2024-03-28T14:07:58.688811+0000","last_active":"2024-03-28T14:08:00.834672+0000","last_peered":"2024-03-28T14:08:00.834672+0000","last_clean":"2024-03-28T14:08:00.834672+0000","last_became_active":"2024-03-28T14:07:58.688054+0000","last_became_peered":"2024-03-28T14:07:58.688054+0000","last_unstale":"2024-03-28T14:08:00.834672+0000","last_undegraded":"2024-03-28T14:08:00.834672+0000","last_fullsized":"2024-03-28T14:08:00.834672+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:07:57.631994+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:07:57.631994+0000","last_clean_scrub_stamp":"2024-03-28T14:07:57.631994+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:04:40.185372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-28T14:08:00.702555+0000","last_change":"2024-03-28T14:07:59.709224+0000","last_active":"2024-03-28T14:08:00.702555+0000","last_peered":"2024-03-28T14:08:00.702555+0000","last_clean":"2024-03-28T14:08:00.702555+0000","last_became_active":"2024-03-28T14:07:59.704202+0000","last_became_peered":"2024-03-28T14:07:59.704202+0000","last_unstale":"2024-03-28T14:08:00.702555+0000","last_undegraded":"2024-03-28T14:08:00.702555+0000","last_fullsized":"2024-03-28T14:08:00.702555+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_clean_scrub_stamp":"2024-03-28T14:07:58.641775+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:57:11.908191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-28T14:08:00.681258+0000","last_change":"2024-03-28T14:07:59.739995+0000","last_active":"2024-03-28T14:08:00.681258+0000","last_peered":"2024-03-28T14:08:00.681258+0000","last_clean":"2024-03-28T14:08:00.681258+0000","last_became_active":"2024-03-28T14:07:59.739369+0000","last_became_peered":"2024-03-28T14:07:59.739369+0000","last_unstale":"2024-03-28T14:08:00.681258+0000","last_undegraded":"2024-03-28T14:08:00.681258+0000","last_fullsized":"2024-03-28T14:08:00.681258+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_clean_scrub_stamp":"2024-03-28T14:07:58.641775+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:34:47.572488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"20'2","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-28T14:08:00.834148+0000","last_change":"2024-03-28T14:07:58.705499+0000","last_active":"2024-03-28T14:08:00.834148+0000","last_peered":"2024-03-28T14:08:00.834148+0000","last_clean":"2024-03-28T14:08:00.834148+0000","last_became_active":"2024-03-28T14:07:58.704848+0000","last_became_peered":"2024-03-28T14:07:58.704848+0000","last_unstale":"2024-03-28T14:08:00.834148+0000","last_undegraded":"2024-03-28T14:08:00.834148+0000","last_fullsized":"2024-03-28T14:08:00.834148+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:07:57.631994+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:07:57.631994+0000","last_clean_scrub_stamp":"2024-03-28T14:07:57.631994+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:09:38.264121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-28T14:08:00.680957+0000","last_change":"2024-03-28T14:07:59.740931+0000","last_active":"2024-03-28T14:08:00.680957+0000","last_peered":"2024-03-28T14:08:00.680957+0000","last_clean":"2024-03-28T14:08:00.680957+0000","last_became_active":"2024-03-28T14:07:59.740255+0000","last_became_peered":"2024-03-28T14:07:59.740255+0000","last_unstale":"2024-03-28T14:08:00.680957+0000","last_undegraded":"2024-03-28T14:08:00.680957+0000","last_fullsized":"2024-03-28T14:08:00.680957+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_clean_scrub_stamp":"2024-03-28T14:07:58.641775+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:49:58.161511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"20'2","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-28T14:08:00.834791+0000","last_change":"2024-03-28T14:07:58.693440+0000","last_active":"2024-03-28T14:08:00.834791+0000","last_peered":"2024-03-28T14:08:00.834791+0000","last_clean":"2024-03-28T14:08:00.834791+0000","last_became_active":"2024-03-28T14:07:58.692096+0000","last_became_peered":"2024-03-28T14:07:58.692096+0000","last_unstale":"2024-03-28T14:08:00.834791+0000","last_undegraded":"2024-03-28T14:08:00.834791+0000","last_fullsized":"2024-03-28T14:08:00.834791+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:07:57.631994+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:07:57.631994+0000","last_clean_scrub_stamp":"2024-03-28T14:07:57.631994+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T15:40:58.102613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-28T14:08:01.244001+0000","last_change":"2024-03-28T14:07:59.744398+0000","last_active":"2024-03-28T14:08:01.244001+0000","last_peered":"2024-03-28T14:08:01.244001+0000","last_clean":"2024-03-28T14:08:01.244001+0000","last_became_active":"2024-03-28T14:07:59.744127+0000","last_became_peered":"2024-03-28T14:07:59.744127+0000","last_unstale":"2024-03-28T14:08:01.244001+0000","last_undegraded":"2024-03-28T14:08:01.244001+0000","last_fullsized":"2024-03-28T14:08:01.244001+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_clean_scrub_stamp":"2024-03-28T14:07:58.641775+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T14:20:33.555524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"20'1","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-28T14:08:00.681112+0000","last_change":"2024-03-28T14:07:58.671168+0000","last_active":"2024-03-28T14:08:00.681112+0000","last_peered":"2024-03-28T14:08:00.681112+0000","last_clean":"2024-03-28T14:08:00.681112+0000","last_became_active":"2024-03-28T14:07:58.670691+0000","last_became_peered":"2024-03-28T14:07:58.670691+0000","last_unstale":"2024-03-28T14:08:00.681112+0000","last_undegraded":"2024-03-28T14:08:00.681112+0000","last_fullsized":"2024-03-28T14:08:00.681112+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:07:57.631994+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:07:57.631994+0000","last_clean_scrub_stamp":"2024-03-28T14:07:57.631994+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T17:34:09.030115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-28T14:08:00.680644+0000","last_change":"2024-03-28T14:07:59.727344+0000","last_active":"2024-03-28T14:08:00.680644+0000","last_peered":"2024-03-28T14:08:00.680644+0000","last_clean":"2024-03-28T14:08:00.680644+0000","last_became_active":"2024-03-28T14:07:59.727023+0000","last_became_peered":"2024-03-28T14:07:59.727023+0000","last_unstale":"2024-03-28T14:08:00.680644+0000","last_undegraded":"2024-03-28T14:08:00.680644+0000","last_fullsized":"2024-03-28T14:08:00.680644+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_clean_scrub_stamp":"2024-03-28T14:07:58.641775+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:24:30.197596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"20'4","reported_seq":18,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-28T14:08:01.245810+0000","last_change":"2024-03-28T14:07:58.676958+0000","last_active":"2024-03-28T14:08:01.245810+0000","last_peered":"2024-03-28T14:08:01.245810+0000","last_clean":"2024-03-28T14:08:01.245810+0000","last_became_active":"2024-03-28T14:07:58.676611+0000","last_became_peered":"2024-03-28T14:07:58.676611+0000","last_unstale":"2024-03-28T14:08:01.245810+0000","last_undegraded":"2024-03-28T14:08:01.245810+0000","last_fullsized":"2024-03-28T14:08:01.245810+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:07:57.631994+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:07:57.631994+0000","last_clean_scrub_stamp":"2024-03-28T14:07:57.631994+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:59:04.674426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574854,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27820,"kb_used_data":796,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344020,"statfs":{"total":96636764160,"available":96608276480,"internally_reserved":0,"allocated":815104,"data_stored":666555,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574855,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27248,"kb_used_data":224,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344592,"statfs":{"total":96636764160,"available":96608862208,"internally_reserved":0,"allocated":229376,"data_stored":77217,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574853,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27240,"kb_used_data":216,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344600,"statfs":{"total":96636764160,"available":96608870400,"internally_reserved":0,"allocated":221184,"data_stored":75695,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":11,"apply_latency_ms":11,"commit_latency_ns":11000000,"apply_latency_ns":11000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574854,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27248,"kb_used_data":224,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344592,"statfs":{"total":96636764160,"available":96608862208,"internally_reserved":0,"allocated":229376,"data_stored":75731,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574856,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27248,"kb_used_data":224,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344592,"statfs":{"total":96636764160,"available":96608862208,"internally_reserved":0,"allocated":229376,"data_stored":77217,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574854,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27236,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344604,"statfs":{"total":96636764160,"available":96608874496,"internally_reserved":0,"allocated":217088,"data_stored":75605,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574852,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27244,"kb_used_data":220,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344596,"statfs":{"total":96636764160,"available":96608866304,"internally_reserved":0,"allocated":225280,"data_stored":75765,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574855,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27824,"kb_used_data":804,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344016,"statfs":{"total":96636764160,"available":96608272384,"internally_reserved":0,"allocated":823296,"data_stored":666611,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-28T14:08:04.134 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-03-28T14:08:04.134 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-03-28T14:08:04.134 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2024-03-28T14:08:04.134 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2024-03-28T14:08:04.135 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2024-03-28T14:08:04.278 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:04.275+0000 7f5a9e848640 1 Processor -- start 2024-03-28T14:08:04.279 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:04.275+0000 7f5a9e848640 1 -- start start 2024-03-28T14:08:04.280 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:04.275+0000 7f5a9e848640 1 --2- >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f5a9806b830 0x7f5a98072ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:04.280 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:04.275+0000 7f5a9e848640 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f5a980b4f70 0x7f5a9813f650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:04.280 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:04.275+0000 7f5a9e848640 1 --2- >> v2:172.21.15.171:3300/0 conn(0x7f5a980b54c0 0x7f5a9806daf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:04.280 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:04.275+0000 7f5a9e848640 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 -- 0x7f5a980b6060 con 0x7f5a980b4f70 2024-03-28T14:08:04.280 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:04.275+0000 7f5a9e848640 1 -- --> v2:172.21.15.171:3300/0 -- mon_getmap magic: 0 -- 0x7f5a9806bc10 con 0x7f5a980b54c0 2024-03-28T14:08:04.280 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:04.275+0000 7f5a9e848640 1 -- --> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] -- mon_getmap magic: 0 -- 0x7f5a980b5a10 con 0x7f5a9806b830 2024-03-28T14:08:04.280 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:04.279+0000 7f5a977fe640 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f5a980b4f70 0x7f5a9813f650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:04.280 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:04.279+0000 7f5a977fe640 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f5a980b4f70 0x7f5a9813f650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:42346/0 (socket says 172.21.15.104:42346) 2024-03-28T14:08:04.280 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:04.279+0000 7f5a977fe640 1 -- 172.21.15.104:0/3907984890 learned_addr learned my addr 172.21.15.104:0/3907984890 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-03-28T14:08:04.281 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:04.279+0000 7f5a97fff640 1 --2- 172.21.15.104:0/3907984890 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f5a9806b830 0x7f5a98072ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:04.281 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:04.279+0000 7f5a977fe640 1 -- 172.21.15.104:0/3907984890 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f5a9806b830 msgr2=0x7f5a98072ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T14:08:04.281 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:04.279+0000 7f5a977fe640 1 --2- 172.21.15.104:0/3907984890 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f5a9806b830 0x7f5a98072ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:04.281 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:04.279+0000 7f5a9cde5640 1 --2- 172.21.15.104:0/3907984890 >> v2:172.21.15.171:3300/0 conn(0x7f5a980b54c0 0x7f5a9806daf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:04.281 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:04.279+0000 7f5a977fe640 1 -- 172.21.15.104:0/3907984890 >> v2:172.21.15.171:3300/0 conn(0x7f5a980b54c0 msgr2=0x7f5a9806daf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T14:08:04.281 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:04.279+0000 7f5a977fe640 1 --2- 172.21.15.104:0/3907984890 >> v2:172.21.15.171:3300/0 conn(0x7f5a980b54c0 0x7f5a9806daf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:04.281 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:04.279+0000 7f5a977fe640 1 -- 172.21.15.104:0/3907984890 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5a980b5cf0 con 0x7f5a980b4f70 2024-03-28T14:08:04.281 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:04.279+0000 7f5a9cde5640 1 --2- 172.21.15.104:0/3907984890 >> v2:172.21.15.171:3300/0 conn(0x7f5a980b54c0 0x7f5a9806daf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T14:08:04.282 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:04.279+0000 7f5a977fe640 1 --2- 172.21.15.104:0/3907984890 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5a980b4f70 0x7f5a9813f650 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f5a9806c160 tx=0x7f5a8001c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c66fe32e1d9b9155 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:08:04.282 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:04.279+0000 7f5a96ffd640 1 -- 172.21.15.104:0/3907984890 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a800b2070 con 0x7f5a980b4f70 2024-03-28T14:08:04.282 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:04.279+0000 7f5a96ffd640 1 -- 172.21.15.104:0/3907984890 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5a800a9070 con 0x7f5a980b4f70 2024-03-28T14:08:04.282 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:04.279+0000 7f5a96ffd640 1 -- 172.21.15.104:0/3907984890 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a800ad880 con 0x7f5a980b4f70 2024-03-28T14:08:04.283 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:04.279+0000 7f5a9e848640 1 -- 172.21.15.104:0/3907984890 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5a980b4f70 msgr2=0x7f5a9813f650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:04.283 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:04.279+0000 7f5a9e848640 1 --2- 172.21.15.104:0/3907984890 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5a980b4f70 0x7f5a9813f650 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f5a9806c160 tx=0x7f5a8001c490 comp rx=0 tx=0).stop 2024-03-28T14:08:04.283 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:04.279+0000 7f5a9e848640 1 -- 172.21.15.104:0/3907984890 shutdown_connections 2024-03-28T14:08:04.283 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:04.279+0000 7f5a9e848640 1 --2- 172.21.15.104:0/3907984890 >> v2:172.21.15.171:3300/0 conn(0x7f5a980b54c0 0x7f5a9806daf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:04.283 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:04.279+0000 7f5a9e848640 1 --2- 172.21.15.104:0/3907984890 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5a980b4f70 0x7f5a9813f650 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:04.283 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:04.279+0000 7f5a9e848640 1 --2- 172.21.15.104:0/3907984890 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f5a9806b830 0x7f5a98072ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:04.283 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:04.279+0000 7f5a9e848640 1 -- 172.21.15.104:0/3907984890 >> 172.21.15.104:0/3907984890 conn(0x7f5a980ba400 msgr2=0x7f5a980ba810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T14:08:04.283 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:04.279+0000 7f5a9e848640 1 -- 172.21.15.104:0/3907984890 shutdown_connections 2024-03-28T14:08:04.284 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:04.279+0000 7f5a9e848640 1 -- 172.21.15.104:0/3907984890 wait complete. 2024-03-28T14:08:04.286 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:04.283+0000 7f5a9e848640 1 Processor -- start 2024-03-28T14:08:04.286 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:04.283+0000 7f5a9e848640 1 -- start start 2024-03-28T14:08:04.286 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:04.283+0000 7f5a9e848640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5a9806b830 0x7f5a980b31d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:04.286 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:04.283+0000 7f5a9e848640 1 --2- >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f5a980b4f70 0x7f5a98142b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:04.286 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:04.283+0000 7f5a9e848640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f5a980b54c0 0x7f5a980b1de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:04.286 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:04.283+0000 7f5a9e848640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f5a98151ff0 con 0x7f5a9806b830 2024-03-28T14:08:04.286 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:04.283+0000 7f5a9e848640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f5a98152170 con 0x7f5a980b54c0 2024-03-28T14:08:04.286 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:04.283+0000 7f5a9e848640 1 -- --> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] -- mon_getmap magic: 0 -- 0x7f5a981522f0 con 0x7f5a980b4f70 2024-03-28T14:08:04.286 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:04.283+0000 7f5a97fff640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5a9806b830 0x7f5a980b31d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:04.286 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:04.283+0000 7f5a97fff640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5a9806b830 0x7f5a980b31d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:42358/0 (socket says 172.21.15.104:42358) 2024-03-28T14:08:04.286 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:04.283+0000 7f5a97fff640 1 -- 172.21.15.104:0/4073623824 learned_addr learned my addr 172.21.15.104:0/4073623824 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-03-28T14:08:04.287 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:04.283+0000 7f5a9cde5640 1 --2- 172.21.15.104:0/4073623824 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f5a980b54c0 0x7f5a980b1de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:04.287 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:04.283+0000 7f5a977fe640 1 --2- 172.21.15.104:0/4073623824 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f5a980b4f70 0x7f5a98142b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:04.287 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:04.283+0000 7f5a97fff640 1 -- 172.21.15.104:0/4073623824 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f5a980b4f70 msgr2=0x7f5a98142b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:04.287 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:04.283+0000 7f5a97fff640 1 --2- 172.21.15.104:0/4073623824 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f5a980b4f70 0x7f5a98142b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:04.287 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:04.283+0000 7f5a97fff640 1 -- 172.21.15.104:0/4073623824 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f5a980b54c0 msgr2=0x7f5a980b1de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:04.287 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:04.283+0000 7f5a97fff640 1 --2- 172.21.15.104:0/4073623824 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f5a980b54c0 0x7f5a980b1de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:04.287 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:04.283+0000 7f5a97fff640 1 -- 172.21.15.104:0/4073623824 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5a98140300 con 0x7f5a9806b830 2024-03-28T14:08:04.287 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:04.283+0000 7f5a9cde5640 1 --2- 172.21.15.104:0/4073623824 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f5a980b54c0 0x7f5a980b1de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T14:08:04.287 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:04.283+0000 7f5a97fff640 1 --2- 172.21.15.104:0/4073623824 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5a9806b830 0x7f5a980b31d0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f5a8800c7f0 tx=0x7f5a8800ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:08:04.287 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:04.283+0000 7f5a94ff9640 1 -- 172.21.15.104:0/4073623824 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a88015070 con 0x7f5a9806b830 2024-03-28T14:08:04.287 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:04.283+0000 7f5a94ff9640 1 -- 172.21.15.104:0/4073623824 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5a88004220 con 0x7f5a9806b830 2024-03-28T14:08:04.287 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:04.283+0000 7f5a94ff9640 1 -- 172.21.15.104:0/4073623824 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a88003620 con 0x7f5a9806b830 2024-03-28T14:08:04.287 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:04.283+0000 7f5a9e848640 1 -- 172.21.15.104:0/4073623824 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5a981405e0 con 0x7f5a9806b830 2024-03-28T14:08:04.287 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:04.283+0000 7f5a9e848640 1 -- 172.21.15.104:0/4073623824 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5a981def00 con 0x7f5a9806b830 2024-03-28T14:08:04.288 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:04.283+0000 7f5a9e848640 1 -- 172.21.15.104:0/4073623824 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5a980687c0 con 0x7f5a9806b830 2024-03-28T14:08:04.288 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:04.283+0000 7f5a94ff9640 1 -- 172.21.15.104:0/4073623824 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) ==== 87803+0+0 (secure 0 0 0) 0x7f5a880037c0 con 0x7f5a9806b830 2024-03-28T14:08:04.288 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:04.283+0000 7f5a94ff9640 1 --2- 172.21.15.104:0/4073623824 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f5a78067fc0 0x7f5a7806a480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:04.288 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:04.283+0000 7f5a94ff9640 1 -- 172.21.15.104:0/4073623824 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f5a88086330 con 0x7f5a9806b830 2024-03-28T14:08:04.289 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:04.287+0000 7f5a977fe640 1 --2- 172.21.15.104:0/4073623824 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f5a78067fc0 0x7f5a7806a480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:04.289 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:04.287+0000 7f5a977fe640 1 --2- 172.21.15.104:0/4073623824 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f5a78067fc0 0x7f5a7806a480 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f5a800af000 tx=0x7f5a80008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:08:04.296 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:04.295+0000 7f5a94ff9640 1 -- 172.21.15.104:0/4073623824 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179117 (secure 0 0 0) 0x7f5a88056af0 con 0x7f5a9806b830 2024-03-28T14:08:04.482 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:04.479+0000 7f5a9e848640 1 -- 172.21.15.104:0/4073623824 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) -- 0x7f5a980b89c0 con 0x7f5a9806b830 2024-03-28T14:08:04.484 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:04.483+0000 7f5a94ff9640 1 -- 172.21.15.104:0/4073623824 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v5) ==== 74+0+167690 (secure 0 0 0) 0x7f5a880563f0 con 0x7f5a9806b830 2024-03-28T14:08:04.486 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-03-28T14:08:04.488 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:04.487+0000 7f5a9e848640 1 -- 172.21.15.104:0/4073623824 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f5a78067fc0 msgr2=0x7f5a7806a480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:04.488 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:04.487+0000 7f5a9e848640 1 --2- 172.21.15.104:0/4073623824 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f5a78067fc0 0x7f5a7806a480 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f5a800af000 tx=0x7f5a80008040 comp rx=0 tx=0).stop 2024-03-28T14:08:04.488 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:04.487+0000 7f5a9e848640 1 -- 172.21.15.104:0/4073623824 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5a9806b830 msgr2=0x7f5a980b31d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:04.488 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:04.487+0000 7f5a9e848640 1 --2- 172.21.15.104:0/4073623824 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5a9806b830 0x7f5a980b31d0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f5a8800c7f0 tx=0x7f5a8800ccb0 comp rx=0 tx=0).stop 2024-03-28T14:08:04.488 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:04.487+0000 7f5a9e848640 1 -- 172.21.15.104:0/4073623824 shutdown_connections 2024-03-28T14:08:04.488 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:04.487+0000 7f5a9e848640 1 --2- 172.21.15.104:0/4073623824 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f5a980b54c0 0x7f5a980b1de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:04.488 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:04.487+0000 7f5a9e848640 1 --2- 172.21.15.104:0/4073623824 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f5a980b4f70 0x7f5a98142b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:04.489 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:04.487+0000 7f5a9e848640 1 --2- 172.21.15.104:0/4073623824 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f5a78067fc0 0x7f5a7806a480 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:04.489 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:04.487+0000 7f5a9e848640 1 --2- 172.21.15.104:0/4073623824 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5a9806b830 0x7f5a980b31d0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:04.489 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:04.487+0000 7f5a9e848640 1 -- 172.21.15.104:0/4073623824 >> 172.21.15.104:0/4073623824 conn(0x7f5a980ba400 msgr2=0x7f5a980b71c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T14:08:04.489 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:04.487+0000 7f5a9e848640 1 -- 172.21.15.104:0/4073623824 shutdown_connections 2024-03-28T14:08:04.489 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:04.487+0000 7f5a9e848640 1 -- 172.21.15.104:0/4073623824 wait complete. 2024-03-28T14:08:04.502 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":5,"flags":0,"active_gid":4100,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6832","nonce":1522752412},{"type":"v1","addr":"172.21.15.171:6833","nonce":1522752412}]},"active_addr":"172.21.15.171:6833/1522752412","active_change":"2024-03-28T14:07:39.561485+0000","active_mgr_features":4540701547738038271,"available":true,"standbys":[{"gid":4113,"name":"y","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.0.0","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.0.0","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.171:0","nonce":1993911710}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.171:0","nonce":4125455980}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.171:0","nonce":2586866367}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.171:0","nonce":3315860852}]}]} 2024-03-28T14:08:04.506 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2024-03-28T14:08:04.506 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2024-03-28T14:08:04.506 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-28T14:08:04.628 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:04.627+0000 7fe4f4e96640 1 Processor -- start 2024-03-28T14:08:04.628 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:04.627+0000 7fe4f4e96640 1 -- start start 2024-03-28T14:08:04.629 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:04.627+0000 7fe4f4e96640 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fe4f014b090 0x7fe4f014b540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:04.629 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:04.627+0000 7fe4f4e96640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe4f006ea00 con 0x7fe4f014ba80 2024-03-28T14:08:04.629 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:04.627+0000 7fe4f4e96640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe4f006e760 con 0x7fe4f00ba970 2024-03-28T14:08:04.629 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:04.627+0000 7fe4f4e96640 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 -- 0x7fe4f00b5910 con 0x7fe4f014b090 2024-03-28T14:08:04.629 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:04.627+0000 7fe4eed9d640 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7fe4f014ba80 0x7fe4f0152390 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:47308/0 (socket says 172.21.15.104:47308) 2024-03-28T14:08:04.629 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:04.627+0000 7fe4eed9d640 1 -- 172.21.15.104:0/525046268 learned_addr learned my addr 172.21.15.104:0/525046268 (peer_addr_for_me v1:172.21.15.104:0/0) 2024-03-28T14:08:04.629 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:04.627+0000 7fe4e5d9b640 1 --2- 172.21.15.104:0/525046268 >> v2:172.21.15.104:3300/0 conn(0x7fe4f014b090 0x7fe4f014b540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:04.629 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:04.627+0000 7fe4edd9b640 1 -- 172.21.15.104:0/525046268 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 322532555 0 0) 0x7fe4e80014a0 con 0x7fe4f014ba80 2024-03-28T14:08:04.629 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:04.627+0000 7fe4edd9b640 1 -- 172.21.15.104:0/525046268 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe4dc003530 con 0x7fe4f014ba80 2024-03-28T14:08:04.629 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:04.627+0000 7fe4e5d9b640 1 -- 172.21.15.104:0/525046268 >> v1:172.21.15.171:6789/0 conn(0x7fe4f00ba970 legacy=0x7fe4f00b5d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:04.630 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:04.627+0000 7fe4edd9b640 1 -- 172.21.15.104:0/525046268 <== mon.0 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4257648020 0 0) 0x7fe4dc003530 con 0x7fe4f014ba80 2024-03-28T14:08:04.630 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:04.627+0000 7fe4e5d9b640 1 -- 172.21.15.104:0/525046268 >> v1:172.21.15.104:6789/0 conn(0x7fe4f014ba80 legacy=0x7fe4f0152390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:04.630 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:04.627+0000 7fe4e5d9b640 1 -- 172.21.15.104:0/525046268 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe4f00bad50 con 0x7fe4f014b090 2024-03-28T14:08:04.631 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:04.631+0000 7fe4e5d9b640 1 --2- 172.21.15.104:0/525046268 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe4f014b090 0x7fe4f014b540 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fe4d80097c0 tx=0x7fe4d801c2a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e011f0d1736e4763 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:08:04.631 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:04.631+0000 7fe4edd9b640 1 -- 172.21.15.104:0/525046268 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe4d80b0070 con 0x7fe4f014b090 2024-03-28T14:08:04.631 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:04.631+0000 7fe4edd9b640 1 -- 172.21.15.104:0/525046268 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe4d80ab400 con 0x7fe4f014b090 2024-03-28T14:08:04.632 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:04.631+0000 7fe4edd9b640 1 -- 172.21.15.104:0/525046268 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe4e8001bd0 con 0x7fe4f014b090 2024-03-28T14:08:04.632 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:04.631+0000 7fe4f4e96640 1 -- 172.21.15.104:0/525046268 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe4f014b090 msgr2=0x7fe4f014b540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:04.632 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:04.631+0000 7fe4f4e96640 1 --2- 172.21.15.104:0/525046268 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe4f014b090 0x7fe4f014b540 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fe4d80097c0 tx=0x7fe4d801c2a0 comp rx=0 tx=0).stop 2024-03-28T14:08:04.632 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:04.631+0000 7fe4f4e96640 1 -- 172.21.15.104:0/525046268 shutdown_connections 2024-03-28T14:08:04.633 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:04.631+0000 7fe4f4e96640 1 --2- 172.21.15.104:0/525046268 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe4f014b090 0x7fe4f014b540 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:04.633 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:04.631+0000 7fe4f4e96640 1 -- 172.21.15.104:0/525046268 >> 172.21.15.104:0/525046268 conn(0x7fe4f006cd70 msgr2=0x7fe4f00ba000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T14:08:04.633 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:04.631+0000 7fe4f4e96640 1 -- 172.21.15.104:0/525046268 shutdown_connections 2024-03-28T14:08:04.633 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:04.631+0000 7fe4f4e96640 1 -- 172.21.15.104:0/525046268 wait complete. 2024-03-28T14:08:04.635 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:04.631+0000 7fe4f4e96640 1 Processor -- start 2024-03-28T14:08:04.636 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:04.631+0000 7fe4f4e96640 1 -- start start 2024-03-28T14:08:04.636 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:04.631+0000 7fe4f4e96640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fe4f00ba970 0x7fe4f00b3ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:04.636 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:04.631+0000 7fe4f4e96640 1 --2- >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fe4f014b090 0x7fe4f00715a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:04.636 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:04.631+0000 7fe4f4e96640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe4f014ba80 0x7fe4f00b13e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:04.636 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:04.631+0000 7fe4f4e96640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7fe4f00af540 con 0x7fe4f014ba80 2024-03-28T14:08:04.636 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:04.631+0000 7fe4f4e96640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7fe4f00af3c0 con 0x7fe4f00ba970 2024-03-28T14:08:04.636 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:04.631+0000 7fe4f4e96640 1 -- --> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] -- mon_getmap magic: 0 -- 0x7fe4f009f470 con 0x7fe4f014b090 2024-03-28T14:08:04.636 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:04.631+0000 7fe4eed9d640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe4f014ba80 0x7fe4f00b13e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:04.636 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:04.631+0000 7fe4eed9d640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe4f014ba80 0x7fe4f00b13e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:42384/0 (socket says 172.21.15.104:42384) 2024-03-28T14:08:04.636 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:04.631+0000 7fe4eed9d640 1 -- 172.21.15.104:0/4221540182 learned_addr learned my addr 172.21.15.104:0/4221540182 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-03-28T14:08:04.637 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:04.631+0000 7fe4ee59c640 1 --2- 172.21.15.104:0/4221540182 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fe4f00ba970 0x7fe4f00b3ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:04.637 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:04.631+0000 7fe4e5d9b640 1 --2- 172.21.15.104:0/4221540182 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fe4f014b090 0x7fe4f00715a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:04.637 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:04.631+0000 7fe4eed9d640 1 -- 172.21.15.104:0/4221540182 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fe4f014b090 msgr2=0x7fe4f00715a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:04.637 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:04.631+0000 7fe4eed9d640 1 --2- 172.21.15.104:0/4221540182 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fe4f014b090 0x7fe4f00715a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:04.637 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:04.631+0000 7fe4eed9d640 1 -- 172.21.15.104:0/4221540182 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fe4f00ba970 msgr2=0x7fe4f00b3ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:04.637 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:04.631+0000 7fe4eed9d640 1 --2- 172.21.15.104:0/4221540182 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fe4f00ba970 0x7fe4f00b3ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:04.637 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:04.631+0000 7fe4eed9d640 1 -- 172.21.15.104:0/4221540182 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe4f00b2ff0 con 0x7fe4f014ba80 2024-03-28T14:08:04.637 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:04.631+0000 7fe4e5d9b640 1 --2- 172.21.15.104:0/4221540182 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fe4f014b090 0x7fe4f00715a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T14:08:04.637 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:04.631+0000 7fe4eed9d640 1 --2- 172.21.15.104:0/4221540182 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe4f014ba80 0x7fe4f00b13e0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fe4e800cc00 tx=0x7fe4e800a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:08:04.637 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:04.631+0000 7fe4e77fe640 1 -- 172.21.15.104:0/4221540182 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe4e800ce10 con 0x7fe4f014ba80 2024-03-28T14:08:04.637 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:04.631+0000 7fe4e77fe640 1 -- 172.21.15.104:0/4221540182 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe4e8012400 con 0x7fe4f014ba80 2024-03-28T14:08:04.637 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:04.631+0000 7fe4f4e96640 1 -- 172.21.15.104:0/4221540182 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe4f00b32d0 con 0x7fe4f014ba80 2024-03-28T14:08:04.637 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:04.631+0000 7fe4e77fe640 1 -- 172.21.15.104:0/4221540182 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe4e80127f0 con 0x7fe4f014ba80 2024-03-28T14:08:04.638 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:04.631+0000 7fe4f4e96640 1 -- 172.21.15.104:0/4221540182 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe4f0072140 con 0x7fe4f014ba80 2024-03-28T14:08:04.638 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:04.635+0000 7fe4f4e96640 1 -- 172.21.15.104:0/4221540182 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe4f00687c0 con 0x7fe4f014ba80 2024-03-28T14:08:04.639 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:04.635+0000 7fe4e77fe640 1 -- 172.21.15.104:0/4221540182 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) ==== 87803+0+0 (secure 0 0 0) 0x7fe4e8012990 con 0x7fe4f014ba80 2024-03-28T14:08:04.639 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:04.635+0000 7fe4e77fe640 1 --2- 172.21.15.104:0/4221540182 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7fe4d0067fc0 0x7fe4d006a480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:04.639 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:04.635+0000 7fe4e77fe640 1 -- 172.21.15.104:0/4221540182 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7fe4e8086d60 con 0x7fe4f014ba80 2024-03-28T14:08:04.639 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:04.635+0000 7fe4ee59c640 1 --2- 172.21.15.104:0/4221540182 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7fe4d0067fc0 0x7fe4d006a480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:04.639 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:04.635+0000 7fe4ee59c640 1 --2- 172.21.15.104:0/4221540182 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7fe4d0067fc0 0x7fe4d006a480 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fe4e0000f70 tx=0x7fe4e0009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:08:04.641 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:04.639+0000 7fe4e77fe640 1 -- 172.21.15.104:0/4221540182 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179117 (secure 0 0 0) 0x7fe4f006ea00 con 0x7fe4f014ba80 2024-03-28T14:08:04.794 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:04.791+0000 7fe4f4e96640 1 -- 172.21.15.104:0/4221540182 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fe4f0145a60 con 0x7fe4f014ba80 2024-03-28T14:08:04.795 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:04.791+0000 7fe4e77fe640 1 -- 172.21.15.104:0/4221540182 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) ==== 74+0+16629 (secure 0 0 0) 0x7fe4e8057500 con 0x7fe4f014ba80 2024-03-28T14:08:04.795 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-03-28T14:08:04.795 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":21,"fsid":"428a90f2-3351-435f-8fcf-f7e66c78a13e","created":"2024-03-28T14:07:34.862458+0000","modified":"2024-03-28T14:08:00.671587+0000","last_up_change":"2024-03-28T14:07:46.028122+0000","last_in_change":"2024-03-28T14:07:37.157486+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-28T14:07:47.622988+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-28T14:07:57.147450+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-28T14:07:57.990975+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"1cb70540-4e42-402f-a446-aaec4ca83007","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6800","nonce":3745690918},{"type":"v1","addr":"172.21.15.104:6801","nonce":3745690918}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":3745690918},{"type":"v1","addr":"172.21.15.104:6803","nonce":3745690918}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":3745690918},{"type":"v1","addr":"172.21.15.104:6807","nonce":3745690918}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":3745690918},{"type":"v1","addr":"172.21.15.104:6805","nonce":3745690918}]},"public_addr":"172.21.15.104:6801/3745690918","cluster_addr":"172.21.15.104:6803/3745690918","heartbeat_back_addr":"172.21.15.104:6807/3745690918","heartbeat_front_addr":"172.21.15.104:6805/3745690918","state":["exists","up"]},{"osd":1,"uuid":"3ab7afdd-b980-418c-81f2-004e16de3dca","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":19438674},{"type":"v1","addr":"172.21.15.104:6810","nonce":19438674}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":19438674},{"type":"v1","addr":"172.21.15.104:6814","nonce":19438674}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":19438674},{"type":"v1","addr":"172.21.15.104:6823","nonce":19438674}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":19438674},{"type":"v1","addr":"172.21.15.104:6819","nonce":19438674}]},"public_addr":"172.21.15.104:6810/19438674","cluster_addr":"172.21.15.104:6814/19438674","heartbeat_back_addr":"172.21.15.104:6823/19438674","heartbeat_front_addr":"172.21.15.104:6819/19438674","state":["exists","up"]},{"osd":2,"uuid":"3e2651c8-ecae-4a9e-a053-27240a79eac2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":2352012626},{"type":"v1","addr":"172.21.15.104:6825","nonce":2352012626}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6826","nonce":2352012626},{"type":"v1","addr":"172.21.15.104:6827","nonce":2352012626}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":2352012626},{"type":"v1","addr":"172.21.15.104:6831","nonce":2352012626}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6828","nonce":2352012626},{"type":"v1","addr":"172.21.15.104:6829","nonce":2352012626}]},"public_addr":"172.21.15.104:6825/2352012626","cluster_addr":"172.21.15.104:6827/2352012626","heartbeat_back_addr":"172.21.15.104:6831/2352012626","heartbeat_front_addr":"172.21.15.104:6829/2352012626","state":["exists","up"]},{"osd":3,"uuid":"925c7e0a-38ef-47cd-b288-8bec8456fb17","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6809","nonce":3568347381},{"type":"v1","addr":"172.21.15.104:6811","nonce":3568347381}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6813","nonce":3568347381},{"type":"v1","addr":"172.21.15.104:6815","nonce":3568347381}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":3568347381},{"type":"v1","addr":"172.21.15.104:6822","nonce":3568347381}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6817","nonce":3568347381},{"type":"v1","addr":"172.21.15.104:6818","nonce":3568347381}]},"public_addr":"172.21.15.104:6811/3568347381","cluster_addr":"172.21.15.104:6815/3568347381","heartbeat_back_addr":"172.21.15.104:6822/3568347381","heartbeat_front_addr":"172.21.15.104:6818/3568347381","state":["exists","up"]},{"osd":4,"uuid":"4a8fac86-e595-4a0e-96e3-57edb3cc3f57","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6800","nonce":2107001077},{"type":"v1","addr":"172.21.15.171:6801","nonce":2107001077}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6802","nonce":2107001077},{"type":"v1","addr":"172.21.15.171:6803","nonce":2107001077}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6806","nonce":2107001077},{"type":"v1","addr":"172.21.15.171:6807","nonce":2107001077}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6804","nonce":2107001077},{"type":"v1","addr":"172.21.15.171:6805","nonce":2107001077}]},"public_addr":"172.21.15.171:6801/2107001077","cluster_addr":"172.21.15.171:6803/2107001077","heartbeat_back_addr":"172.21.15.171:6807/2107001077","heartbeat_front_addr":"172.21.15.171:6805/2107001077","state":["exists","up"]},{"osd":5,"uuid":"e48d50b0-beae-40ae-b15a-9a84dbfe88ab","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6808","nonce":2934826925},{"type":"v1","addr":"172.21.15.171:6810","nonce":2934826925}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6812","nonce":2934826925},{"type":"v1","addr":"172.21.15.171:6814","nonce":2934826925}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6819","nonce":2934826925},{"type":"v1","addr":"172.21.15.171:6821","nonce":2934826925}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6816","nonce":2934826925},{"type":"v1","addr":"172.21.15.171:6817","nonce":2934826925}]},"public_addr":"172.21.15.171:6810/2934826925","cluster_addr":"172.21.15.171:6814/2934826925","heartbeat_back_addr":"172.21.15.171:6821/2934826925","heartbeat_front_addr":"172.21.15.171:6817/2934826925","state":["exists","up"]},{"osd":6,"uuid":"5e7c3292-9936-4f35-b267-aed9d4e49323","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6824","nonce":436742919},{"type":"v1","addr":"172.21.15.171:6825","nonce":436742919}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6826","nonce":436742919},{"type":"v1","addr":"172.21.15.171:6827","nonce":436742919}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6830","nonce":436742919},{"type":"v1","addr":"172.21.15.171:6831","nonce":436742919}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6828","nonce":436742919},{"type":"v1","addr":"172.21.15.171:6829","nonce":436742919}]},"public_addr":"172.21.15.171:6825/436742919","cluster_addr":"172.21.15.171:6827/436742919","heartbeat_back_addr":"172.21.15.171:6831/436742919","heartbeat_front_addr":"172.21.15.171:6829/436742919","state":["exists","up"]},{"osd":7,"uuid":"339b8414-d7f0-4fb0-a446-efaf635cb7ff","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6809","nonce":2055631175},{"type":"v1","addr":"172.21.15.171:6811","nonce":2055631175}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6813","nonce":2055631175},{"type":"v1","addr":"172.21.15.171:6815","nonce":2055631175}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6822","nonce":2055631175},{"type":"v1","addr":"172.21.15.171:6823","nonce":2055631175}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6818","nonce":2055631175},{"type":"v1","addr":"172.21.15.171:6820","nonce":2055631175}]},"public_addr":"172.21.15.171:6811/2055631175","cluster_addr":"172.21.15.171:6815/2055631175","heartbeat_back_addr":"172.21.15.171:6823/2055631175","heartbeat_front_addr":"172.21.15.171:6820/2055631175","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T14:07:42.668881+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T14:07:42.708397+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T14:07:42.634191+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T14:07:42.733662+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T14:07:42.651541+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T14:07:42.783181+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T14:07:42.766310+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T14:07:42.670797+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-28T14:08:04.796 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:04.795+0000 7fe4f4e96640 1 -- 172.21.15.104:0/4221540182 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7fe4d0067fc0 msgr2=0x7fe4d006a480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:04.796 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:04.795+0000 7fe4f4e96640 1 --2- 172.21.15.104:0/4221540182 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7fe4d0067fc0 0x7fe4d006a480 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fe4e0000f70 tx=0x7fe4e0009040 comp rx=0 tx=0).stop 2024-03-28T14:08:04.797 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:04.795+0000 7fe4f4e96640 1 -- 172.21.15.104:0/4221540182 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe4f014ba80 msgr2=0x7fe4f00b13e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:04.797 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:04.795+0000 7fe4f4e96640 1 --2- 172.21.15.104:0/4221540182 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe4f014ba80 0x7fe4f00b13e0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fe4e800cc00 tx=0x7fe4e800a430 comp rx=0 tx=0).stop 2024-03-28T14:08:04.797 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:04.795+0000 7fe4f4e96640 1 -- 172.21.15.104:0/4221540182 shutdown_connections 2024-03-28T14:08:04.797 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:04.795+0000 7fe4f4e96640 1 --2- 172.21.15.104:0/4221540182 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7fe4d0067fc0 0x7fe4d006a480 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:04.797 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:04.795+0000 7fe4f4e96640 1 --2- 172.21.15.104:0/4221540182 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe4f014ba80 0x7fe4f00b13e0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:04.797 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:04.795+0000 7fe4f4e96640 1 --2- 172.21.15.104:0/4221540182 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fe4f014b090 0x7fe4f00715a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:04.797 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:04.795+0000 7fe4f4e96640 1 --2- 172.21.15.104:0/4221540182 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fe4f00ba970 0x7fe4f00b3ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:04.797 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:04.795+0000 7fe4f4e96640 1 -- 172.21.15.104:0/4221540182 >> 172.21.15.104:0/4221540182 conn(0x7fe4f006cd70 msgr2=0x7fe4f00b0190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T14:08:04.797 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:04.795+0000 7fe4f4e96640 1 -- 172.21.15.104:0/4221540182 shutdown_connections 2024-03-28T14:08:04.797 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:04.795+0000 7fe4f4e96640 1 -- 172.21.15.104:0/4221540182 wait complete. 2024-03-28T14:08:04.812 INFO:tasks.ceph.ceph_manager.ceph:all up! 2024-03-28T14:08:04.813 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-28T14:08:04.923 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:04.919+0000 7fd07a309640 1 Processor -- start 2024-03-28T14:08:04.923 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:04.919+0000 7fd07a309640 1 -- start start 2024-03-28T14:08:04.925 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:04.923+0000 7fd07a309640 1 --2- >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fd07402c460 0x7fd0740b3ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:04.926 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:04.923+0000 7fd07a309640 1 --2- >> v2:172.21.15.171:3300/0 conn(0x7fd07413fc50 0x7fd0741456c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:04.926 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:04.923+0000 7fd07a309640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd07402c840 con 0x7fd0740b4530 2024-03-28T14:08:04.926 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:04.923+0000 7fd07a309640 1 -- --> v2:172.21.15.171:3300/0 -- mon_getmap magic: 0 -- 0x7fd0740b4f20 con 0x7fd07413fc50 2024-03-28T14:08:04.926 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:04.923+0000 7fd07a309640 1 -- --> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] -- mon_getmap magic: 0 -- 0x7fd0740b5200 con 0x7fd07402c460 2024-03-28T14:08:04.926 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:04.923+0000 7fd073fff640 1 --2- >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fd07402c460 0x7fd0740b3ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:04.926 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:04.923+0000 7fd0788a6640 1 --2- >> v2:172.21.15.171:3300/0 conn(0x7fd07413fc50 0x7fd0741456c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:04.926 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:04.923+0000 7fd073fff640 1 --2- >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fd07402c460 0x7fd0740b3ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.171:3301/0 says I am v2:172.21.15.104:55880/0 (socket says 172.21.15.104:55880) 2024-03-28T14:08:04.926 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:04.923+0000 7fd073fff640 1 -- 172.21.15.104:0/1098039491 learned_addr learned my addr 172.21.15.104:0/1098039491 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-03-28T14:08:04.926 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:04.923+0000 7fd072ffd640 1 -- 172.21.15.104:0/1098039491 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1698492053 0 0) 0x7fd07402c840 con 0x7fd0740b4530 2024-03-28T14:08:04.926 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:04.923+0000 7fd072ffd640 1 -- 172.21.15.104:0/1098039491 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd05c003530 con 0x7fd0740b4530 2024-03-28T14:08:04.927 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:04.923+0000 7fd073fff640 1 -- 172.21.15.104:0/1098039491 >> v2:172.21.15.171:3300/0 conn(0x7fd07413fc50 msgr2=0x7fd0741456c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T14:08:04.927 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:04.923+0000 7fd073fff640 1 --2- 172.21.15.104:0/1098039491 >> v2:172.21.15.171:3300/0 conn(0x7fd07413fc50 0x7fd0741456c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:04.927 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:04.923+0000 7fd073fff640 1 -- 172.21.15.104:0/1098039491 >> v1:172.21.15.171:6789/0 conn(0x7fd0740b4530 legacy=0x7fd07413f540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:04.927 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:04.923+0000 7fd073fff640 1 -- 172.21.15.104:0/1098039491 --> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd0740b54e0 con 0x7fd07402c460 2024-03-28T14:08:04.928 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:04.927+0000 7fd073fff640 1 --2- 172.21.15.104:0/1098039491 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fd07402c460 0x7fd0740b3ff0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fd0640098a0 tx=0x7fd06401c4b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=199beaa3b0241b54 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:08:04.928 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:04.927+0000 7fd072ffd640 1 -- 172.21.15.104:0/1098039491 <== mon.2 v2:172.21.15.171:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0640ab5f0 con 0x7fd07402c460 2024-03-28T14:08:04.928 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:04.927+0000 7fd072ffd640 1 -- 172.21.15.104:0/1098039491 <== mon.2 v2:172.21.15.171:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd0640b0070 con 0x7fd07402c460 2024-03-28T14:08:04.928 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:04.927+0000 7fd072ffd640 1 -- 172.21.15.104:0/1098039491 <== mon.2 v2:172.21.15.171:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0640ab990 con 0x7fd07402c460 2024-03-28T14:08:04.929 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:04.927+0000 7fd07a309640 1 -- 172.21.15.104:0/1098039491 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fd07402c460 msgr2=0x7fd0740b3ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:04.929 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:04.927+0000 7fd07a309640 1 --2- 172.21.15.104:0/1098039491 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fd07402c460 0x7fd0740b3ff0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fd0640098a0 tx=0x7fd06401c4b0 comp rx=0 tx=0).stop 2024-03-28T14:08:04.929 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:04.927+0000 7fd07a309640 1 -- 172.21.15.104:0/1098039491 shutdown_connections 2024-03-28T14:08:04.929 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:04.927+0000 7fd07a309640 1 --2- 172.21.15.104:0/1098039491 >> v2:172.21.15.171:3300/0 conn(0x7fd07413fc50 0x7fd0741456c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:04.929 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:04.927+0000 7fd07a309640 1 --2- 172.21.15.104:0/1098039491 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fd07402c460 0x7fd0740b3ff0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:04.929 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:04.927+0000 7fd07a309640 1 -- 172.21.15.104:0/1098039491 >> 172.21.15.104:0/1098039491 conn(0x7fd0740b95f0 msgr2=0x7fd0740b99d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T14:08:04.929 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:04.927+0000 7fd07a309640 1 -- 172.21.15.104:0/1098039491 shutdown_connections 2024-03-28T14:08:04.929 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:04.927+0000 7fd07a309640 1 -- 172.21.15.104:0/1098039491 wait complete. 2024-03-28T14:08:04.930 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:04.927+0000 7fd07a309640 1 Processor -- start 2024-03-28T14:08:04.930 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:04.927+0000 7fd07a309640 1 -- start start 2024-03-28T14:08:04.932 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:04.927+0000 7fd07a309640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fd07402c460 0x7fd074071f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:04.933 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:04.927+0000 7fd07a309640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd0740b4530 0x7fd074072880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:04.933 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:04.927+0000 7fd07a309640 1 --2- >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fd07413fc50 0x7fd0740b1f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:04.933 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:04.927+0000 7fd07a309640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7fd074152b30 con 0x7fd0740b4530 2024-03-28T14:08:04.933 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:04.927+0000 7fd07a309640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7fd0741529b0 con 0x7fd07402c460 2024-03-28T14:08:04.933 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:04.927+0000 7fd07a309640 1 -- --> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] -- mon_getmap magic: 0 -- 0x7fd074152cb0 con 0x7fd07413fc50 2024-03-28T14:08:04.933 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:04.931+0000 7fd0737fe640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd0740b4530 0x7fd074072880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:04.933 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:04.931+0000 7fd0737fe640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd0740b4530 0x7fd074072880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:42396/0 (socket says 172.21.15.104:42396) 2024-03-28T14:08:04.933 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:04.931+0000 7fd0737fe640 1 -- 172.21.15.104:0/4013034141 learned_addr learned my addr 172.21.15.104:0/4013034141 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-03-28T14:08:04.933 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:04.931+0000 7fd073fff640 1 --2- 172.21.15.104:0/4013034141 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fd07402c460 0x7fd074071f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:04.933 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:04.931+0000 7fd0788a6640 1 --2- 172.21.15.104:0/4013034141 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fd07413fc50 0x7fd0740b1f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:04.933 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:04.931+0000 7fd0737fe640 1 -- 172.21.15.104:0/4013034141 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fd07413fc50 msgr2=0x7fd0740b1f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:04.934 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:04.931+0000 7fd0737fe640 1 --2- 172.21.15.104:0/4013034141 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fd07413fc50 0x7fd0740b1f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:04.934 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:04.931+0000 7fd0737fe640 1 -- 172.21.15.104:0/4013034141 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fd07402c460 msgr2=0x7fd074071f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:04.934 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:04.931+0000 7fd0737fe640 1 --2- 172.21.15.104:0/4013034141 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fd07402c460 0x7fd074071f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:04.934 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:04.931+0000 7fd0737fe640 1 -- 172.21.15.104:0/4013034141 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd074072fb0 con 0x7fd0740b4530 2024-03-28T14:08:04.934 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:04.931+0000 7fd0737fe640 1 --2- 172.21.15.104:0/4013034141 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd0740b4530 0x7fd074072880 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fd06800c440 tx=0x7fd06800c900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:08:04.934 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:04.931+0000 7fd070ff9640 1 -- 172.21.15.104:0/4013034141 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0680052d0 con 0x7fd0740b4530 2024-03-28T14:08:04.934 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:04.931+0000 7fd070ff9640 1 -- 172.21.15.104:0/4013034141 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd068005470 con 0x7fd0740b4530 2024-03-28T14:08:04.934 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:04.931+0000 7fd070ff9640 1 -- 172.21.15.104:0/4013034141 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd068012870 con 0x7fd0740b4530 2024-03-28T14:08:04.934 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:04.931+0000 7fd07a309640 1 -- 172.21.15.104:0/4013034141 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd074073220 con 0x7fd0740b4530 2024-03-28T14:08:04.934 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:04.931+0000 7fd07a309640 1 -- 172.21.15.104:0/4013034141 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd074070d00 con 0x7fd0740b4530 2024-03-28T14:08:04.935 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:04.931+0000 7fd07a309640 1 -- 172.21.15.104:0/4013034141 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd0740687c0 con 0x7fd0740b4530 2024-03-28T14:08:04.935 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:04.931+0000 7fd070ff9640 1 -- 172.21.15.104:0/4013034141 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) ==== 87803+0+0 (secure 0 0 0) 0x7fd068012a10 con 0x7fd0740b4530 2024-03-28T14:08:04.936 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:04.931+0000 7fd070ff9640 1 --2- 172.21.15.104:0/4013034141 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7fd060067fc0 0x7fd06006a480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:04.936 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:04.931+0000 7fd070ff9640 1 -- 172.21.15.104:0/4013034141 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7fd0680860c0 con 0x7fd0740b4530 2024-03-28T14:08:04.936 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:04.931+0000 7fd073fff640 1 --2- 172.21.15.104:0/4013034141 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7fd060067fc0 0x7fd06006a480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:04.936 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:04.935+0000 7fd073fff640 1 --2- 172.21.15.104:0/4013034141 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7fd060067fc0 0x7fd06006a480 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fd0640af000 tx=0x7fd0640a7000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:08:04.938 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:04.935+0000 7fd070ff9640 1 -- 172.21.15.104:0/4013034141 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179117 (secure 0 0 0) 0x7fd068056900 con 0x7fd0740b4530 2024-03-28T14:08:05.098 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.095+0000 7fd07a309640 1 -- 172.21.15.104:0/4013034141 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fd07406dcc0 con 0x7fd0740b4530 2024-03-28T14:08:05.100 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.099+0000 7fd070ff9640 1 -- 172.21.15.104:0/4013034141 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) ==== 74+0+16629 (secure 0 0 0) 0x7fd068056200 con 0x7fd0740b4530 2024-03-28T14:08:05.100 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-03-28T14:08:05.100 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":21,"fsid":"428a90f2-3351-435f-8fcf-f7e66c78a13e","created":"2024-03-28T14:07:34.862458+0000","modified":"2024-03-28T14:08:00.671587+0000","last_up_change":"2024-03-28T14:07:46.028122+0000","last_in_change":"2024-03-28T14:07:37.157486+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-28T14:07:47.622988+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-28T14:07:57.147450+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-28T14:07:57.990975+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"1cb70540-4e42-402f-a446-aaec4ca83007","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6800","nonce":3745690918},{"type":"v1","addr":"172.21.15.104:6801","nonce":3745690918}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":3745690918},{"type":"v1","addr":"172.21.15.104:6803","nonce":3745690918}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":3745690918},{"type":"v1","addr":"172.21.15.104:6807","nonce":3745690918}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":3745690918},{"type":"v1","addr":"172.21.15.104:6805","nonce":3745690918}]},"public_addr":"172.21.15.104:6801/3745690918","cluster_addr":"172.21.15.104:6803/3745690918","heartbeat_back_addr":"172.21.15.104:6807/3745690918","heartbeat_front_addr":"172.21.15.104:6805/3745690918","state":["exists","up"]},{"osd":1,"uuid":"3ab7afdd-b980-418c-81f2-004e16de3dca","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":19438674},{"type":"v1","addr":"172.21.15.104:6810","nonce":19438674}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":19438674},{"type":"v1","addr":"172.21.15.104:6814","nonce":19438674}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":19438674},{"type":"v1","addr":"172.21.15.104:6823","nonce":19438674}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":19438674},{"type":"v1","addr":"172.21.15.104:6819","nonce":19438674}]},"public_addr":"172.21.15.104:6810/19438674","cluster_addr":"172.21.15.104:6814/19438674","heartbeat_back_addr":"172.21.15.104:6823/19438674","heartbeat_front_addr":"172.21.15.104:6819/19438674","state":["exists","up"]},{"osd":2,"uuid":"3e2651c8-ecae-4a9e-a053-27240a79eac2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":2352012626},{"type":"v1","addr":"172.21.15.104:6825","nonce":2352012626}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6826","nonce":2352012626},{"type":"v1","addr":"172.21.15.104:6827","nonce":2352012626}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":2352012626},{"type":"v1","addr":"172.21.15.104:6831","nonce":2352012626}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6828","nonce":2352012626},{"type":"v1","addr":"172.21.15.104:6829","nonce":2352012626}]},"public_addr":"172.21.15.104:6825/2352012626","cluster_addr":"172.21.15.104:6827/2352012626","heartbeat_back_addr":"172.21.15.104:6831/2352012626","heartbeat_front_addr":"172.21.15.104:6829/2352012626","state":["exists","up"]},{"osd":3,"uuid":"925c7e0a-38ef-47cd-b288-8bec8456fb17","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6809","nonce":3568347381},{"type":"v1","addr":"172.21.15.104:6811","nonce":3568347381}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6813","nonce":3568347381},{"type":"v1","addr":"172.21.15.104:6815","nonce":3568347381}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":3568347381},{"type":"v1","addr":"172.21.15.104:6822","nonce":3568347381}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6817","nonce":3568347381},{"type":"v1","addr":"172.21.15.104:6818","nonce":3568347381}]},"public_addr":"172.21.15.104:6811/3568347381","cluster_addr":"172.21.15.104:6815/3568347381","heartbeat_back_addr":"172.21.15.104:6822/3568347381","heartbeat_front_addr":"172.21.15.104:6818/3568347381","state":["exists","up"]},{"osd":4,"uuid":"4a8fac86-e595-4a0e-96e3-57edb3cc3f57","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6800","nonce":2107001077},{"type":"v1","addr":"172.21.15.171:6801","nonce":2107001077}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6802","nonce":2107001077},{"type":"v1","addr":"172.21.15.171:6803","nonce":2107001077}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6806","nonce":2107001077},{"type":"v1","addr":"172.21.15.171:6807","nonce":2107001077}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6804","nonce":2107001077},{"type":"v1","addr":"172.21.15.171:6805","nonce":2107001077}]},"public_addr":"172.21.15.171:6801/2107001077","cluster_addr":"172.21.15.171:6803/2107001077","heartbeat_back_addr":"172.21.15.171:6807/2107001077","heartbeat_front_addr":"172.21.15.171:6805/2107001077","state":["exists","up"]},{"osd":5,"uuid":"e48d50b0-beae-40ae-b15a-9a84dbfe88ab","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6808","nonce":2934826925},{"type":"v1","addr":"172.21.15.171:6810","nonce":2934826925}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6812","nonce":2934826925},{"type":"v1","addr":"172.21.15.171:6814","nonce":2934826925}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6819","nonce":2934826925},{"type":"v1","addr":"172.21.15.171:6821","nonce":2934826925}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6816","nonce":2934826925},{"type":"v1","addr":"172.21.15.171:6817","nonce":2934826925}]},"public_addr":"172.21.15.171:6810/2934826925","cluster_addr":"172.21.15.171:6814/2934826925","heartbeat_back_addr":"172.21.15.171:6821/2934826925","heartbeat_front_addr":"172.21.15.171:6817/2934826925","state":["exists","up"]},{"osd":6,"uuid":"5e7c3292-9936-4f35-b267-aed9d4e49323","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6824","nonce":436742919},{"type":"v1","addr":"172.21.15.171:6825","nonce":436742919}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6826","nonce":436742919},{"type":"v1","addr":"172.21.15.171:6827","nonce":436742919}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6830","nonce":436742919},{"type":"v1","addr":"172.21.15.171:6831","nonce":436742919}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6828","nonce":436742919},{"type":"v1","addr":"172.21.15.171:6829","nonce":436742919}]},"public_addr":"172.21.15.171:6825/436742919","cluster_addr":"172.21.15.171:6827/436742919","heartbeat_back_addr":"172.21.15.171:6831/436742919","heartbeat_front_addr":"172.21.15.171:6829/436742919","state":["exists","up"]},{"osd":7,"uuid":"339b8414-d7f0-4fb0-a446-efaf635cb7ff","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6809","nonce":2055631175},{"type":"v1","addr":"172.21.15.171:6811","nonce":2055631175}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6813","nonce":2055631175},{"type":"v1","addr":"172.21.15.171:6815","nonce":2055631175}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6822","nonce":2055631175},{"type":"v1","addr":"172.21.15.171:6823","nonce":2055631175}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6818","nonce":2055631175},{"type":"v1","addr":"172.21.15.171:6820","nonce":2055631175}]},"public_addr":"172.21.15.171:6811/2055631175","cluster_addr":"172.21.15.171:6815/2055631175","heartbeat_back_addr":"172.21.15.171:6823/2055631175","heartbeat_front_addr":"172.21.15.171:6820/2055631175","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T14:07:42.668881+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T14:07:42.708397+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T14:07:42.634191+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T14:07:42.733662+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T14:07:42.651541+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T14:07:42.783181+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T14:07:42.766310+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T14:07:42.670797+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-28T14:08:05.102 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.099+0000 7fd07a309640 1 -- 172.21.15.104:0/4013034141 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7fd060067fc0 msgr2=0x7fd06006a480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:05.102 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.099+0000 7fd07a309640 1 --2- 172.21.15.104:0/4013034141 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7fd060067fc0 0x7fd06006a480 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fd0640af000 tx=0x7fd0640a7000 comp rx=0 tx=0).stop 2024-03-28T14:08:05.102 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.099+0000 7fd07a309640 1 -- 172.21.15.104:0/4013034141 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd0740b4530 msgr2=0x7fd074072880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:05.102 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.099+0000 7fd07a309640 1 --2- 172.21.15.104:0/4013034141 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd0740b4530 0x7fd074072880 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fd06800c440 tx=0x7fd06800c900 comp rx=0 tx=0).stop 2024-03-28T14:08:05.102 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.099+0000 7fd07a309640 1 -- 172.21.15.104:0/4013034141 shutdown_connections 2024-03-28T14:08:05.102 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.099+0000 7fd07a309640 1 --2- 172.21.15.104:0/4013034141 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fd07413fc50 0x7fd0740b1f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:05.102 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.099+0000 7fd07a309640 1 --2- 172.21.15.104:0/4013034141 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7fd060067fc0 0x7fd06006a480 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:05.102 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.099+0000 7fd07a309640 1 --2- 172.21.15.104:0/4013034141 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd0740b4530 0x7fd074072880 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:05.102 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.099+0000 7fd07a309640 1 --2- 172.21.15.104:0/4013034141 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fd07402c460 0x7fd074071f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:05.102 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.099+0000 7fd07a309640 1 -- 172.21.15.104:0/4013034141 >> 172.21.15.104:0/4013034141 conn(0x7fd0740b95f0 msgr2=0x7fd07413d330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T14:08:05.103 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.099+0000 7fd07a309640 1 -- 172.21.15.104:0/4013034141 shutdown_connections 2024-03-28T14:08:05.103 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.099+0000 7fd07a309640 1 -- 172.21.15.104:0/4013034141 wait complete. 2024-03-28T14:08:05.124 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2024-03-28T14:08:05.125 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2024-03-28T14:08:05.125 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2024-03-28T14:08:05.126 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2024-03-28T14:08:05.126 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 flush_pg_stats 2024-03-28T14:08:05.127 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 flush_pg_stats 2024-03-28T14:08:05.127 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 flush_pg_stats 2024-03-28T14:08:05.127 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 flush_pg_stats 2024-03-28T14:08:05.307 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.303+0000 7f4d9726f640 1 Processor -- start 2024-03-28T14:08:05.307 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.303+0000 7f4d9726f640 1 -- start start 2024-03-28T14:08:05.307 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.303+0000 7f4d9726f640 1 --2- >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f4d9006de20 0x7f4d90144730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:05.307 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.303+0000 7f4d9726f640 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f4d900733f0 0x7f4d900b02f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:05.307 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.303+0000 7f4d9726f640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4d9006e200 con 0x7f4d900b6260 2024-03-28T14:08:05.307 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.303+0000 7f4d9726f640 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 -- 0x7f4d900b0830 con 0x7f4d900733f0 2024-03-28T14:08:05.307 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.303+0000 7f4d9726f640 1 -- --> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] -- mon_getmap magic: 0 -- 0x7f4d900af820 con 0x7f4d9006de20 2024-03-28T14:08:05.308 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.303+0000 7f4d9580c640 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f4d900b6260 0x7f4d9009c3c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:47310/0 (socket says 172.21.15.104:47310) 2024-03-28T14:08:05.308 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.303+0000 7f4d9580c640 1 -- 172.21.15.104:0/423853276 learned_addr learned my addr 172.21.15.104:0/423853276 (peer_addr_for_me v1:172.21.15.104:0/0) 2024-03-28T14:08:05.308 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.303+0000 7f4d9500b640 1 --2- 172.21.15.104:0/423853276 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f4d9006de20 0x7f4d90144730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:05.308 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.303+0000 7f4d9480a640 1 --2- 172.21.15.104:0/423853276 >> v2:172.21.15.104:3300/0 conn(0x7f4d900733f0 0x7f4d900b02f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:05.308 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.307+0000 7f4d9500b640 1 -- 172.21.15.104:0/423853276 >> v2:172.21.15.104:3300/0 conn(0x7f4d900733f0 msgr2=0x7f4d900b02f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T14:08:05.308 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.307+0000 7f4d9500b640 1 --2- 172.21.15.104:0/423853276 >> v2:172.21.15.104:3300/0 conn(0x7f4d900733f0 0x7f4d900b02f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:05.309 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.307+0000 7f4d9500b640 1 -- 172.21.15.104:0/423853276 >> v1:172.21.15.104:6789/0 conn(0x7f4d900b6260 legacy=0x7f4d9009c3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:05.309 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.307+0000 7f4d9500b640 1 -- 172.21.15.104:0/423853276 --> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4d900af2e0 con 0x7f4d9006de20 2024-03-28T14:08:05.309 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.307+0000 7f4d9480a640 1 --2- 172.21.15.104:0/423853276 >> v2:172.21.15.104:3300/0 conn(0x7f4d900733f0 0x7f4d900b02f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T14:08:05.309 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.307+0000 7f4d9500b640 1 --2- 172.21.15.104:0/423853276 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f4d9006de20 0x7f4d90144730 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f4d8000ba60 tx=0x7f4d8001e970 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ee12e4c17c88eac6 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:08:05.310 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.307+0000 7f4d87fff640 1 -- 172.21.15.104:0/423853276 <== mon.2 v2:172.21.15.171:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d800b4070 con 0x7f4d9006de20 2024-03-28T14:08:05.310 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.307+0000 7f4d87fff640 1 -- 172.21.15.104:0/423853276 <== mon.2 v2:172.21.15.171:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4d800a79a0 con 0x7f4d9006de20 2024-03-28T14:08:05.311 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.307+0000 7f4d9726f640 1 -- 172.21.15.104:0/423853276 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f4d9006de20 msgr2=0x7f4d90144730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:05.311 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.307+0000 7f4d9726f640 1 --2- 172.21.15.104:0/423853276 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f4d9006de20 0x7f4d90144730 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f4d8000ba60 tx=0x7f4d8001e970 comp rx=0 tx=0).stop 2024-03-28T14:08:05.311 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.311+0000 7f4d9726f640 1 -- 172.21.15.104:0/423853276 shutdown_connections 2024-03-28T14:08:05.311 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.311+0000 7f4d9726f640 1 --2- 172.21.15.104:0/423853276 >> v2:172.21.15.104:3300/0 conn(0x7f4d900733f0 0x7f4d900b02f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:05.312 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.311+0000 7f4d9726f640 1 --2- 172.21.15.104:0/423853276 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f4d9006de20 0x7f4d90144730 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:05.312 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.311+0000 7f4d9726f640 1 -- 172.21.15.104:0/423853276 >> 172.21.15.104:0/423853276 conn(0x7f4d900ba390 msgr2=0x7f4d900ba7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T14:08:05.312 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.311+0000 7f4d9726f640 1 -- 172.21.15.104:0/423853276 shutdown_connections 2024-03-28T14:08:05.312 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.311+0000 7f4d9726f640 1 -- 172.21.15.104:0/423853276 wait complete. 2024-03-28T14:08:05.313 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.311+0000 7f4d9726f640 1 Processor -- start 2024-03-28T14:08:05.313 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.311+0000 7f4d9726f640 1 -- start start 2024-03-28T14:08:05.315 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.311+0000 7f4d9726f640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f4d9006de20 0x7f4d9009f300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:05.315 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.311+0000 7f4d9726f640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4d900733f0 0x7f4d900b2850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:05.315 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.311+0000 7f4d9726f640 1 --2- >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f4d900b6260 0x7f4d9013d170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:05.315 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.311+0000 7f4d9726f640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f4d9009cad0 con 0x7f4d900733f0 2024-03-28T14:08:05.315 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.311+0000 7f4d9726f640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f4d900afe60 con 0x7f4d9006de20 2024-03-28T14:08:05.316 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.311+0000 7f4d9726f640 1 -- --> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] -- mon_getmap magic: 0 -- 0x7f4d9009cc50 con 0x7f4d900b6260 2024-03-28T14:08:05.316 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.311+0000 7f4d9480a640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4d900733f0 0x7f4d900b2850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:05.316 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.311+0000 7f4d9480a640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4d900733f0 0x7f4d900b2850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:42420/0 (socket says 172.21.15.104:42420) 2024-03-28T14:08:05.316 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.311+0000 7f4d9480a640 1 -- 172.21.15.104:0/2140275844 learned_addr learned my addr 172.21.15.104:0/2140275844 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-03-28T14:08:05.316 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.315+0000 7f4d9500b640 1 --2- 172.21.15.104:0/2140275844 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f4d9006de20 0x7f4d9009f300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:05.316 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.315+0000 7f4d9580c640 1 --2- 172.21.15.104:0/2140275844 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f4d900b6260 0x7f4d9013d170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:05.317 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.315+0000 7f4d9480a640 1 -- 172.21.15.104:0/2140275844 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f4d900b6260 msgr2=0x7f4d9013d170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:05.317 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.315+0000 7f4d9480a640 1 --2- 172.21.15.104:0/2140275844 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f4d900b6260 0x7f4d9013d170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:05.317 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.315+0000 7f4d9480a640 1 -- 172.21.15.104:0/2140275844 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f4d9006de20 msgr2=0x7f4d9009f300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:05.317 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.315+0000 7f4d9480a640 1 --2- 172.21.15.104:0/2140275844 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f4d9006de20 0x7f4d9009f300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:05.317 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.315+0000 7f4d9480a640 1 -- 172.21.15.104:0/2140275844 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4d9013fda0 con 0x7f4d900733f0 2024-03-28T14:08:05.317 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.315+0000 7f4d9480a640 1 --2- 172.21.15.104:0/2140275844 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4d900733f0 0x7f4d900b2850 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f4d8c009e70 tx=0x7f4d8c00c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:08:05.317 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.315+0000 7f4d85ffb640 1 -- 172.21.15.104:0/2140275844 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d8c01f070 con 0x7f4d900733f0 2024-03-28T14:08:05.318 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.315+0000 7f4d85ffb640 1 -- 172.21.15.104:0/2140275844 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4d8c01a410 con 0x7f4d900733f0 2024-03-28T14:08:05.318 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.315+0000 7f4d85ffb640 1 -- 172.21.15.104:0/2140275844 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d8c01a7c0 con 0x7f4d900733f0 2024-03-28T14:08:05.318 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.315+0000 7f4d9726f640 1 -- 172.21.15.104:0/2140275844 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4d90140080 con 0x7f4d900733f0 2024-03-28T14:08:05.318 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.315+0000 7f4d9726f640 1 -- 172.21.15.104:0/2140275844 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4d900b4050 con 0x7f4d900733f0 2024-03-28T14:08:05.318 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.315+0000 7f71a9754640 1 Processor -- start 2024-03-28T14:08:05.319 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.319+0000 7f4d85ffb640 1 -- 172.21.15.104:0/2140275844 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) ==== 87803+0+0 (secure 0 0 0) 0x7f4d8c022070 con 0x7f4d900733f0 2024-03-28T14:08:05.320 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.319+0000 7f4d85ffb640 1 --2- 172.21.15.104:0/2140275844 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f4d74068010 0x7f4d7406a4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:05.320 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.319+0000 7f4d85ffb640 1 -- 172.21.15.104:0/2140275844 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f4d8c089ea0 con 0x7f4d900733f0 2024-03-28T14:08:05.320 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.319+0000 7f4d9500b640 1 --2- 172.21.15.104:0/2140275844 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f4d74068010 0x7f4d7406a4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:05.321 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.319+0000 7f4d9500b640 1 --2- 172.21.15.104:0/2140275844 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f4d74068010 0x7f4d7406a4d0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f4d800b1000 tx=0x7f4d80008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:08:05.321 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.315+0000 7f71a9754640 1 -- start start 2024-03-28T14:08:05.322 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.315+0000 7f71a9754640 1 --2- >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f71a40add50 0x7f71a4157a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:05.323 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.315+0000 7f71a9754640 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f71a4157fc0 0x7f71a4158430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:05.323 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.315+0000 7f71a9754640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f71a40ae600 con 0x7f71a40b8c50 2024-03-28T14:08:05.323 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.315+0000 7f71a9754640 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 -- 0x7f71a40b9030 con 0x7f71a4157fc0 2024-03-28T14:08:05.323 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.315+0000 7f71a9754640 1 -- --> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] -- mon_getmap magic: 0 -- 0x7f71a40b0100 con 0x7f71a40add50 2024-03-28T14:08:05.323 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.315+0000 7f71a8f53640 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f71a4157fc0 0x7f71a4158430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:05.324 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.315+0000 7f71a8f53640 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f71a4157fc0 0x7f71a4158430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:42436/0 (socket says 172.21.15.104:42436) 2024-03-28T14:08:05.324 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.315+0000 7f71a8f53640 1 -- 172.21.15.104:0/3550563918 learned_addr learned my addr 172.21.15.104:0/3550563918 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-03-28T14:08:05.324 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.323+0000 7f89addc5640 1 Processor -- start 2024-03-28T14:08:05.324 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.323+0000 7f89addc5640 1 -- start start 2024-03-28T14:08:05.325 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.323+0000 7f89addc5640 1 --2- >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f89a003ad90 0x7f89a00ca580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:05.326 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.323+0000 7f89ad5c4640 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f89a00214d0 0x7f89a0021980 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:47322/0 (socket says 172.21.15.104:47322) 2024-03-28T14:08:05.327 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.323+0000 7f89ad5c4640 1 -- 172.21.15.104:0/1427423539 learned_addr learned my addr 172.21.15.104:0/1427423539 (peer_addr_for_me v1:172.21.15.104:0/0) 2024-03-28T14:08:05.327 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.323+0000 7f89acdc3640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7f89a000c360 0x7f89a003a680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.104:48142/0 (socket says 172.21.15.104:48142) 2024-03-28T14:08:05.327 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.323+0000 7f89addc5640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f89a000c740 con 0x7f89a00214d0 2024-03-28T14:08:05.327 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.323+0000 7f89a7fff640 1 --2- 172.21.15.104:0/1427423539 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f89a003ad90 0x7f89a00ca580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:05.327 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.323+0000 7f89addc5640 1 -- 172.21.15.104:0/1427423539 --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f89a00cad10 con 0x7f89a000c360 2024-03-28T14:08:05.327 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.323+0000 7f89addc5640 1 -- 172.21.15.104:0/1427423539 --> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] -- mon_getmap magic: 0 -- 0x7f89a00caac0 con 0x7f89a003ad90 2024-03-28T14:08:05.328 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.315+0000 7f71a8f53640 1 -- 172.21.15.104:0/3550563918 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f71a40add50 msgr2=0x7f71a4157a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T14:08:05.328 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.315+0000 7f71a8f53640 1 --2- 172.21.15.104:0/3550563918 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f71a40add50 0x7f71a4157a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:05.328 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.315+0000 7f71a37fe640 1 -- 172.21.15.104:0/3550563918 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3876001305 0 0) 0x7f7194001120 con 0x7f71a40b8c50 2024-03-28T14:08:05.328 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.315+0000 7f71a8f53640 1 -- 172.21.15.104:0/3550563918 >> v1:172.21.15.171:6789/0 conn(0x7f71a40b8c50 legacy=0x7f71a408b5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:05.328 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.315+0000 7f71a8f53640 1 -- 172.21.15.104:0/3550563918 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f71a40b03e0 con 0x7f71a4157fc0 2024-03-28T14:08:05.328 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.319+0000 7f71a8f53640 1 --2- 172.21.15.104:0/3550563918 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f71a4157fc0 0x7f71a4158430 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f719c008ec0 tx=0x7f719c021250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=722a273392406cbc server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:08:05.329 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.323+0000 7f89a77fe640 1 -- 172.21.15.104:0/1427423539 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2815076827 0 0) 0x7f89a00cad10 con 0x7f89a000c360 2024-03-28T14:08:05.329 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.323+0000 7f89a77fe640 1 -- 172.21.15.104:0/1427423539 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8998003530 con 0x7f89a000c360 2024-03-28T14:08:05.329 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.323+0000 7f89a77fe640 1 -- 172.21.15.104:0/1427423539 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3661446220 0 0) 0x7f89a000c740 con 0x7f89a00214d0 2024-03-28T14:08:05.329 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.323+0000 7f89a77fe640 1 -- 172.21.15.104:0/1427423539 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f89a00cad10 con 0x7f89a00214d0 2024-03-28T14:08:05.330 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.319+0000 7f71a37fe640 1 -- 172.21.15.104:0/3550563918 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f719c0a9e60 con 0x7f71a4157fc0 2024-03-28T14:08:05.330 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.319+0000 7f71a37fe640 1 -- 172.21.15.104:0/3550563918 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f719c0b7b50 con 0x7f71a4157fc0 2024-03-28T14:08:05.330 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.319+0000 7f71a37fe640 1 -- 172.21.15.104:0/3550563918 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f719c013180 con 0x7f71a4157fc0 2024-03-28T14:08:05.330 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.319+0000 7f71a9754640 1 -- 172.21.15.104:0/3550563918 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f71a4157fc0 msgr2=0x7f71a4158430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:05.330 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.319+0000 7f71a9754640 1 --2- 172.21.15.104:0/3550563918 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f71a4157fc0 0x7f71a4158430 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f719c008ec0 tx=0x7f719c021250 comp rx=0 tx=0).stop 2024-03-28T14:08:05.330 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.319+0000 7f71a9754640 1 -- 172.21.15.104:0/3550563918 shutdown_connections 2024-03-28T14:08:05.330 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.319+0000 7f71a9754640 1 --2- 172.21.15.104:0/3550563918 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f71a4157fc0 0x7f71a4158430 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:05.330 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.319+0000 7f71a9754640 1 --2- 172.21.15.104:0/3550563918 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f71a40add50 0x7f71a4157a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:05.331 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.319+0000 7f71a9754640 1 -- 172.21.15.104:0/3550563918 >> 172.21.15.104:0/3550563918 conn(0x7f71a406c530 msgr2=0x7f71a406c940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T14:08:05.331 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.319+0000 7f71a9754640 1 -- 172.21.15.104:0/3550563918 shutdown_connections 2024-03-28T14:08:05.331 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.319+0000 7f71a9754640 1 -- 172.21.15.104:0/3550563918 wait complete. 2024-03-28T14:08:05.331 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.323+0000 7f71a9754640 1 Processor -- start 2024-03-28T14:08:05.331 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.323+0000 7f71a9754640 1 -- start start 2024-03-28T14:08:05.331 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.323+0000 7f71a9754640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f71a40add50 0x7f71a414df70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:05.331 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.323+0000 7f71a9754640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f71a40b8c50 0x7f71a41684f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:05.331 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.323+0000 7f71a9754640 1 --2- >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f71a4157fc0 0x7f71a408a340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:05.331 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.323+0000 7f71a9754640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f71a4161560 con 0x7f71a40add50 2024-03-28T14:08:05.331 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.323+0000 7f71a9754640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f71a408bcf0 con 0x7f71a40b8c50 2024-03-28T14:08:05.331 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.323+0000 7f71a9754640 1 -- --> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] -- mon_getmap magic: 0 -- 0x7f71a41616e0 con 0x7f71a4157fc0 2024-03-28T14:08:05.332 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.323+0000 7f71a3fff640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f71a40add50 0x7f71a414df70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:05.332 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.323+0000 7f71a3fff640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f71a40add50 0x7f71a414df70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:42452/0 (socket says 172.21.15.104:42452) 2024-03-28T14:08:05.332 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.323+0000 7f71a3fff640 1 -- 172.21.15.104:0/407177848 learned_addr learned my addr 172.21.15.104:0/407177848 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-03-28T14:08:05.332 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.323+0000 7f71a3fff640 1 -- 172.21.15.104:0/407177848 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f71a4157fc0 msgr2=0x7f71a408a340 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-28T14:08:05.332 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.323+0000 7f71a3fff640 1 --2- 172.21.15.104:0/407177848 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f71a4157fc0 0x7f71a408a340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:05.332 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.323+0000 7f71a3fff640 1 -- 172.21.15.104:0/407177848 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f71a40b8c50 msgr2=0x7f71a41684f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:05.332 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.323+0000 7f71a3fff640 1 --2- 172.21.15.104:0/407177848 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f71a40b8c50 0x7f71a41684f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:05.332 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.323+0000 7f71a3fff640 1 -- 172.21.15.104:0/407177848 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f71a414e670 con 0x7f71a40add50 2024-03-28T14:08:05.333 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.307+0000 7f101fc9d640 1 Processor -- start 2024-03-28T14:08:05.333 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.307+0000 7f101fc9d640 1 -- start start 2024-03-28T14:08:05.333 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.307+0000 7f101fc9d640 1 --2- >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f101806c8f0 0x7f10180adff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:05.333 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.307+0000 7f101fc9d640 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f10180af180 0x7f101808b4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:05.333 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.307+0000 7f101fc9d640 1 --2- >> v2:172.21.15.171:3300/0 conn(0x7f101808ba10 0x7f10180b3730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:05.334 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.307+0000 7f101fc9d640 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 -- 0x7f101809e3c0 con 0x7f10180af180 2024-03-28T14:08:05.334 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.307+0000 7f101fc9d640 1 -- --> v2:172.21.15.171:3300/0 -- mon_getmap magic: 0 -- 0x7f101806ccd0 con 0x7f101808ba10 2024-03-28T14:08:05.334 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.307+0000 7f101fc9d640 1 -- --> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] -- mon_getmap magic: 0 -- 0x7f10180af6d0 con 0x7f101806c8f0 2024-03-28T14:08:05.334 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.307+0000 7f101d238640 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f10180af180 0x7f101808b4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:05.334 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.307+0000 7f101d238640 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f10180af180 0x7f101808b4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:42416/0 (socket says 172.21.15.104:42416) 2024-03-28T14:08:05.334 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.307+0000 7f101d238640 1 -- 172.21.15.104:0/843050351 learned_addr learned my addr 172.21.15.104:0/843050351 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-03-28T14:08:05.334 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.307+0000 7f101da39640 1 --2- 172.21.15.104:0/843050351 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f101806c8f0 0x7f10180adff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:05.334 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.307+0000 7f101e23a640 1 --2- 172.21.15.104:0/843050351 >> v2:172.21.15.171:3300/0 conn(0x7f101808ba10 0x7f10180b3730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:05.335 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.327+0000 7f89a7fff640 1 -- 172.21.15.104:0/1427423539 >> v1:172.21.15.171:6789/0 conn(0x7f89a000c360 legacy=0x7f89a003a680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:05.335 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.327+0000 7f89a7fff640 1 -- 172.21.15.104:0/1427423539 >> v1:172.21.15.104:6789/0 conn(0x7f89a00214d0 legacy=0x7f89a0021980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:05.335 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.327+0000 7f89a7fff640 1 -- 172.21.15.104:0/1427423539 --> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f89a00caff0 con 0x7f89a003ad90 2024-03-28T14:08:05.335 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.327+0000 7f84dac14640 1 Processor -- start 2024-03-28T14:08:05.335 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.327+0000 7f84dac14640 1 -- start start 2024-03-28T14:08:05.336 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.327+0000 7f71a3fff640 1 --2- 172.21.15.104:0/407177848 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f71a40add50 0x7f71a414df70 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f71940507b0 tx=0x7f7194050c70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:08:05.336 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.327+0000 7f71a17fa640 1 -- 172.21.15.104:0/407177848 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7194050e60 con 0x7f71a40add50 2024-03-28T14:08:05.336 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.327+0000 7f71a9754640 1 -- 172.21.15.104:0/407177848 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f71a4167fd0 con 0x7f71a40add50 2024-03-28T14:08:05.336 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.327+0000 7f71a9754640 1 -- 172.21.15.104:0/407177848 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f71a4085d90 con 0x7f71a40add50 2024-03-28T14:08:05.336 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.327+0000 7f71a17fa640 1 -- 172.21.15.104:0/407177848 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7194005290 con 0x7f71a40add50 2024-03-28T14:08:05.336 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.327+0000 7f71a17fa640 1 -- 172.21.15.104:0/407177848 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f719404e640 con 0x7f71a40add50 2024-03-28T14:08:05.337 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.327+0000 7f84dac14640 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f84d40b47a0 0x7f84d41453f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:05.337 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.327+0000 7f84dac14640 1 --2- >> v2:172.21.15.171:3300/0 conn(0x7f84d40add50 0x7f84d414fc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:05.337 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.327+0000 7f84dac14640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f84d40af370 con 0x7f84d40b9090 2024-03-28T14:08:05.337 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.327+0000 7f84dac14640 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 -- 0x7f84d40ae1e0 con 0x7f84d40b47a0 2024-03-28T14:08:05.337 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.327+0000 7f84dac14640 1 -- --> v2:172.21.15.171:3300/0 -- mon_getmap magic: 0 -- 0x7f84d40b51b0 con 0x7f84d40add50 2024-03-28T14:08:05.337 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.327+0000 7f4d9726f640 1 --2- 172.21.15.104:0/2140275844 >> [v2:172.21.15.171:6808/2934826925,v1:172.21.15.171:6810/2934826925] conn(0x7f4d9013f870 0x7f4d9013d6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:05.338 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.327+0000 7f4d9726f640 1 -- 172.21.15.104:0/2140275844 --> [v2:172.21.15.171:6808/2934826925,v1:172.21.15.171:6810/2934826925] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f4d900b4250 con 0x7f4d9013f870 2024-03-28T14:08:05.338 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.327+0000 7f4d9580c640 1 --2- 172.21.15.104:0/2140275844 >> [v2:172.21.15.171:6808/2934826925,v1:172.21.15.171:6810/2934826925] conn(0x7f4d9013f870 0x7f4d9013d6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:05.338 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.327+0000 7f84da413640 1 --2- >> v2:172.21.15.171:3300/0 conn(0x7f84d40add50 0x7f84d414fc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:05.338 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.327+0000 7f84da413640 1 --2- >> v2:172.21.15.171:3300/0 conn(0x7f84d40add50 0x7f84d414fc90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.171:3300/0 says I am v2:172.21.15.104:53540/0 (socket says 172.21.15.104:53540) 2024-03-28T14:08:05.338 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.327+0000 7f84da413640 1 -- 172.21.15.104:0/4161375878 learned_addr learned my addr 172.21.15.104:0/4161375878 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-03-28T14:08:05.338 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.327+0000 7f71a17fa640 1 -- 172.21.15.104:0/407177848 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) ==== 87803+0+0 (secure 0 0 0) 0x7f719404e7e0 con 0x7f71a40add50 2024-03-28T14:08:05.339 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.327+0000 7f4d9580c640 1 --2- 172.21.15.104:0/2140275844 >> [v2:172.21.15.171:6808/2934826925,v1:172.21.15.171:6810/2934826925] conn(0x7f4d9013f870 0x7f4d9013d6b0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:08:05.339 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.327+0000 7f89a7fff640 1 --2- 172.21.15.104:0/1427423539 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f89a003ad90 0x7f89a00ca580 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f89940097f0 tx=0x7f899401c3e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a2906018deeb0485 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:08:05.339 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.327+0000 7f89a77fe640 1 -- 172.21.15.104:0/1427423539 <== mon.2 v2:172.21.15.171:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89940b0070 con 0x7f89a003ad90 2024-03-28T14:08:05.339 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.331+0000 7f89a77fe640 1 -- 172.21.15.104:0/1427423539 <== mon.2 v2:172.21.15.171:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f89940ab600 con 0x7f89a003ad90 2024-03-28T14:08:05.340 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.327+0000 7f84da413640 1 -- 172.21.15.104:0/4161375878 >> v2:172.21.15.104:3300/0 conn(0x7f84d40b47a0 msgr2=0x7f84d41453f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-28T14:08:05.340 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.327+0000 7f84da413640 1 --2- 172.21.15.104:0/4161375878 >> v2:172.21.15.104:3300/0 conn(0x7f84d40b47a0 0x7f84d41453f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:05.340 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.327+0000 7f84da413640 1 -- 172.21.15.104:0/4161375878 >> v1:172.21.15.171:6789/0 conn(0x7f84d40b9090 legacy=0x7f84d4145ae0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-28T14:08:05.340 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.327+0000 7f84da413640 1 -- 172.21.15.104:0/4161375878 --> v2:172.21.15.171:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f84d40b9470 con 0x7f84d40add50 2024-03-28T14:08:05.341 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.331+0000 7f71a17fa640 1 --2- 172.21.15.104:0/407177848 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f7184067fc0 0x7f718406a480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:05.341 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.331+0000 7f71a17fa640 1 -- 172.21.15.104:0/407177848 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f71940ca270 con 0x7f71a40add50 2024-03-28T14:08:05.341 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.331+0000 7f719bfff640 1 --2- 172.21.15.104:0/407177848 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f7184067fc0 0x7f718406a480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:05.341 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.331+0000 7f89addc5640 1 -- 172.21.15.104:0/1427423539 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f89a003ad90 msgr2=0x7f89a00ca580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:05.341 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.331+0000 7f89addc5640 1 --2- 172.21.15.104:0/1427423539 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f89a003ad90 0x7f89a00ca580 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f89940097f0 tx=0x7f899401c3e0 comp rx=0 tx=0).stop 2024-03-28T14:08:05.342 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.331+0000 7f4d85ffb640 1 -- 172.21.15.104:0/2140275844 <== osd.5 v2:172.21.15.171:6808/2934826925 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f4d900b4250 con 0x7f4d9013f870 2024-03-28T14:08:05.342 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.331+0000 7f89addc5640 1 -- 172.21.15.104:0/1427423539 shutdown_connections 2024-03-28T14:08:05.342 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.331+0000 7f89addc5640 1 --2- 172.21.15.104:0/1427423539 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f89a003ad90 0x7f89a00ca580 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:05.342 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.331+0000 7f89addc5640 1 -- 172.21.15.104:0/1427423539 >> 172.21.15.104:0/1427423539 conn(0x7f89a000b5e0 msgr2=0x7f89a000b9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T14:08:05.343 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.331+0000 7f71a9754640 1 --2- 172.21.15.104:0/407177848 >> [v2:172.21.15.104:6824/2352012626,v1:172.21.15.104:6825/2352012626] conn(0x7f7188001660 0x7f7188003b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:05.343 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.331+0000 7f71a9754640 1 -- 172.21.15.104:0/407177848 --> [v2:172.21.15.104:6824/2352012626,v1:172.21.15.104:6825/2352012626] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f7188006bc0 con 0x7f7188001660 2024-03-28T14:08:05.343 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.331+0000 7f71a8f53640 1 --2- 172.21.15.104:0/407177848 >> [v2:172.21.15.104:6824/2352012626,v1:172.21.15.104:6825/2352012626] conn(0x7f7188001660 0x7f7188003b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:05.343 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.331+0000 7f89addc5640 1 -- 172.21.15.104:0/1427423539 shutdown_connections 2024-03-28T14:08:05.344 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.331+0000 7f84da413640 1 --2- 172.21.15.104:0/4161375878 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f84d40add50 0x7f84d414fc90 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f84d40b28d0 tx=0x7f84d001c470 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e9015f0f0e0de87a server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:08:05.344 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.331+0000 7f89addc5640 1 -- 172.21.15.104:0/1427423539 wait complete. 2024-03-28T14:08:05.344 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.331+0000 7f84d8c10640 1 -- 172.21.15.104:0/4161375878 <== mon.1 v2:172.21.15.171:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84d00b0070 con 0x7f84d40add50 2024-03-28T14:08:05.345 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.331+0000 7f84d8c10640 1 -- 172.21.15.104:0/4161375878 <== mon.1 v2:172.21.15.171:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f84d00ab600 con 0x7f84d40add50 2024-03-28T14:08:05.345 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.331+0000 7f89addc5640 1 Processor -- start 2024-03-28T14:08:05.345 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.331+0000 7f84dac14640 1 -- 172.21.15.104:0/4161375878 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f84d40add50 msgr2=0x7f84d414fc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:05.345 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.331+0000 7f84dac14640 1 --2- 172.21.15.104:0/4161375878 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f84d40add50 0x7f84d414fc90 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f84d40b28d0 tx=0x7f84d001c470 comp rx=0 tx=0).stop 2024-03-28T14:08:05.345 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.331+0000 7f89addc5640 1 -- start start 2024-03-28T14:08:05.346 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.335+0000 7f71a8f53640 1 --2- 172.21.15.104:0/407177848 >> [v2:172.21.15.104:6824/2352012626,v1:172.21.15.104:6825/2352012626] conn(0x7f7188001660 0x7f7188003b20 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:08:05.347 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.335+0000 7f89addc5640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f89a000c360 0x7f89a0024d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:05.347 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.335+0000 7f89addc5640 1 --2- >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f89a00214d0 0x7f89a016cbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:05.347 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.335+0000 7f89addc5640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f89a003ad90 0x7f89a0007bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:05.347 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.335+0000 7f89addc5640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f89a00d8860 con 0x7f89a000c360 2024-03-28T14:08:05.347 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.335+0000 7f89addc5640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f89a001fa90 con 0x7f89a003ad90 2024-03-28T14:08:05.347 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.335+0000 7f89addc5640 1 -- --> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] -- mon_getmap magic: 0 -- 0x7f89a00d89e0 con 0x7f89a00214d0 2024-03-28T14:08:05.347 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.335+0000 7f89a7fff640 1 --2- >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f89a00214d0 0x7f89a016cbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:05.347 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.335+0000 7f89a7fff640 1 --2- >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f89a00214d0 0x7f89a016cbd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.171:3301/0 says I am v2:172.21.15.104:55980/0 (socket says 172.21.15.104:55980) 2024-03-28T14:08:05.347 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.335+0000 7f89a7fff640 1 -- 172.21.15.104:0/1108949963 learned_addr learned my addr 172.21.15.104:0/1108949963 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-03-28T14:08:05.347 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.335+0000 7f89acdc3640 1 --2- 172.21.15.104:0/1108949963 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f89a000c360 0x7f89a0024d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:05.347 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.335+0000 7f89ad5c4640 1 --2- 172.21.15.104:0/1108949963 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f89a003ad90 0x7f89a0007bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:05.348 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.335+0000 7f71a17fa640 1 -- 172.21.15.104:0/407177848 <== osd.2 v2:172.21.15.104:6824/2352012626 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f7188006bc0 con 0x7f7188001660 2024-03-28T14:08:05.349 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.327+0000 7f101d238640 1 -- 172.21.15.104:0/843050351 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f101806c8f0 msgr2=0x7f10180adff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T14:08:05.349 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.327+0000 7f101d238640 1 --2- 172.21.15.104:0/843050351 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f101806c8f0 0x7f10180adff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:05.349 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.327+0000 7f101d238640 1 -- 172.21.15.104:0/843050351 >> v2:172.21.15.171:3300/0 conn(0x7f101808ba10 msgr2=0x7f10180b3730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T14:08:05.349 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.327+0000 7f101d238640 1 --2- 172.21.15.104:0/843050351 >> v2:172.21.15.171:3300/0 conn(0x7f101808ba10 0x7f10180b3730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:05.349 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.327+0000 7f101d238640 1 -- 172.21.15.104:0/843050351 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f10180af9b0 con 0x7f10180af180 2024-03-28T14:08:05.350 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.327+0000 7f101d238640 1 --2- 172.21.15.104:0/843050351 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f10180af180 0x7f101808b4d0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f1008006fd0 tx=0x7f100801cb50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4e9fac40ef4e6509 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:08:05.350 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.331+0000 7f101ca37640 1 -- 172.21.15.104:0/843050351 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10080ab070 con 0x7f10180af180 2024-03-28T14:08:05.350 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.331+0000 7f101ca37640 1 -- 172.21.15.104:0/843050351 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1008004230 con 0x7f10180af180 2024-03-28T14:08:05.350 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.331+0000 7f101ca37640 1 -- 172.21.15.104:0/843050351 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10080a58d0 con 0x7f10180af180 2024-03-28T14:08:05.350 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.331+0000 7f101fc9d640 1 -- 172.21.15.104:0/843050351 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f10180af180 msgr2=0x7f101808b4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:05.350 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.331+0000 7f101fc9d640 1 --2- 172.21.15.104:0/843050351 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f10180af180 0x7f101808b4d0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f1008006fd0 tx=0x7f100801cb50 comp rx=0 tx=0).stop 2024-03-28T14:08:05.350 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.331+0000 7f101fc9d640 1 -- 172.21.15.104:0/843050351 shutdown_connections 2024-03-28T14:08:05.350 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.331+0000 7f101fc9d640 1 --2- 172.21.15.104:0/843050351 >> v2:172.21.15.171:3300/0 conn(0x7f101808ba10 0x7f10180b3730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:05.350 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.331+0000 7f101fc9d640 1 --2- 172.21.15.104:0/843050351 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f10180af180 0x7f101808b4d0 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:05.350 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.331+0000 7f101fc9d640 1 --2- 172.21.15.104:0/843050351 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f101806c8f0 0x7f10180adff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:05.350 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.331+0000 7f101fc9d640 1 -- 172.21.15.104:0/843050351 >> 172.21.15.104:0/843050351 conn(0x7f10180b8d30 msgr2=0x7f10180b9140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T14:08:05.350 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.331+0000 7f101fc9d640 1 -- 172.21.15.104:0/843050351 shutdown_connections 2024-03-28T14:08:05.351 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.331+0000 7f101fc9d640 1 -- 172.21.15.104:0/843050351 wait complete. 2024-03-28T14:08:05.351 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.331+0000 7f101fc9d640 1 Processor -- start 2024-03-28T14:08:05.351 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.331+0000 7f101fc9d640 1 -- start start 2024-03-28T14:08:05.351 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.331+0000 7f101fc9d640 1 --2- >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f101806c8f0 0x7f10181f0220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:05.351 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.331+0000 7f101fc9d640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f101808ba10 0x7f1018159200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:05.351 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.331+0000 7f101fc9d640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f10180af180 0x7f101809cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:05.351 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.331+0000 7f101fc9d640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f10180ae530 con 0x7f10180af180 2024-03-28T14:08:05.351 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.331+0000 7f101fc9d640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f10180b5460 con 0x7f101808ba10 2024-03-28T14:08:05.351 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.331+0000 7f101fc9d640 1 -- --> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] -- mon_getmap magic: 0 -- 0x7f10180b3010 con 0x7f101806c8f0 2024-03-28T14:08:05.351 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.331+0000 7f101e23a640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f10180af180 0x7f101809cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:05.351 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.331+0000 7f101e23a640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f10180af180 0x7f101809cc60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:42464/0 (socket says 172.21.15.104:42464) 2024-03-28T14:08:05.351 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.331+0000 7f101e23a640 1 -- 172.21.15.104:0/3841593285 learned_addr learned my addr 172.21.15.104:0/3841593285 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-03-28T14:08:05.351 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.331+0000 7f101e23a640 1 -- 172.21.15.104:0/3841593285 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f101806c8f0 msgr2=0x7f10181f0220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:05.351 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.331+0000 7f101e23a640 1 --2- 172.21.15.104:0/3841593285 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f101806c8f0 0x7f10181f0220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:05.352 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.331+0000 7f101e23a640 1 -- 172.21.15.104:0/3841593285 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f101808ba10 msgr2=0x7f1018159200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:05.352 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.331+0000 7f101e23a640 1 --2- 172.21.15.104:0/3841593285 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f101808ba10 0x7f1018159200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:05.352 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.331+0000 7f101e23a640 1 -- 172.21.15.104:0/3841593285 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f101815b640 con 0x7f10180af180 2024-03-28T14:08:05.352 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.335+0000 7f101e23a640 1 --2- 172.21.15.104:0/3841593285 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f10180af180 0x7f101809cc60 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f1010052fd0 tx=0x7f10100504a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:08:05.352 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.335+0000 7f100e7fc640 1 -- 172.21.15.104:0/3841593285 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f101005b070 con 0x7f10180af180 2024-03-28T14:08:05.352 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.335+0000 7f100e7fc640 1 -- 172.21.15.104:0/3841593285 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f10100573f0 con 0x7f10180af180 2024-03-28T14:08:05.352 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.335+0000 7f100e7fc640 1 -- 172.21.15.104:0/3841593285 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10100577e0 con 0x7f10180af180 2024-03-28T14:08:05.352 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.335+0000 7f101fc9d640 1 -- 172.21.15.104:0/3841593285 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1018159740 con 0x7f10180af180 2024-03-28T14:08:05.352 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.335+0000 7f101fc9d640 1 -- 172.21.15.104:0/3841593285 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f101809d1a0 con 0x7f10180af180 2024-03-28T14:08:05.352 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.335+0000 7f89acdc3640 1 -- 172.21.15.104:0/1108949963 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f89a00214d0 msgr2=0x7f89a016cbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:05.353 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.335+0000 7f89acdc3640 1 --2- 172.21.15.104:0/1108949963 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f89a00214d0 0x7f89a016cbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:05.353 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.335+0000 7f89acdc3640 1 -- 172.21.15.104:0/1108949963 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f89a003ad90 msgr2=0x7f89a0007bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:05.353 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.335+0000 7f89acdc3640 1 --2- 172.21.15.104:0/1108949963 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f89a003ad90 0x7f89a0007bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:05.353 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.335+0000 7f89acdc3640 1 -- 172.21.15.104:0/1108949963 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f89a00d59b0 con 0x7f89a000c360 2024-03-28T14:08:05.353 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.335+0000 7f101fc9d640 1 -- 172.21.15.104:0/3841593285 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f10180687c0 con 0x7f10180af180 2024-03-28T14:08:05.353 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.339+0000 7f89acdc3640 1 --2- 172.21.15.104:0/1108949963 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f89a000c360 0x7f89a0024d10 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f899c00c390 tx=0x7f899c00c850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:08:05.354 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.339+0000 7f84dac14640 1 -- 172.21.15.104:0/4161375878 shutdown_connections 2024-03-28T14:08:05.354 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.339+0000 7f84dac14640 1 --2- 172.21.15.104:0/4161375878 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f84d40add50 0x7f84d414fc90 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:05.354 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.339+0000 7f84dac14640 1 --2- 172.21.15.104:0/4161375878 >> v2:172.21.15.104:3300/0 conn(0x7f84d40b47a0 0x7f84d41453f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:05.354 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.339+0000 7f84dac14640 1 -- 172.21.15.104:0/4161375878 >> 172.21.15.104:0/4161375878 conn(0x7f84d406c530 msgr2=0x7f84d406c940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T14:08:05.354 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.339+0000 7f84dac14640 1 -- 172.21.15.104:0/4161375878 shutdown_connections 2024-03-28T14:08:05.354 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.339+0000 7f84dac14640 1 -- 172.21.15.104:0/4161375878 wait complete. 2024-03-28T14:08:05.355 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.339+0000 7f719bfff640 1 --2- 172.21.15.104:0/407177848 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f7184067fc0 0x7f718406a480 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f71a4087ad0 tx=0x7f718c006c00 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:08:05.357 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.339+0000 7f84dac14640 1 Processor -- start 2024-03-28T14:08:05.357 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.339+0000 7f84dac14640 1 -- start start 2024-03-28T14:08:05.357 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.339+0000 7f84dac14640 1 --2- >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f84d40add50 0x7f84d413eef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:05.357 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.339+0000 7f84dac14640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f84d40b47a0 0x7f84d4138cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:05.357 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.339+0000 7f84dac14640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f84d40b9090 0x7f84d413db90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:05.358 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.339+0000 7f84dac14640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f84d40732d0 con 0x7f84d40b47a0 2024-03-28T14:08:05.358 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.339+0000 7f84dac14640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f84d4073150 con 0x7f84d40b9090 2024-03-28T14:08:05.358 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.339+0000 7f84dac14640 1 -- --> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] -- mon_getmap magic: 0 -- 0x7f84d41529f0 con 0x7f84d40add50 2024-03-28T14:08:05.358 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.339+0000 7f84d9411640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f84d40b47a0 0x7f84d4138cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:05.358 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.339+0000 7f84da413640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f84d40b9090 0x7f84d413db90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:05.358 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.339+0000 7f84d9411640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f84d40b47a0 0x7f84d4138cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:42488/0 (socket says 172.21.15.104:42488) 2024-03-28T14:08:05.358 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.339+0000 7f84d9411640 1 -- 172.21.15.104:0/55806039 learned_addr learned my addr 172.21.15.104:0/55806039 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-03-28T14:08:05.358 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.339+0000 7f84d9c12640 1 --2- 172.21.15.104:0/55806039 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f84d40add50 0x7f84d413eef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:05.358 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.339+0000 7f84d9411640 1 -- 172.21.15.104:0/55806039 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f84d40add50 msgr2=0x7f84d413eef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:05.358 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.339+0000 7f84d9411640 1 --2- 172.21.15.104:0/55806039 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f84d40add50 0x7f84d413eef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:05.359 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.339+0000 7f84d9411640 1 -- 172.21.15.104:0/55806039 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f84d40b9090 msgr2=0x7f84d413db90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:05.359 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.339+0000 7f84d9411640 1 --2- 172.21.15.104:0/55806039 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f84d40b9090 0x7f84d413db90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:05.359 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.339+0000 7f84d9411640 1 -- 172.21.15.104:0/55806039 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f84d413f5d0 con 0x7f84d40b47a0 2024-03-28T14:08:05.359 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.343+0000 7f84d9411640 1 --2- 172.21.15.104:0/55806039 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f84d40b47a0 0x7f84d4138cc0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f84cc00b9e0 tx=0x7f84cc00bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:08:05.359 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.343+0000 7f84ca7fc640 1 -- 172.21.15.104:0/55806039 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84cc019070 con 0x7f84d40b47a0 2024-03-28T14:08:05.359 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.343+0000 7f84ca7fc640 1 -- 172.21.15.104:0/55806039 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f84cc00cba0 con 0x7f84d40b47a0 2024-03-28T14:08:05.359 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.343+0000 7f84dac14640 1 -- 172.21.15.104:0/55806039 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f84d406f770 con 0x7f84d40b47a0 2024-03-28T14:08:05.359 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.343+0000 7f84ca7fc640 1 -- 172.21.15.104:0/55806039 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84cc014440 con 0x7f84d40b47a0 2024-03-28T14:08:05.360 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.343+0000 7f84dac14640 1 -- 172.21.15.104:0/55806039 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f84d40b1e30 con 0x7f84d40b47a0 2024-03-28T14:08:05.360 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.343+0000 7f84dac14640 1 -- 172.21.15.104:0/55806039 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f84d4068640 con 0x7f84d40b47a0 2024-03-28T14:08:05.360 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.343+0000 7f89a57fa640 1 -- 172.21.15.104:0/1108949963 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f899c0074e0 con 0x7f89a000c360 2024-03-28T14:08:05.360 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.343+0000 7f89a57fa640 1 -- 172.21.15.104:0/1108949963 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f899c0047f0 con 0x7f89a000c360 2024-03-28T14:08:05.361 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.343+0000 7f89a57fa640 1 -- 172.21.15.104:0/1108949963 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f899c0076a0 con 0x7f89a000c360 2024-03-28T14:08:05.361 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.343+0000 7f89addc5640 1 -- 172.21.15.104:0/1108949963 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f89a0039920 con 0x7f89a000c360 2024-03-28T14:08:05.361 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.343+0000 7f89addc5640 1 -- 172.21.15.104:0/1108949963 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f89a0009070 con 0x7f89a000c360 2024-03-28T14:08:05.361 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.343+0000 7f100e7fc640 1 -- 172.21.15.104:0/3841593285 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) ==== 87803+0+0 (secure 0 0 0) 0x7f101005e070 con 0x7f10180af180 2024-03-28T14:08:05.361 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.343+0000 7f100e7fc640 1 --2- 172.21.15.104:0/3841593285 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f0ffc067fc0 0x7f0ffc06a480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:05.361 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.343+0000 7f100e7fc640 1 -- 172.21.15.104:0/3841593285 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f10100cb360 con 0x7f10180af180 2024-03-28T14:08:05.361 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.343+0000 7f100e7fc640 1 --2- 172.21.15.104:0/3841593285 >> [v2:172.21.15.171:6824/436742919,v1:172.21.15.171:6825/436742919] conn(0x7f0ffc070160 0x7f0ffc072580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:05.361 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.343+0000 7f100e7fc640 1 -- 172.21.15.104:0/3841593285 --> [v2:172.21.15.171:6824/436742919,v1:172.21.15.171:6825/436742919] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f0ffc072c30 con 0x7f0ffc070160 2024-03-28T14:08:05.361 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.343+0000 7f100e7fc640 1 -- 172.21.15.104:0/3841593285 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f10100cb7f0 con 0x7f10180af180 2024-03-28T14:08:05.361 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.343+0000 7f101da39640 1 --2- 172.21.15.104:0/3841593285 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f0ffc067fc0 0x7f0ffc06a480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:05.362 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.343+0000 7f101d238640 1 --2- 172.21.15.104:0/3841593285 >> [v2:172.21.15.171:6824/436742919,v1:172.21.15.171:6825/436742919] conn(0x7f0ffc070160 0x7f0ffc072580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:05.362 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.343+0000 7f89addc5640 1 -- 172.21.15.104:0/1108949963 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f89a00057f0 con 0x7f89a000c360 2024-03-28T14:08:05.362 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.343+0000 7f84ca7fc640 1 -- 172.21.15.104:0/55806039 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) ==== 87803+0+0 (secure 0 0 0) 0x7f84cc01c070 con 0x7f84d40b47a0 2024-03-28T14:08:05.362 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.343+0000 7f84ca7fc640 1 --2- 172.21.15.104:0/55806039 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f84b8067fc0 0x7f84b806a480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:05.362 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.343+0000 7f84ca7fc640 1 -- 172.21.15.104:0/55806039 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f84cc0861d0 con 0x7f84d40b47a0 2024-03-28T14:08:05.363 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.343+0000 7f101da39640 1 --2- 172.21.15.104:0/3841593285 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f0ffc067fc0 0x7f0ffc06a480 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f101400ee10 tx=0x7f1014010040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:08:05.363 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.343+0000 7f84ca7fc640 1 --2- 172.21.15.104:0/55806039 >> [v2:172.21.15.104:6808/19438674,v1:172.21.15.104:6810/19438674] conn(0x7f84b80701c0 0x7f84b80725e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:05.363 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.343+0000 7f84ca7fc640 1 -- 172.21.15.104:0/55806039 --> [v2:172.21.15.104:6808/19438674,v1:172.21.15.104:6810/19438674] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f84b8072c90 con 0x7f84b80701c0 2024-03-28T14:08:05.364 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.343+0000 7f84ca7fc640 1 -- 172.21.15.104:0/55806039 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f84cc08d050 con 0x7f84d40b47a0 2024-03-28T14:08:05.364 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.343+0000 7f101d238640 1 --2- 172.21.15.104:0/3841593285 >> [v2:172.21.15.171:6824/436742919,v1:172.21.15.171:6825/436742919] conn(0x7f0ffc070160 0x7f0ffc072580 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:08:05.364 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.343+0000 7f84d9c12640 1 --2- 172.21.15.104:0/55806039 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f84b8067fc0 0x7f84b806a480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:05.365 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.343+0000 7f89a57fa640 1 -- 172.21.15.104:0/1108949963 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) ==== 87803+0+0 (secure 0 0 0) 0x7f899c01a020 con 0x7f89a000c360 2024-03-28T14:08:05.365 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.347+0000 7f89a57fa640 1 --2- 172.21.15.104:0/1108949963 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f898c067fc0 0x7f898c06a480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:05.365 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.347+0000 7f89a57fa640 1 -- 172.21.15.104:0/1108949963 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f899c085b30 con 0x7f89a000c360 2024-03-28T14:08:05.365 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.347+0000 7f89a57fa640 1 --2- 172.21.15.104:0/1108949963 >> [v2:172.21.15.104:6800/3745690918,v1:172.21.15.104:6801/3745690918] conn(0x7f898c0701c0 0x7f898c0725e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:05.365 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.347+0000 7f89a57fa640 1 -- 172.21.15.104:0/1108949963 --> [v2:172.21.15.104:6800/3745690918,v1:172.21.15.104:6801/3745690918] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f898c072c90 con 0x7f898c0701c0 2024-03-28T14:08:05.365 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.347+0000 7f89a57fa640 1 -- 172.21.15.104:0/1108949963 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f899c08a2c0 con 0x7f89a000c360 2024-03-28T14:08:05.365 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.347+0000 7f89a7fff640 1 --2- 172.21.15.104:0/1108949963 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f898c067fc0 0x7f898c06a480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:05.366 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.347+0000 7f89ad5c4640 1 --2- 172.21.15.104:0/1108949963 >> [v2:172.21.15.104:6800/3745690918,v1:172.21.15.104:6801/3745690918] conn(0x7f898c0701c0 0x7f898c0725e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:05.366 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.347+0000 7f89ad5c4640 1 --2- 172.21.15.104:0/1108949963 >> [v2:172.21.15.104:6800/3745690918,v1:172.21.15.104:6801/3745690918] conn(0x7f898c0701c0 0x7f898c0725e0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:08:05.366 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.343+0000 7f84da413640 1 --2- 172.21.15.104:0/55806039 >> [v2:172.21.15.104:6808/19438674,v1:172.21.15.104:6810/19438674] conn(0x7f84b80701c0 0x7f84b80725e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:05.366 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.347+0000 7f84da413640 1 --2- 172.21.15.104:0/55806039 >> [v2:172.21.15.104:6808/19438674,v1:172.21.15.104:6810/19438674] conn(0x7f84b80701c0 0x7f84b80725e0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:08:05.366 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.347+0000 7f84d9c12640 1 --2- 172.21.15.104:0/55806039 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f84b8067fc0 0x7f84b806a480 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f84c400b7f0 tx=0x7f84c4008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:08:05.367 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.347+0000 7f100e7fc640 1 -- 172.21.15.104:0/3841593285 <== osd.6 v2:172.21.15.171:6824/436742919 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f0ffc072c30 con 0x7f0ffc070160 2024-03-28T14:08:05.367 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.347+0000 7f89a7fff640 1 --2- 172.21.15.104:0/1108949963 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f898c067fc0 0x7f898c06a480 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f899401c420 tx=0x7f89940a6000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:08:05.367 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.347+0000 7f84ca7fc640 1 -- 172.21.15.104:0/55806039 <== osd.1 v2:172.21.15.104:6808/19438674 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f84b8072c90 con 0x7f84b80701c0 2024-03-28T14:08:05.368 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.355+0000 7f89a57fa640 1 -- 172.21.15.104:0/1108949963 <== osd.0 v2:172.21.15.104:6800/3745690918 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f898c072c90 con 0x7f898c0701c0 2024-03-28T14:08:05.368 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.355+0000 7f4d9726f640 1 -- 172.21.15.104:0/2140275844 --> [v2:172.21.15.171:6808/2934826925,v1:172.21.15.171:6810/2934826925] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f4d900687c0 con 0x7f4d9013f870 2024-03-28T14:08:05.368 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.355+0000 7f4d85ffb640 1 -- 172.21.15.104:0/2140275844 <== osd.5 v2:172.21.15.171:6808/2934826925 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f4d900687c0 con 0x7f4d9013f870 2024-03-28T14:08:05.369 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.355+0000 7f4d9726f640 1 -- 172.21.15.104:0/2140275844 >> [v2:172.21.15.171:6808/2934826925,v1:172.21.15.171:6810/2934826925] conn(0x7f4d9013f870 msgr2=0x7f4d9013d6b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:05.369 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.355+0000 7f4d9726f640 1 --2- 172.21.15.104:0/2140275844 >> [v2:172.21.15.171:6808/2934826925,v1:172.21.15.171:6810/2934826925] conn(0x7f4d9013f870 0x7f4d9013d6b0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:05.369 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.359+0000 7f4d9726f640 1 -- 172.21.15.104:0/2140275844 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f4d74068010 msgr2=0x7f4d7406a4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:05.369 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.359+0000 7f4d9726f640 1 --2- 172.21.15.104:0/2140275844 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f4d74068010 0x7f4d7406a4d0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f4d800b1000 tx=0x7f4d80008040 comp rx=0 tx=0).stop 2024-03-28T14:08:05.369 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.359+0000 7f4d9726f640 1 -- 172.21.15.104:0/2140275844 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4d900733f0 msgr2=0x7f4d900b2850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:05.370 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.359+0000 7f4d9726f640 1 --2- 172.21.15.104:0/2140275844 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4d900733f0 0x7f4d900b2850 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f4d8c009e70 tx=0x7f4d8c00c520 comp rx=0 tx=0).stop 2024-03-28T14:08:05.370 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.359+0000 7f4d9580c640 1 -- 172.21.15.104:0/2140275844 reap_dead start 2024-03-28T14:08:05.370 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.359+0000 7f4d9726f640 1 -- 172.21.15.104:0/2140275844 shutdown_connections 2024-03-28T14:08:05.370 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.359+0000 7f4d9726f640 1 -- 172.21.15.104:0/2140275844 >> 172.21.15.104:0/2140275844 conn(0x7f4d900ba390 msgr2=0x7f4d90143a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T14:08:05.370 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.359+0000 7f4d9726f640 1 -- 172.21.15.104:0/2140275844 shutdown_connections 2024-03-28T14:08:05.370 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.359+0000 7f4d9726f640 1 -- 172.21.15.104:0/2140275844 wait complete. 2024-03-28T14:08:05.371 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.359+0000 7f5a8400b640 1 Processor -- start 2024-03-28T14:08:05.371 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.359+0000 7f5a8400b640 1 -- start start 2024-03-28T14:08:05.371 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.359+0000 7f5a8400b640 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f5a7c09deb0 0x7f5a7c146e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:05.371 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.359+0000 7f5a8400b640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5a7c09e290 con 0x7f5a7c0aeba0 2024-03-28T14:08:05.371 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.359+0000 7f5a8400b640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5a7c0afb10 con 0x7f5a7c0b2e60 2024-03-28T14:08:05.371 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.359+0000 7f5a8400b640 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 -- 0x7f5a7c0b47f0 con 0x7f5a7c09deb0 2024-03-28T14:08:05.371 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.359+0000 7f5a815a6640 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f5a7c0aeba0 0x7f5a7c0b2750 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:47328/0 (socket says 172.21.15.104:47328) 2024-03-28T14:08:05.372 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.359+0000 7f5a815a6640 1 -- 172.21.15.104:0/2827658942 learned_addr learned my addr 172.21.15.104:0/2827658942 (peer_addr_for_me v1:172.21.15.104:0/0) 2024-03-28T14:08:05.372 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.359+0000 7f5a81da7640 1 --2- 172.21.15.104:0/2827658942 >> v2:172.21.15.104:3300/0 conn(0x7f5a7c09deb0 0x7f5a7c146e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:05.372 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.359+0000 7f5a81da7640 1 -- 172.21.15.104:0/2827658942 >> v1:172.21.15.171:6789/0 conn(0x7f5a7c0b2e60 legacy=0x7f5a7c14e190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:05.372 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.359+0000 7f5a81da7640 1 -- 172.21.15.104:0/2827658942 >> v1:172.21.15.104:6789/0 conn(0x7f5a7c0aeba0 legacy=0x7f5a7c0b2750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:05.372 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.359+0000 7f5a81da7640 1 -- 172.21.15.104:0/2827658942 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5a7c0afdf0 con 0x7f5a7c09deb0 2024-03-28T14:08:05.372 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.363+0000 7f5a81da7640 1 --2- 172.21.15.104:0/2827658942 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5a7c09deb0 0x7f5a7c146e60 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f5a7c0b4970 tx=0x7f5a6c01ea00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b9791c4b967f9bc3 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:08:05.372 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.367+0000 7f5a80da5640 1 -- 172.21.15.104:0/2827658942 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a6c0ad070 con 0x7f5a7c09deb0 2024-03-28T14:08:05.372 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.367+0000 7f5a80da5640 1 -- 172.21.15.104:0/2827658942 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5a6c0a78d0 con 0x7f5a7c09deb0 2024-03-28T14:08:05.372 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.367+0000 7f5a8400b640 1 -- 172.21.15.104:0/2827658942 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5a7c09deb0 msgr2=0x7f5a7c146e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:05.372 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.367+0000 7f5a8400b640 1 --2- 172.21.15.104:0/2827658942 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5a7c09deb0 0x7f5a7c146e60 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f5a7c0b4970 tx=0x7f5a6c01ea00 comp rx=0 tx=0).stop 2024-03-28T14:08:05.373 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.367+0000 7f71a9754640 1 -- 172.21.15.104:0/407177848 --> [v2:172.21.15.104:6824/2352012626,v1:172.21.15.104:6825/2352012626] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f7188005c50 con 0x7f7188001660 2024-03-28T14:08:05.373 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.367+0000 7f5a8400b640 1 -- 172.21.15.104:0/2827658942 shutdown_connections 2024-03-28T14:08:05.373 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.367+0000 7f5a8400b640 1 --2- 172.21.15.104:0/2827658942 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5a7c09deb0 0x7f5a7c146e60 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:05.373 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.367+0000 7f5a8400b640 1 -- 172.21.15.104:0/2827658942 >> 172.21.15.104:0/2827658942 conn(0x7f5a7c0b8d30 msgr2=0x7f5a7c0b9140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T14:08:05.373 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.367+0000 7f5a8400b640 1 -- 172.21.15.104:0/2827658942 shutdown_connections 2024-03-28T14:08:05.374 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.367+0000 7f44fe62b640 1 Processor -- start 2024-03-28T14:08:05.374 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.367+0000 7f44fe62b640 1 -- start start 2024-03-28T14:08:05.374 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.367+0000 7f71a17fa640 1 -- 172.21.15.104:0/407177848 <== osd.2 v2:172.21.15.104:6824/2352012626 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f7188005c50 con 0x7f7188001660 2024-03-28T14:08:05.375 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.367+0000 7f719a7fc640 1 -- 172.21.15.104:0/407177848 >> [v2:172.21.15.104:6824/2352012626,v1:172.21.15.104:6825/2352012626] conn(0x7f7188001660 msgr2=0x7f7188003b20 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:05.375 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.367+0000 7f719a7fc640 1 --2- 172.21.15.104:0/407177848 >> [v2:172.21.15.104:6824/2352012626,v1:172.21.15.104:6825/2352012626] conn(0x7f7188001660 0x7f7188003b20 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:05.375 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.367+0000 7f719a7fc640 1 -- 172.21.15.104:0/407177848 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f7184067fc0 msgr2=0x7f718406a480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:05.375 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.367+0000 7f719a7fc640 1 --2- 172.21.15.104:0/407177848 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f7184067fc0 0x7f718406a480 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f71a4087ad0 tx=0x7f718c006c00 comp rx=0 tx=0).stop 2024-03-28T14:08:05.375 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.367+0000 7f719a7fc640 1 -- 172.21.15.104:0/407177848 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f71a40add50 msgr2=0x7f71a414df70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:05.375 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.367+0000 7f719a7fc640 1 --2- 172.21.15.104:0/407177848 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f71a40add50 0x7f71a414df70 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f71940507b0 tx=0x7f7194050c70 comp rx=0 tx=0).stop 2024-03-28T14:08:05.375 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.367+0000 7f71a8f53640 1 -- 172.21.15.104:0/407177848 reap_dead start 2024-03-28T14:08:05.376 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.367+0000 7f719a7fc640 1 -- 172.21.15.104:0/407177848 shutdown_connections 2024-03-28T14:08:05.376 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.367+0000 7f719a7fc640 1 -- 172.21.15.104:0/407177848 >> 172.21.15.104:0/407177848 conn(0x7f71a406c530 msgr2=0x7f71a40b5c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T14:08:05.376 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.367+0000 7f719a7fc640 1 -- 172.21.15.104:0/407177848 shutdown_connections 2024-03-28T14:08:05.376 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.367+0000 7f719a7fc640 1 -- 172.21.15.104:0/407177848 wait complete. 2024-03-28T14:08:05.377 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.367+0000 7f5a8400b640 1 -- 172.21.15.104:0/2827658942 wait complete. 2024-03-28T14:08:05.377 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.367+0000 7f5a8400b640 1 Processor -- start 2024-03-28T14:08:05.377 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.367+0000 7f5a8400b640 1 -- start start 2024-03-28T14:08:05.378 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.371+0000 7f5a8400b640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f5a7c09deb0 0x7f5a7c13d1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:05.378 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.371+0000 7f5a8400b640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5a7c0b2e60 0x7f5a7c09c060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:05.378 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.371+0000 7f5a8400b640 1 --2- >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f5a7c09c5a0 0x7f5a7c09af00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:05.378 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.371+0000 7f5a8400b640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f5a7c1520d0 con 0x7f5a7c0b2e60 2024-03-28T14:08:05.378 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.371+0000 7f5a8400b640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f5a7c0af750 con 0x7f5a7c09deb0 2024-03-28T14:08:05.378 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.371+0000 7f5a8400b640 1 -- --> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] -- mon_getmap magic: 0 -- 0x7f5a7c152250 con 0x7f5a7c09c5a0 2024-03-28T14:08:05.378 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.371+0000 7f5a815a6640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5a7c0b2e60 0x7f5a7c09c060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:05.378 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.371+0000 7f5a815a6640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5a7c0b2e60 0x7f5a7c09c060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:42502/0 (socket says 172.21.15.104:42502) 2024-03-28T14:08:05.378 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.371+0000 7f5a815a6640 1 -- 172.21.15.104:0/3881840328 learned_addr learned my addr 172.21.15.104:0/3881840328 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-03-28T14:08:05.378 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.371+0000 7f5a81da7640 1 --2- 172.21.15.104:0/3881840328 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f5a7c09deb0 0x7f5a7c13d1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:05.378 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.371+0000 7f5a825a8640 1 --2- 172.21.15.104:0/3881840328 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f5a7c09c5a0 0x7f5a7c09af00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:05.378 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.371+0000 7f5a815a6640 1 -- 172.21.15.104:0/3881840328 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f5a7c09c5a0 msgr2=0x7f5a7c09af00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:05.378 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.371+0000 7f5a815a6640 1 --2- 172.21.15.104:0/3881840328 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f5a7c09c5a0 0x7f5a7c09af00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:05.379 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.371+0000 7f5a815a6640 1 -- 172.21.15.104:0/3881840328 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f5a7c09deb0 msgr2=0x7f5a7c13d1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:05.379 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.371+0000 7f5a815a6640 1 --2- 172.21.15.104:0/3881840328 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f5a7c09deb0 0x7f5a7c13d1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:05.379 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.371+0000 7f5a815a6640 1 -- 172.21.15.104:0/3881840328 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5a7c09cc80 con 0x7f5a7c0b2e60 2024-03-28T14:08:05.379 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.371+0000 7f5a815a6640 1 --2- 172.21.15.104:0/3881840328 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5a7c0b2e60 0x7f5a7c09c060 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f5a7800c6e0 tx=0x7f5a7800cba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:08:05.379 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.371+0000 7f5a727fc640 1 -- 172.21.15.104:0/3881840328 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a78007960 con 0x7f5a7c0b2e60 2024-03-28T14:08:05.379 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.371+0000 7f5a727fc640 1 -- 172.21.15.104:0/3881840328 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5a78008040 con 0x7f5a7c0b2e60 2024-03-28T14:08:05.381 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.371+0000 7f44fe62b640 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f44f80add50 0x7f44f80b89f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:05.381 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.371+0000 7f44fe62b640 1 --2- >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f44f808ba90 0x7f44f80b3b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:05.381 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.371+0000 7f44fd629640 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f44f80add50 0x7f44f80b89f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:05.381 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.371+0000 7f44fd629640 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f44f80add50 0x7f44f80b89f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:42500/0 (socket says 172.21.15.104:42500) 2024-03-28T14:08:05.381 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.371+0000 7f44fd629640 1 -- 172.21.15.104:0/3273991919 learned_addr learned my addr 172.21.15.104:0/3273991919 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-03-28T14:08:05.381 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.371+0000 7f44fe62b640 1 -- 172.21.15.104:0/3273991919 --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f44f80b0100 con 0x7f44f808b5e0 2024-03-28T14:08:05.382 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.371+0000 7f44fe62b640 1 -- 172.21.15.104:0/3273991919 --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 -- 0x7f44f80b03e0 con 0x7f44f80add50 2024-03-28T14:08:05.382 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.371+0000 7f44fe62b640 1 -- 172.21.15.104:0/3273991919 --> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] -- mon_getmap magic: 0 -- 0x7f44f80b4c20 con 0x7f44f808ba90 2024-03-28T14:08:05.382 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.371+0000 7f44fde2a640 1 --2- 172.21.15.104:0/3273991919 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f44f808ba90 0x7f44f80b3b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:05.382 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.371+0000 7f44effff640 1 -- 172.21.15.104:0/3273991919 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 335502102 0 0) 0x7f44f80b0100 con 0x7f44f808b5e0 2024-03-28T14:08:05.382 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.371+0000 7f44effff640 1 -- 172.21.15.104:0/3273991919 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f44e4014540 con 0x7f44f808b5e0 2024-03-28T14:08:05.382 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.371+0000 7f44fd629640 1 -- 172.21.15.104:0/3273991919 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f44f808ba90 msgr2=0x7f44f80b3b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T14:08:05.382 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.371+0000 7f44fd629640 1 --2- 172.21.15.104:0/3273991919 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f44f808ba90 0x7f44f80b3b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:05.382 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.371+0000 7f44fd629640 1 -- 172.21.15.104:0/3273991919 >> v1:172.21.15.171:6789/0 conn(0x7f44f808b5e0 legacy=0x7f44f80b3460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:05.382 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.371+0000 7f44fd629640 1 -- 172.21.15.104:0/3273991919 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f44f80b4f00 con 0x7f44f80add50 2024-03-28T14:08:05.383 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.371+0000 7f5a727fc640 1 -- 172.21.15.104:0/3881840328 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a78007d40 con 0x7f5a7c0b2e60 2024-03-28T14:08:05.383 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.371+0000 7f5a8400b640 1 -- 172.21.15.104:0/3881840328 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5a7c13c7b0 con 0x7f5a7c0b2e60 2024-03-28T14:08:05.383 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.371+0000 7f5a8400b640 1 -- 172.21.15.104:0/3881840328 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5a7c09b610 con 0x7f5a7c0b2e60 2024-03-28T14:08:05.383 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.375+0000 7f5a727fc640 1 -- 172.21.15.104:0/3881840328 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) ==== 87803+0+0 (secure 0 0 0) 0x7f5a7800f410 con 0x7f5a7c0b2e60 2024-03-28T14:08:05.383 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.375+0000 7f5a727fc640 1 --2- 172.21.15.104:0/3881840328 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f5a60067fc0 0x7f5a6006a480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:05.383 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.375+0000 7f5a727fc640 1 -- 172.21.15.104:0/3881840328 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f5a780865b0 con 0x7f5a7c0b2e60 2024-03-28T14:08:05.384 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.375+0000 7f44fd629640 1 --2- 172.21.15.104:0/3273991919 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f44f80add50 0x7f44f80b89f0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f44f4009620 tx=0x7f44f401c8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8290ec8666ef4234 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:08:05.384 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.375+0000 7f44effff640 1 -- 172.21.15.104:0/3273991919 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44f40ac020 con 0x7f44f80add50 2024-03-28T14:08:05.384 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.375+0000 7f44effff640 1 -- 172.21.15.104:0/3273991919 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f44f4005cc0 con 0x7f44f80add50 2024-03-28T14:08:05.384 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.375+0000 7f44effff640 1 -- 172.21.15.104:0/3273991919 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44f4007330 con 0x7f44f80add50 2024-03-28T14:08:05.384 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.375+0000 7f44fe62b640 1 -- 172.21.15.104:0/3273991919 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f44f80add50 msgr2=0x7f44f80b89f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:05.384 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.375+0000 7f44fe62b640 1 --2- 172.21.15.104:0/3273991919 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f44f80add50 0x7f44f80b89f0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f44f4009620 tx=0x7f44f401c8a0 comp rx=0 tx=0).stop 2024-03-28T14:08:05.385 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.375+0000 7f44fe62b640 1 -- 172.21.15.104:0/3273991919 shutdown_connections 2024-03-28T14:08:05.385 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.375+0000 7f44fe62b640 1 --2- 172.21.15.104:0/3273991919 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f44f808ba90 0x7f44f80b3b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:05.385 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.375+0000 7f44fe62b640 1 --2- 172.21.15.104:0/3273991919 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f44f80add50 0x7f44f80b89f0 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:05.385 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.375+0000 7f44fe62b640 1 -- 172.21.15.104:0/3273991919 >> 172.21.15.104:0/3273991919 conn(0x7f44f806c530 msgr2=0x7f44f806c940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T14:08:05.385 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.375+0000 7f44fe62b640 1 -- 172.21.15.104:0/3273991919 shutdown_connections 2024-03-28T14:08:05.385 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.375+0000 7f44fe62b640 1 -- 172.21.15.104:0/3273991919 wait complete. 2024-03-28T14:08:05.385 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.375+0000 7f5a81da7640 1 --2- 172.21.15.104:0/3881840328 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f5a60067fc0 0x7f5a6006a480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:05.386 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.379+0000 7f101fc9d640 1 -- 172.21.15.104:0/3841593285 --> [v2:172.21.15.171:6824/436742919,v1:172.21.15.171:6825/436742919] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f10180b10c0 con 0x7f0ffc070160 2024-03-28T14:08:05.386 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.379+0000 7f5a8400b640 1 --2- 172.21.15.104:0/3881840328 >> [v2:172.21.15.171:6800/2107001077,v1:172.21.15.171:6801/2107001077] conn(0x7f5a7c0aeba0 0x7f5a7c13af00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:05.386 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.371+0000 7f8e95590640 1 Processor -- start 2024-03-28T14:08:05.387 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.379+0000 7f5a8400b640 1 -- 172.21.15.104:0/3881840328 --> [v2:172.21.15.171:6800/2107001077,v1:172.21.15.171:6801/2107001077] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f5a7c13dcd0 con 0x7f5a7c0aeba0 2024-03-28T14:08:05.387 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.379+0000 7f89addc5640 1 -- 172.21.15.104:0/1108949963 --> [v2:172.21.15.104:6800/3745690918,v1:172.21.15.104:6801/3745690918] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f89a001f370 con 0x7f898c0701c0 2024-03-28T14:08:05.388 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.371+0000 7f8e95590640 1 -- start start 2024-03-28T14:08:05.388 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.371+0000 7f8e95590640 1 --2- >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f8e9006c8f0 0x7f8e900adff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:05.388 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.379+0000 7f84dac14640 1 -- 172.21.15.104:0/55806039 --> [v2:172.21.15.104:6808/19438674,v1:172.21.15.104:6810/19438674] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f84d40b6be0 con 0x7f84b80701c0 2024-03-28T14:08:05.389 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.371+0000 7f8e95590640 1 --2- >> v2:172.21.15.171:3300/0 conn(0x7f8e900af180 0x7f8e9008b4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:05.389 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.371+0000 7f8e95590640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8e9006ccd0 con 0x7f8e9008ba10 2024-03-28T14:08:05.389 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.371+0000 7f8e95590640 1 -- --> v2:172.21.15.171:3300/0 -- mon_getmap magic: 0 -- 0x7f8e9009e3c0 con 0x7f8e900af180 2024-03-28T14:08:05.390 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.379+0000 7f89a57fa640 1 -- 172.21.15.104:0/1108949963 <== osd.0 v2:172.21.15.104:6800/3745690918 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f89a001f370 con 0x7f898c0701c0 2024-03-28T14:08:05.390 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.379+0000 7f5a825a8640 1 --2- 172.21.15.104:0/3881840328 >> [v2:172.21.15.171:6800/2107001077,v1:172.21.15.171:6801/2107001077] conn(0x7f5a7c0aeba0 0x7f5a7c13af00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:05.391 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.379+0000 7f100e7fc640 1 -- 172.21.15.104:0/3841593285 <== osd.6 v2:172.21.15.171:6824/436742919 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f10180b10c0 con 0x7f0ffc070160 2024-03-28T14:08:05.391 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.379+0000 7f84ca7fc640 1 -- 172.21.15.104:0/55806039 <== osd.1 v2:172.21.15.104:6808/19438674 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f84d40b6be0 con 0x7f84b80701c0 2024-03-28T14:08:05.392 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.379+0000 7f5a81da7640 1 --2- 172.21.15.104:0/3881840328 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f5a60067fc0 0x7f5a6006a480 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f5a6c0b2000 tx=0x7f5a6c009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:08:05.392 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.379+0000 7f5a825a8640 1 --2- 172.21.15.104:0/3881840328 >> [v2:172.21.15.171:6800/2107001077,v1:172.21.15.171:6801/2107001077] conn(0x7f5a7c0aeba0 0x7f5a7c13af00 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:08:05.393 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.379+0000 7f89addc5640 1 -- 172.21.15.104:0/1108949963 >> [v2:172.21.15.104:6800/3745690918,v1:172.21.15.104:6801/3745690918] conn(0x7f898c0701c0 msgr2=0x7f898c0725e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:05.393 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.379+0000 7f89addc5640 1 --2- 172.21.15.104:0/1108949963 >> [v2:172.21.15.104:6800/3745690918,v1:172.21.15.104:6801/3745690918] conn(0x7f898c0701c0 0x7f898c0725e0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:05.393 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.379+0000 7f44fe62b640 1 Processor -- start 2024-03-28T14:08:05.394 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.379+0000 7f89addc5640 1 -- 172.21.15.104:0/1108949963 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f898c067fc0 msgr2=0x7f898c06a480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:05.394 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.379+0000 7f89addc5640 1 --2- 172.21.15.104:0/1108949963 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f898c067fc0 0x7f898c06a480 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f899401c420 tx=0x7f89940a6000 comp rx=0 tx=0).stop 2024-03-28T14:08:05.394 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.379+0000 7f89addc5640 1 -- 172.21.15.104:0/1108949963 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f89a000c360 msgr2=0x7f89a0024d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:05.394 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.379+0000 7f89addc5640 1 --2- 172.21.15.104:0/1108949963 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f89a000c360 0x7f89a0024d10 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f899c00c390 tx=0x7f899c00c850 comp rx=0 tx=0).stop 2024-03-28T14:08:05.394 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.379+0000 7f89ad5c4640 1 -- 172.21.15.104:0/1108949963 reap_dead start 2024-03-28T14:08:05.394 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.379+0000 7f89addc5640 1 -- 172.21.15.104:0/1108949963 shutdown_connections 2024-03-28T14:08:05.394 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.379+0000 7f89addc5640 1 -- 172.21.15.104:0/1108949963 >> 172.21.15.104:0/1108949963 conn(0x7f89a000b5e0 msgr2=0x7f89a001d9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T14:08:05.394 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.379+0000 7f89addc5640 1 -- 172.21.15.104:0/1108949963 shutdown_connections 2024-03-28T14:08:05.394 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.379+0000 7f89addc5640 1 -- 172.21.15.104:0/1108949963 wait complete. 2024-03-28T14:08:05.395 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.371+0000 7f8e95590640 1 -- --> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] -- mon_getmap magic: 0 -- 0x7f8e900af6d0 con 0x7f8e9006c8f0 2024-03-28T14:08:05.395 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.371+0000 7f8e8effd640 1 --2- >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f8e9006c8f0 0x7f8e900adff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:05.395 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.371+0000 7f8e8effd640 1 --2- >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f8e9006c8f0 0x7f8e900adff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.171:3301/0 says I am v2:172.21.15.104:56010/0 (socket says 172.21.15.104:56010) 2024-03-28T14:08:05.395 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.371+0000 7f8e8effd640 1 -- 172.21.15.104:0/3706561960 learned_addr learned my addr 172.21.15.104:0/3706561960 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-03-28T14:08:05.395 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.371+0000 7f8e87fff640 1 --2- 172.21.15.104:0/3706561960 >> v2:172.21.15.171:3300/0 conn(0x7f8e900af180 0x7f8e9008b4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:05.395 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.371+0000 7f8e8e7fc640 1 -- 172.21.15.104:0/3706561960 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1145752457 0 0) 0x7f8e9006ccd0 con 0x7f8e9008ba10 2024-03-28T14:08:05.395 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.379+0000 7f8e8e7fc640 1 -- 172.21.15.104:0/3706561960 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8e7c007940 con 0x7f8e9008ba10 2024-03-28T14:08:05.395 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.379+0000 7f8e87fff640 1 -- 172.21.15.104:0/3706561960 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f8e9006c8f0 msgr2=0x7f8e900adff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T14:08:05.396 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.379+0000 7f8e87fff640 1 --2- 172.21.15.104:0/3706561960 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f8e9006c8f0 0x7f8e900adff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:05.396 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.379+0000 7f8e87fff640 1 -- 172.21.15.104:0/3706561960 >> v1:172.21.15.171:6789/0 conn(0x7f8e9008ba10 legacy=0x7f8e900b3730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:05.396 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.379+0000 7f8e87fff640 1 -- 172.21.15.104:0/3706561960 --> v2:172.21.15.171:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8e900af9b0 con 0x7f8e900af180 2024-03-28T14:08:05.396 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.379+0000 7f44fe62b640 1 -- start start 2024-03-28T14:08:05.396 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.379+0000 7f44fe62b640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f44f808b5e0 0x7f44f8156d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:05.396 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.379+0000 7f44fd629640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f44f808b5e0 0x7f44f8156d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:05.396 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.379+0000 7f44fd629640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f44f808b5e0 0x7f44f8156d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:42512/0 (socket says 172.21.15.104:42512) 2024-03-28T14:08:05.397 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.379+0000 7f44fd629640 1 -- 172.21.15.104:0/1187803630 learned_addr learned my addr 172.21.15.104:0/1187803630 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-03-28T14:08:05.397 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.379+0000 7f44fe62b640 1 --2- 172.21.15.104:0/1187803630 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f44f808ba90 0x7f44f806fb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:05.397 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.379+0000 7f44fe62b640 1 --2- 172.21.15.104:0/1187803630 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f44f80add50 0x7f44f8070090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:05.397 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.379+0000 7f84b7fff640 1 -- 172.21.15.104:0/55806039 >> [v2:172.21.15.104:6808/19438674,v1:172.21.15.104:6810/19438674] conn(0x7f84b80701c0 msgr2=0x7f84b80725e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:05.397 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.379+0000 7f84b7fff640 1 --2- 172.21.15.104:0/55806039 >> [v2:172.21.15.104:6808/19438674,v1:172.21.15.104:6810/19438674] conn(0x7f84b80701c0 0x7f84b80725e0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:05.398 INFO:teuthology.orchestra.run.smithi104.stdout:55834574854 2024-03-28T14:08:05.398 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-03-28T14:08:05.399 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.379+0000 7f8e87fff640 1 --2- 172.21.15.104:0/3706561960 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f8e900af180 0x7f8e9008b4d0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f8e8000b8a0 tx=0x7f8e8001c8f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8d99743274d3cae3 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:08:05.399 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.383+0000 7f84b7fff640 1 -- 172.21.15.104:0/55806039 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f84b8067fc0 msgr2=0x7f84b806a480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:05.399 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.383+0000 7f84b7fff640 1 --2- 172.21.15.104:0/55806039 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f84b8067fc0 0x7f84b806a480 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f84c400b7f0 tx=0x7f84c4008040 comp rx=0 tx=0).stop 2024-03-28T14:08:05.400 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.383+0000 7f84b7fff640 1 -- 172.21.15.104:0/55806039 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f84d40b47a0 msgr2=0x7f84d4138cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:05.400 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.383+0000 7f84b7fff640 1 --2- 172.21.15.104:0/55806039 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f84d40b47a0 0x7f84d4138cc0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f84cc00b9e0 tx=0x7f84cc00bea0 comp rx=0 tx=0).stop 2024-03-28T14:08:05.400 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.383+0000 7f84da413640 1 -- 172.21.15.104:0/55806039 reap_dead start 2024-03-28T14:08:05.400 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.383+0000 7f84b7fff640 1 -- 172.21.15.104:0/55806039 shutdown_connections 2024-03-28T14:08:05.400 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.383+0000 7f84b7fff640 1 -- 172.21.15.104:0/55806039 >> 172.21.15.104:0/55806039 conn(0x7f84d406c530 msgr2=0x7f84d40b73e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T14:08:05.400 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.383+0000 7f8e8e7fc640 1 -- 172.21.15.104:0/3706561960 <== mon.1 v2:172.21.15.171:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e800b2070 con 0x7f8e900af180 2024-03-28T14:08:05.400 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.383+0000 7f5a727fc640 1 -- 172.21.15.104:0/3881840328 <== osd.4 v2:172.21.15.171:6800/2107001077 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f5a7c13dcd0 con 0x7f5a7c0aeba0 2024-03-28T14:08:05.401 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.379+0000 7f44fe62b640 1 -- 172.21.15.104:0/1187803630 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f44f81596a0 con 0x7f44f808b5e0 2024-03-28T14:08:05.401 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.379+0000 7f44fe62b640 1 -- 172.21.15.104:0/1187803630 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f44f8159520 con 0x7f44f80add50 2024-03-28T14:08:05.401 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.379+0000 7f44fe62b640 1 -- 172.21.15.104:0/1187803630 --> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] -- mon_getmap magic: 0 -- 0x7f44f8159820 con 0x7f44f808ba90 2024-03-28T14:08:05.401 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.383+0000 7f44fce28640 1 --2- 172.21.15.104:0/1187803630 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f44f808ba90 0x7f44f806fb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:05.402 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.383+0000 7f44fde2a640 1 --2- 172.21.15.104:0/1187803630 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f44f80add50 0x7f44f8070090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:05.402 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.383+0000 7f44fd629640 1 -- 172.21.15.104:0/1187803630 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f44f808ba90 msgr2=0x7f44f806fb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:05.402 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.383+0000 7f44fd629640 1 --2- 172.21.15.104:0/1187803630 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f44f808ba90 0x7f44f806fb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:05.402 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.383+0000 7f44fd629640 1 -- 172.21.15.104:0/1187803630 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f44f80add50 msgr2=0x7f44f8070090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:05.402 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.383+0000 7f44fd629640 1 --2- 172.21.15.104:0/1187803630 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f44f80add50 0x7f44f8070090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:05.402 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.383+0000 7f44fd629640 1 -- 172.21.15.104:0/1187803630 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f44f8072d50 con 0x7f44f808b5e0 2024-03-28T14:08:05.402 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.383+0000 7f44fd629640 1 --2- 172.21.15.104:0/1187803630 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f44f808b5e0 0x7f44f8156d50 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f44f401a4d0 tx=0x7f44f4019640 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:08:05.402 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.383+0000 7f44edffb640 1 -- 172.21.15.104:0/1187803630 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44f40ac070 con 0x7f44f808b5e0 2024-03-28T14:08:05.402 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.383+0000 7f44edffb640 1 -- 172.21.15.104:0/1187803630 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f44f40b5070 con 0x7f44f808b5e0 2024-03-28T14:08:05.402 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.383+0000 7f44edffb640 1 -- 172.21.15.104:0/1187803630 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44f4007740 con 0x7f44f808b5e0 2024-03-28T14:08:05.403 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.383+0000 7f8e8e7fc640 1 -- 172.21.15.104:0/3706561960 <== mon.1 v2:172.21.15.171:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8e800a5880 con 0x7f8e900af180 2024-03-28T14:08:05.403 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.383+0000 7f8e95590640 1 -- 172.21.15.104:0/3706561960 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f8e900af180 msgr2=0x7f8e9008b4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:05.403 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.383+0000 7f8e95590640 1 --2- 172.21.15.104:0/3706561960 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f8e900af180 0x7f8e9008b4d0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f8e8000b8a0 tx=0x7f8e8001c8f0 comp rx=0 tx=0).stop 2024-03-28T14:08:05.403 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.383+0000 7f8e95590640 1 -- 172.21.15.104:0/3706561960 shutdown_connections 2024-03-28T14:08:05.404 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.383+0000 7f8e95590640 1 --2- 172.21.15.104:0/3706561960 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f8e900af180 0x7f8e9008b4d0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:05.404 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.383+0000 7f8e95590640 1 --2- 172.21.15.104:0/3706561960 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f8e9006c8f0 0x7f8e900adff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:05.404 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.383+0000 7f8e95590640 1 -- 172.21.15.104:0/3706561960 >> 172.21.15.104:0/3706561960 conn(0x7f8e900b8d30 msgr2=0x7f8e900b9140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T14:08:05.404 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.383+0000 7f8e95590640 1 -- 172.21.15.104:0/3706561960 shutdown_connections 2024-03-28T14:08:05.404 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.383+0000 7f8e95590640 1 -- 172.21.15.104:0/3706561960 wait complete. 2024-03-28T14:08:05.404 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.383+0000 7f8e95590640 1 Processor -- start 2024-03-28T14:08:05.404 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.383+0000 7f8e95590640 1 -- start start 2024-03-28T14:08:05.404 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.383+0000 7f44fe62b640 1 -- 172.21.15.104:0/1187803630 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f44f8071b40 con 0x7f44f808b5e0 2024-03-28T14:08:05.404 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.383+0000 7f44fe62b640 1 -- 172.21.15.104:0/1187803630 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f44f809af40 con 0x7f44f808b5e0 2024-03-28T14:08:05.404 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.383+0000 7f44fe62b640 1 -- 172.21.15.104:0/1187803630 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f44f8068640 con 0x7f44f808b5e0 2024-03-28T14:08:05.405 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.383+0000 7f44edffb640 1 -- 172.21.15.104:0/1187803630 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) ==== 87803+0+0 (secure 0 0 0) 0x7f44f40af020 con 0x7f44f808b5e0 2024-03-28T14:08:05.405 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.383+0000 7f8e95590640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f8e9006c8f0 0x7f8e901f16a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:05.405 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.387+0000 7f8e95590640 1 --2- >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f8e9008ba10 0x7f8e90201320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:05.405 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.387+0000 7f8e95590640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8e900af180 0x7f8e90201860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:05.405 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.387+0000 7f8e95590640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f8e900ae530 con 0x7f8e900af180 2024-03-28T14:08:05.405 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.387+0000 7f8e95590640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f8e900b5490 con 0x7f8e9006c8f0 2024-03-28T14:08:05.405 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.387+0000 7f8e95590640 1 -- --> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] -- mon_getmap magic: 0 -- 0x7f8e900b3010 con 0x7f8e9008ba10 2024-03-28T14:08:05.405 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.387+0000 7f44edffb640 1 --2- 172.21.15.104:0/1187803630 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f44dc067fc0 0x7f44dc06a480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:05.406 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.387+0000 7f8e8f7fe640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8e900af180 0x7f8e90201860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:05.406 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.387+0000 7f8e8f7fe640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8e900af180 0x7f8e90201860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:42524/0 (socket says 172.21.15.104:42524) 2024-03-28T14:08:05.406 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.387+0000 7f8e8f7fe640 1 -- 172.21.15.104:0/1621002039 learned_addr learned my addr 172.21.15.104:0/1621002039 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-03-28T14:08:05.406 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.387+0000 7f8e8effd640 1 --2- 172.21.15.104:0/1621002039 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f8e9006c8f0 0x7f8e901f16a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:05.407 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.387+0000 7f44edffb640 1 -- 172.21.15.104:0/1187803630 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f44f411fb30 con 0x7f44f808b5e0 2024-03-28T14:08:05.407 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.387+0000 7f44edffb640 1 --2- 172.21.15.104:0/1187803630 >> [v2:172.21.15.104:6809/3568347381,v1:172.21.15.104:6811/3568347381] conn(0x7f44dc0701c0 0x7f44dc0725e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:05.407 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.387+0000 7f44edffb640 1 -- 172.21.15.104:0/1187803630 --> [v2:172.21.15.104:6809/3568347381,v1:172.21.15.104:6811/3568347381] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f44f40b3e50 con 0x7f44dc0701c0 2024-03-28T14:08:05.407 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.387+0000 7f44edffb640 1 -- 172.21.15.104:0/1187803630 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f44f41242c0 con 0x7f44f808b5e0 2024-03-28T14:08:05.407 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.387+0000 7f44fde2a640 1 --2- 172.21.15.104:0/1187803630 >> [v2:172.21.15.104:6809/3568347381,v1:172.21.15.104:6811/3568347381] conn(0x7f44dc0701c0 0x7f44dc0725e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:05.407 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.387+0000 7f44fce28640 1 --2- 172.21.15.104:0/1187803630 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f44dc067fc0 0x7f44dc06a480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:05.407 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.387+0000 7f0ffbfff640 1 -- 172.21.15.104:0/3841593285 >> [v2:172.21.15.171:6824/436742919,v1:172.21.15.171:6825/436742919] conn(0x7f0ffc070160 msgr2=0x7f0ffc072580 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:05.408 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.387+0000 7f0ffbfff640 1 --2- 172.21.15.104:0/3841593285 >> [v2:172.21.15.171:6824/436742919,v1:172.21.15.171:6825/436742919] conn(0x7f0ffc070160 0x7f0ffc072580 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:05.408 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.387+0000 7f8e8f7fe640 1 -- 172.21.15.104:0/1621002039 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f8e9008ba10 msgr2=0x7f8e90201320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:05.408 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.387+0000 7f8e8f7fe640 1 --2- 172.21.15.104:0/1621002039 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f8e9008ba10 0x7f8e90201320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:05.408 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.387+0000 7f8e8f7fe640 1 -- 172.21.15.104:0/1621002039 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f8e9006c8f0 msgr2=0x7f8e901f16a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:05.408 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.387+0000 7f8e8f7fe640 1 --2- 172.21.15.104:0/1621002039 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f8e9006c8f0 0x7f8e901f16a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:05.408 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.387+0000 7f8e8f7fe640 1 -- 172.21.15.104:0/1621002039 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8e901fe340 con 0x7f8e900af180 2024-03-28T14:08:05.409 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.387+0000 7f0ffbfff640 1 -- 172.21.15.104:0/3841593285 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f0ffc067fc0 msgr2=0x7f0ffc06a480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:05.409 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.387+0000 7f0ffbfff640 1 --2- 172.21.15.104:0/3841593285 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f0ffc067fc0 0x7f0ffc06a480 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f101400ee10 tx=0x7f1014010040 comp rx=0 tx=0).stop 2024-03-28T14:08:05.409 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.387+0000 7f0ffbfff640 1 -- 172.21.15.104:0/3841593285 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f10180af180 msgr2=0x7f101809cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:05.409 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.387+0000 7f0ffbfff640 1 --2- 172.21.15.104:0/3841593285 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f10180af180 0x7f101809cc60 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f1010052fd0 tx=0x7f10100504a0 comp rx=0 tx=0).stop 2024-03-28T14:08:05.409 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.387+0000 7f101e23a640 1 -- 172.21.15.104:0/3841593285 reap_dead start 2024-03-28T14:08:05.409 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.387+0000 7f8e8f7fe640 1 --2- 172.21.15.104:0/1621002039 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8e900af180 0x7f8e90201860 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f8e8800c730 tx=0x7f8e8800cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:08:05.410 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.387+0000 7f84b7fff640 1 -- 172.21.15.104:0/55806039 shutdown_connections 2024-03-28T14:08:05.410 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.387+0000 7f84b7fff640 1 -- 172.21.15.104:0/55806039 wait complete. 2024-03-28T14:08:05.410 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.387+0000 7f44fde2a640 1 --2- 172.21.15.104:0/1187803630 >> [v2:172.21.15.104:6809/3568347381,v1:172.21.15.104:6811/3568347381] conn(0x7f44dc0701c0 0x7f44dc0725e0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:08:05.414 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.387+0000 7f8e877fe640 1 -- 172.21.15.104:0/1621002039 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e88003b80 con 0x7f8e900af180 2024-03-28T14:08:05.414 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.387+0000 7f8e877fe640 1 -- 172.21.15.104:0/1621002039 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8e88003d40 con 0x7f8e900af180 2024-03-28T14:08:05.415 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.387+0000 7f8e95590640 1 -- 172.21.15.104:0/1621002039 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8e901fe620 con 0x7f8e900af180 2024-03-28T14:08:05.415 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.387+0000 7f8e877fe640 1 -- 172.21.15.104:0/1621002039 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e8800aa30 con 0x7f8e900af180 2024-03-28T14:08:05.415 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.387+0000 7f8e95590640 1 -- 172.21.15.104:0/1621002039 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8e901fddb0 con 0x7f8e900af180 2024-03-28T14:08:05.415 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.387+0000 7f44fce28640 1 --2- 172.21.15.104:0/1187803630 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f44dc067fc0 0x7f44dc06a480 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f44f809e820 tx=0x7f44e8009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:08:05.415 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.387+0000 7f44edffb640 1 -- 172.21.15.104:0/1187803630 <== osd.3 v2:172.21.15.104:6809/3568347381 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f44f40b3e50 con 0x7f44dc0701c0 2024-03-28T14:08:05.415 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.387+0000 7f8e95590640 1 -- 172.21.15.104:0/1621002039 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f8e900687c0 con 0x7f8e900af180 2024-03-28T14:08:05.416 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.391+0000 7f0ffbfff640 1 -- 172.21.15.104:0/3841593285 shutdown_connections 2024-03-28T14:08:05.416 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.391+0000 7f0ffbfff640 1 -- 172.21.15.104:0/3841593285 >> 172.21.15.104:0/3841593285 conn(0x7f10180b8d30 msgr2=0x7f1018154c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T14:08:05.416 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.391+0000 7f8e877fe640 1 -- 172.21.15.104:0/1621002039 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) ==== 87803+0+0 (secure 0 0 0) 0x7f8e8800abd0 con 0x7f8e900af180 2024-03-28T14:08:05.416 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.391+0000 7f0ffbfff640 1 -- 172.21.15.104:0/3841593285 shutdown_connections 2024-03-28T14:08:05.417 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.391+0000 7f0ffbfff640 1 -- 172.21.15.104:0/3841593285 wait complete. 2024-03-28T14:08:05.418 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.391+0000 7f8e877fe640 1 --2- 172.21.15.104:0/1621002039 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f8e70067fc0 0x7f8e7006a480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:05.418 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.391+0000 7f8e877fe640 1 -- 172.21.15.104:0/1621002039 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f8e88086ca0 con 0x7f8e900af180 2024-03-28T14:08:05.418 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.391+0000 7f8e877fe640 1 --2- 172.21.15.104:0/1621002039 >> [v2:172.21.15.171:6809/2055631175,v1:172.21.15.171:6811/2055631175] conn(0x7f8e700701c0 0x7f8e700725e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:05.418 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.391+0000 7f8e877fe640 1 -- 172.21.15.104:0/1621002039 --> [v2:172.21.15.171:6809/2055631175,v1:172.21.15.171:6811/2055631175] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f8e8800bf30 con 0x7f8e700701c0 2024-03-28T14:08:05.418 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.391+0000 7f8e877fe640 1 -- 172.21.15.104:0/1621002039 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f8e88087130 con 0x7f8e900af180 2024-03-28T14:08:05.418 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.391+0000 7f8e8effd640 1 --2- 172.21.15.104:0/1621002039 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f8e70067fc0 0x7f8e7006a480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:05.418 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.391+0000 7f8e87fff640 1 --2- 172.21.15.104:0/1621002039 >> [v2:172.21.15.171:6809/2055631175,v1:172.21.15.171:6811/2055631175] conn(0x7f8e700701c0 0x7f8e700725e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:05.419 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.391+0000 7f8e8effd640 1 --2- 172.21.15.104:0/1621002039 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f8e70067fc0 0x7f8e7006a480 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f8e9009d570 tx=0x7f8e7800c040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:08:05.419 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.391+0000 7f8e87fff640 1 --2- 172.21.15.104:0/1621002039 >> [v2:172.21.15.171:6809/2055631175,v1:172.21.15.171:6811/2055631175] conn(0x7f8e700701c0 0x7f8e700725e0 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:08:05.419 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.395+0000 7f8e877fe640 1 -- 172.21.15.104:0/1621002039 <== osd.7 v2:172.21.15.171:6809/2055631175 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f8e8800bf30 con 0x7f8e700701c0 2024-03-28T14:08:05.420 INFO:teuthology.orchestra.run.smithi104.stdout:55834574855 2024-03-28T14:08:05.420 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-03-28T14:08:05.421 INFO:teuthology.orchestra.run.smithi104.stdout:55834574856 2024-03-28T14:08:05.421 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-03-28T14:08:05.422 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.407+0000 7f5a8400b640 1 -- 172.21.15.104:0/3881840328 --> [v2:172.21.15.171:6800/2107001077,v1:172.21.15.171:6801/2107001077] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f5a7c144b30 con 0x7f5a7c0aeba0 2024-03-28T14:08:05.422 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.407+0000 7f5a727fc640 1 -- 172.21.15.104:0/3881840328 <== osd.4 v2:172.21.15.171:6800/2107001077 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f5a7c144b30 con 0x7f5a7c0aeba0 2024-03-28T14:08:05.424 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.407+0000 7f5a8400b640 1 -- 172.21.15.104:0/3881840328 >> [v2:172.21.15.171:6800/2107001077,v1:172.21.15.171:6801/2107001077] conn(0x7f5a7c0aeba0 msgr2=0x7f5a7c13af00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:05.424 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.407+0000 7f5a8400b640 1 --2- 172.21.15.104:0/3881840328 >> [v2:172.21.15.171:6800/2107001077,v1:172.21.15.171:6801/2107001077] conn(0x7f5a7c0aeba0 0x7f5a7c13af00 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:05.424 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.407+0000 7f5a8400b640 1 -- 172.21.15.104:0/3881840328 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f5a60067fc0 msgr2=0x7f5a6006a480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:05.424 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.407+0000 7f5a8400b640 1 --2- 172.21.15.104:0/3881840328 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f5a60067fc0 0x7f5a6006a480 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f5a6c0b2000 tx=0x7f5a6c009040 comp rx=0 tx=0).stop 2024-03-28T14:08:05.424 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.407+0000 7f5a8400b640 1 -- 172.21.15.104:0/3881840328 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5a7c0b2e60 msgr2=0x7f5a7c09c060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:05.424 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.407+0000 7f5a8400b640 1 --2- 172.21.15.104:0/3881840328 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5a7c0b2e60 0x7f5a7c09c060 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f5a7800c6e0 tx=0x7f5a7800cba0 comp rx=0 tx=0).stop 2024-03-28T14:08:05.424 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.407+0000 7f5a825a8640 1 -- 172.21.15.104:0/3881840328 reap_dead start 2024-03-28T14:08:05.424 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.407+0000 7f5a8400b640 1 -- 172.21.15.104:0/3881840328 shutdown_connections 2024-03-28T14:08:05.424 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.407+0000 7f5a8400b640 1 -- 172.21.15.104:0/3881840328 >> 172.21.15.104:0/3881840328 conn(0x7f5a7c0b8d30 msgr2=0x7f5a7c145420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T14:08:05.424 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.407+0000 7f5a8400b640 1 -- 172.21.15.104:0/3881840328 shutdown_connections 2024-03-28T14:08:05.424 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.407+0000 7f5a8400b640 1 -- 172.21.15.104:0/3881840328 wait complete. 2024-03-28T14:08:05.425 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.411+0000 7f8e95590640 1 -- 172.21.15.104:0/1621002039 --> [v2:172.21.15.171:6809/2055631175,v1:172.21.15.171:6811/2055631175] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f8e9009ccc0 con 0x7f8e700701c0 2024-03-28T14:08:05.425 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.411+0000 7f8e877fe640 1 -- 172.21.15.104:0/1621002039 <== osd.7 v2:172.21.15.171:6809/2055631175 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f8e800040c0 con 0x7f8e700701c0 2024-03-28T14:08:05.426 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.411+0000 7f8e95590640 1 -- 172.21.15.104:0/1621002039 >> [v2:172.21.15.171:6809/2055631175,v1:172.21.15.171:6811/2055631175] conn(0x7f8e700701c0 msgr2=0x7f8e700725e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:05.426 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.411+0000 7f8e95590640 1 --2- 172.21.15.104:0/1621002039 >> [v2:172.21.15.171:6809/2055631175,v1:172.21.15.171:6811/2055631175] conn(0x7f8e700701c0 0x7f8e700725e0 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:05.426 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.415+0000 7f8e95590640 1 -- 172.21.15.104:0/1621002039 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f8e70067fc0 msgr2=0x7f8e7006a480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:05.426 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.415+0000 7f8e95590640 1 --2- 172.21.15.104:0/1621002039 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f8e70067fc0 0x7f8e7006a480 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f8e9009d570 tx=0x7f8e7800c040 comp rx=0 tx=0).stop 2024-03-28T14:08:05.426 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.415+0000 7f8e95590640 1 -- 172.21.15.104:0/1621002039 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8e900af180 msgr2=0x7f8e90201860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:05.427 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.415+0000 7f8e95590640 1 --2- 172.21.15.104:0/1621002039 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8e900af180 0x7f8e90201860 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f8e8800c730 tx=0x7f8e8800cbf0 comp rx=0 tx=0).stop 2024-03-28T14:08:05.427 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.415+0000 7f8e8f7fe640 1 -- 172.21.15.104:0/1621002039 reap_dead start 2024-03-28T14:08:05.427 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.415+0000 7f44fe62b640 1 -- 172.21.15.104:0/1187803630 --> [v2:172.21.15.104:6809/3568347381,v1:172.21.15.104:6811/3568347381] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f44f80716b0 con 0x7f44dc0701c0 2024-03-28T14:08:05.428 INFO:teuthology.orchestra.run.smithi104.stdout:55834574853 2024-03-28T14:08:05.428 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-03-28T14:08:05.429 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.415+0000 7f44edffb640 1 -- 172.21.15.104:0/1187803630 <== osd.3 v2:172.21.15.104:6809/3568347381 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f44f80716b0 con 0x7f44dc0701c0 2024-03-28T14:08:05.430 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.415+0000 7f44fe62b640 1 -- 172.21.15.104:0/1187803630 >> [v2:172.21.15.104:6809/3568347381,v1:172.21.15.104:6811/3568347381] conn(0x7f44dc0701c0 msgr2=0x7f44dc0725e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:05.430 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.415+0000 7f44fe62b640 1 --2- 172.21.15.104:0/1187803630 >> [v2:172.21.15.104:6809/3568347381,v1:172.21.15.104:6811/3568347381] conn(0x7f44dc0701c0 0x7f44dc0725e0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:05.431 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.415+0000 7f44fe62b640 1 -- 172.21.15.104:0/1187803630 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f44dc067fc0 msgr2=0x7f44dc06a480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:05.431 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.415+0000 7f44fe62b640 1 --2- 172.21.15.104:0/1187803630 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f44dc067fc0 0x7f44dc06a480 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f44f809e820 tx=0x7f44e8009040 comp rx=0 tx=0).stop 2024-03-28T14:08:05.431 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.415+0000 7f44fe62b640 1 -- 172.21.15.104:0/1187803630 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f44f808b5e0 msgr2=0x7f44f8156d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:05.431 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.415+0000 7f44fe62b640 1 --2- 172.21.15.104:0/1187803630 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f44f808b5e0 0x7f44f8156d50 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f44f401a4d0 tx=0x7f44f4019640 comp rx=0 tx=0).stop 2024-03-28T14:08:05.431 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.415+0000 7f44fde2a640 1 -- 172.21.15.104:0/1187803630 reap_dead start 2024-03-28T14:08:05.431 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.415+0000 7f44fe62b640 1 -- 172.21.15.104:0/1187803630 shutdown_connections 2024-03-28T14:08:05.431 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.415+0000 7f44fe62b640 1 -- 172.21.15.104:0/1187803630 >> 172.21.15.104:0/1187803630 conn(0x7f44f806c530 msgr2=0x7f44f81550a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T14:08:05.431 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.415+0000 7f44fe62b640 1 -- 172.21.15.104:0/1187803630 shutdown_connections 2024-03-28T14:08:05.432 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.415+0000 7f44fe62b640 1 -- 172.21.15.104:0/1187803630 wait complete. 2024-03-28T14:08:05.432 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.419+0000 7f8e95590640 1 -- 172.21.15.104:0/1621002039 shutdown_connections 2024-03-28T14:08:05.432 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.419+0000 7f8e95590640 1 -- 172.21.15.104:0/1621002039 >> 172.21.15.104:0/1621002039 conn(0x7f8e900b8d30 msgr2=0x7f8e900b9110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T14:08:05.432 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.419+0000 7f8e95590640 1 -- 172.21.15.104:0/1621002039 shutdown_connections 2024-03-28T14:08:05.432 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.419+0000 7f8e95590640 1 -- 172.21.15.104:0/1621002039 wait complete. 2024-03-28T14:08:05.434 INFO:teuthology.orchestra.run.smithi104.stdout:55834574856 2024-03-28T14:08:05.434 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2024-03-28T14:08:05.435 INFO:teuthology.orchestra.run.smithi104.stdout:55834574855 2024-03-28T14:08:05.435 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-03-28T14:08:05.443 INFO:teuthology.orchestra.run.smithi104.stdout:55834574855 2024-03-28T14:08:05.443 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2024-03-28T14:08:05.446 INFO:teuthology.orchestra.run.smithi104.stdout:55834574857 2024-03-28T14:08:05.447 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-03-28T14:08:05.558 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.555+0000 7f315e3aa640 1 Processor -- start 2024-03-28T14:08:05.560 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.555+0000 7f315e3aa640 1 -- start start 2024-03-28T14:08:05.560 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.559+0000 7f315e3aa640 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f31580b4910 0x7f3158146b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:05.560 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.559+0000 7f315e3aa640 1 --2- >> v2:172.21.15.171:3300/0 conn(0x7f31580aee40 0x7f31580b29b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:05.560 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.559+0000 7f315e3aa640 1 --2- >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f315806dce0 0x7f31581435b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:05.560 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.559+0000 7f315e3aa640 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 -- 0x7f31580afea0 con 0x7f31580b4910 2024-03-28T14:08:05.560 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.559+0000 7f315e3aa640 1 -- --> v2:172.21.15.171:3300/0 -- mon_getmap magic: 0 -- 0x7f315806e4e0 con 0x7f31580aee40 2024-03-28T14:08:05.560 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.559+0000 7f315e3aa640 1 -- --> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] -- mon_getmap magic: 0 -- 0x7f31580b4cf0 con 0x7f315806dce0 2024-03-28T14:08:05.560 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.559+0000 7f3157fff640 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f31580b4910 0x7f3158146b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:05.560 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.559+0000 7f3157fff640 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f31580b4910 0x7f3158146b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:42526/0 (socket says 172.21.15.104:42526) 2024-03-28T14:08:05.561 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.559+0000 7f3157fff640 1 -- 172.21.15.104:0/659925508 learned_addr learned my addr 172.21.15.104:0/659925508 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-03-28T14:08:05.561 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.559+0000 7f3157fff640 1 -- 172.21.15.104:0/659925508 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f315806dce0 msgr2=0x7f31581435b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T14:08:05.561 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.559+0000 7f3157fff640 1 --2- 172.21.15.104:0/659925508 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f315806dce0 0x7f31581435b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:05.561 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.559+0000 7f3157fff640 1 -- 172.21.15.104:0/659925508 >> v2:172.21.15.171:3300/0 conn(0x7f31580aee40 msgr2=0x7f31580b29b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T14:08:05.561 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.559+0000 7f3157fff640 1 --2- 172.21.15.104:0/659925508 >> v2:172.21.15.171:3300/0 conn(0x7f31580aee40 0x7f31580b29b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:05.561 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.559+0000 7f3157fff640 1 -- 172.21.15.104:0/659925508 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f315806e200 con 0x7f31580b4910 2024-03-28T14:08:05.563 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.563+0000 7f3157fff640 1 --2- 172.21.15.104:0/659925508 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f31580b4910 0x7f3158146b20 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f315806be00 tx=0x7f315001ed20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=223c15ab24756ad8 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:08:05.564 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.563+0000 7f3156ffd640 1 -- 172.21.15.104:0/659925508 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31500ad070 con 0x7f31580b4910 2024-03-28T14:08:05.564 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.563+0000 7f3156ffd640 1 -- 172.21.15.104:0/659925508 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f31500a7c40 con 0x7f31580b4910 2024-03-28T14:08:05.566 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.563+0000 7f315e3aa640 1 -- 172.21.15.104:0/659925508 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f31580b4910 msgr2=0x7f3158146b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:05.566 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.563+0000 7f315e3aa640 1 --2- 172.21.15.104:0/659925508 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f31580b4910 0x7f3158146b20 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f315806be00 tx=0x7f315001ed20 comp rx=0 tx=0).stop 2024-03-28T14:08:05.566 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.563+0000 7f315e3aa640 1 -- 172.21.15.104:0/659925508 shutdown_connections 2024-03-28T14:08:05.566 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.563+0000 7f315e3aa640 1 --2- 172.21.15.104:0/659925508 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f315806dce0 0x7f31581435b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:05.566 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.563+0000 7f315e3aa640 1 --2- 172.21.15.104:0/659925508 >> v2:172.21.15.171:3300/0 conn(0x7f31580aee40 0x7f31580b29b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:05.566 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.563+0000 7f315e3aa640 1 --2- 172.21.15.104:0/659925508 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f31580b4910 0x7f3158146b20 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:05.566 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.563+0000 7f315e3aa640 1 -- 172.21.15.104:0/659925508 >> 172.21.15.104:0/659925508 conn(0x7f31580b57a0 msgr2=0x7f31580b5bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T14:08:05.567 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.563+0000 7f315e3aa640 1 -- 172.21.15.104:0/659925508 shutdown_connections 2024-03-28T14:08:05.567 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.563+0000 7f315e3aa640 1 -- 172.21.15.104:0/659925508 wait complete. 2024-03-28T14:08:05.570 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.567+0000 7f315e3aa640 1 Processor -- start 2024-03-28T14:08:05.570 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.567+0000 7f315e3aa640 1 -- start start 2024-03-28T14:08:05.570 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.567+0000 7f315e3aa640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f315806dce0 0x7f315813a710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:05.570 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.567+0000 7f315e3aa640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f31580aee40 0x7f31581df5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:05.570 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.567+0000 7f315e3aa640 1 --2- >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f31581dfae0 0x7f31581341b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:05.570 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.567+0000 7f315e3aa640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f31580aded0 con 0x7f315806dce0 2024-03-28T14:08:05.571 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.567+0000 7f315e3aa640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f31580add50 con 0x7f31580aee40 2024-03-28T14:08:05.571 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.567+0000 7f315e3aa640 1 -- --> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] -- mon_getmap magic: 0 -- 0x7f3158144300 con 0x7f31581dfae0 2024-03-28T14:08:05.571 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.567+0000 7f3157fff640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f315806dce0 0x7f315813a710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:05.572 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.567+0000 7f3157fff640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f315806dce0 0x7f315813a710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:42530/0 (socket says 172.21.15.104:42530) 2024-03-28T14:08:05.572 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.567+0000 7f3157fff640 1 -- 172.21.15.104:0/1258104020 learned_addr learned my addr 172.21.15.104:0/1258104020 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-03-28T14:08:05.572 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.567+0000 7f31577fe640 1 --2- 172.21.15.104:0/1258104020 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f31580aee40 0x7f31581df5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:05.572 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.567+0000 7f315c947640 1 --2- 172.21.15.104:0/1258104020 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f31581dfae0 0x7f31581341b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:05.572 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.567+0000 7f3157fff640 1 -- 172.21.15.104:0/1258104020 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f31581dfae0 msgr2=0x7f31581341b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:05.572 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.567+0000 7f3157fff640 1 --2- 172.21.15.104:0/1258104020 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f31581dfae0 0x7f31581341b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:05.572 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.567+0000 7f3157fff640 1 -- 172.21.15.104:0/1258104020 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f31580aee40 msgr2=0x7f31581df5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:05.572 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.567+0000 7f3157fff640 1 --2- 172.21.15.104:0/1258104020 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f31580aee40 0x7f31581df5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:05.572 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.567+0000 7f3157fff640 1 -- 172.21.15.104:0/1258104020 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3158146010 con 0x7f315806dce0 2024-03-28T14:08:05.572 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.567+0000 7f31577fe640 1 --2- 172.21.15.104:0/1258104020 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f31580aee40 0x7f31581df5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T14:08:05.572 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.567+0000 7f315c947640 1 --2- 172.21.15.104:0/1258104020 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f31581dfae0 0x7f31581341b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T14:08:05.572 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.567+0000 7f3157fff640 1 --2- 172.21.15.104:0/1258104020 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f315806dce0 0x7f315813a710 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f315001d3d0 tx=0x7f315001bac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:08:05.573 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.567+0000 7f3154ff9640 1 -- 172.21.15.104:0/1258104020 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31500c0070 con 0x7f315806dce0 2024-03-28T14:08:05.573 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.567+0000 7f3154ff9640 1 -- 172.21.15.104:0/1258104020 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f31500a8da0 con 0x7f315806dce0 2024-03-28T14:08:05.573 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.571+0000 7f3154ff9640 1 -- 172.21.15.104:0/1258104020 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31500ad040 con 0x7f315806dce0 2024-03-28T14:08:05.573 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.571+0000 7f315e3aa640 1 -- 172.21.15.104:0/1258104020 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f31581dffa0 con 0x7f315806dce0 2024-03-28T14:08:05.574 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.571+0000 7f315e3aa640 1 -- 172.21.15.104:0/1258104020 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f31581e0240 con 0x7f315806dce0 2024-03-28T14:08:05.574 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.571+0000 7f315e3aa640 1 -- 172.21.15.104:0/1258104020 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3158068640 con 0x7f315806dce0 2024-03-28T14:08:05.574 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.571+0000 7f3154ff9640 1 -- 172.21.15.104:0/1258104020 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) ==== 87803+0+0 (secure 0 0 0) 0x7f31500c3070 con 0x7f315806dce0 2024-03-28T14:08:05.574 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.571+0000 7f3154ff9640 1 --2- 172.21.15.104:0/1258104020 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f3144067fc0 0x7f314406a480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:05.575 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.571+0000 7f31577fe640 1 --2- 172.21.15.104:0/1258104020 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f3144067fc0 0x7f314406a480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:05.575 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.575+0000 7f31577fe640 1 --2- 172.21.15.104:0/1258104020 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f3144067fc0 0x7f314406a480 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f314c006fd0 tx=0x7f314c008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:08:05.583 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.579+0000 7f3154ff9640 1 -- 172.21.15.104:0/1258104020 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f3150127060 con 0x7f315806dce0 2024-03-28T14:08:05.583 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.583+0000 7f3154ff9640 1 -- 172.21.15.104:0/1258104020 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179117 (secure 0 0 0) 0x7f31500189a0 con 0x7f315806dce0 2024-03-28T14:08:05.612 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.603+0000 7f6b814be640 1 Processor -- start 2024-03-28T14:08:05.615 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.611+0000 7f6b814be640 1 -- start start 2024-03-28T14:08:05.615 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.611+0000 7f6b814be640 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f6b7c0add50 0x7f6b7c14de20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:05.615 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.611+0000 7f6b814be640 1 --2- >> v2:172.21.15.171:3300/0 conn(0x7f6b7c14e360 0x7f6b7c14e810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:05.615 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.615+0000 7f6b814be640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6b7c0b8fe0 con 0x7f6b7c0b8c00 2024-03-28T14:08:05.615 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.615+0000 7f6b814be640 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 -- 0x7f6b7c0b4a80 con 0x7f6b7c0add50 2024-03-28T14:08:05.615 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.615+0000 7f6b814be640 1 -- --> v2:172.21.15.171:3300/0 -- mon_getmap magic: 0 -- 0x7f6b7c0b0100 con 0x7f6b7c14e360 2024-03-28T14:08:05.616 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.615+0000 7ffa00296640 1 Processor -- start 2024-03-28T14:08:05.616 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.615+0000 7f6b7b7fe640 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f6b7c0add50 0x7f6b7c14de20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:05.616 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.615+0000 7f6b7b7fe640 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f6b7c0add50 0x7f6b7c14de20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:42542/0 (socket says 172.21.15.104:42542) 2024-03-28T14:08:05.616 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.615+0000 7f6b7b7fe640 1 -- 172.21.15.104:0/149477101 learned_addr learned my addr 172.21.15.104:0/149477101 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-03-28T14:08:05.616 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.615+0000 7f6b7b7fe640 1 -- 172.21.15.104:0/149477101 >> v2:172.21.15.171:3300/0 conn(0x7f6b7c14e360 msgr2=0x7f6b7c14e810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T14:08:05.617 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.615+0000 7f6b7b7fe640 1 --2- 172.21.15.104:0/149477101 >> v2:172.21.15.171:3300/0 conn(0x7f6b7c14e360 0x7f6b7c14e810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:05.617 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.615+0000 7f6b7b7fe640 1 -- 172.21.15.104:0/149477101 >> v1:172.21.15.171:6789/0 conn(0x7f6b7c0b8c00 legacy=0x7f6b7c0ae8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:05.617 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.615+0000 7f6b7b7fe640 1 -- 172.21.15.104:0/149477101 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6b7c0b03e0 con 0x7f6b7c0add50 2024-03-28T14:08:05.617 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.615+0000 7ffa00296640 1 -- start start 2024-03-28T14:08:05.617 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.615+0000 7ffa00296640 1 --2- >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7ff9f80b4910 0x7ff9f8146b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:05.617 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.615+0000 7ffa00296640 1 --2- >> v2:172.21.15.171:3300/0 conn(0x7ff9f806dce0 0x7ff9f814de90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:05.618 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.615+0000 7ffa00296640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff9f80afea0 con 0x7ff9f80aee40 2024-03-28T14:08:05.618 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.615+0000 7ffa00296640 1 -- --> v2:172.21.15.171:3300/0 -- mon_getmap magic: 0 -- 0x7ff9f806e4e0 con 0x7ff9f806dce0 2024-03-28T14:08:05.618 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.615+0000 7ffa00296640 1 -- --> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] -- mon_getmap magic: 0 -- 0x7ff9f80b4cf0 con 0x7ff9f80b4910 2024-03-28T14:08:05.618 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.615+0000 7ff9fe032640 1 --2- >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7ff9f80b4910 0x7ff9f8146b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:05.618 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.615+0000 7f0273b77640 1 Processor -- start 2024-03-28T14:08:05.619 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.615+0000 7ff9fe032640 1 --2- >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7ff9f80b4910 0x7ff9f8146b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.171:3301/0 says I am v2:172.21.15.104:56068/0 (socket says 172.21.15.104:56068) 2024-03-28T14:08:05.619 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.615+0000 7ff9fe032640 1 -- 172.21.15.104:0/4186682390 learned_addr learned my addr 172.21.15.104:0/4186682390 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-03-28T14:08:05.619 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.615+0000 7ff9fe833640 1 --2- 172.21.15.104:0/4186682390 >> v2:172.21.15.171:3300/0 conn(0x7ff9f806dce0 0x7ff9f814de90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:05.619 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.615+0000 7ff9fe032640 1 -- 172.21.15.104:0/4186682390 >> v2:172.21.15.171:3300/0 conn(0x7ff9f806dce0 msgr2=0x7ff9f814de90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T14:08:05.619 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.615+0000 7ff9fe032640 1 --2- 172.21.15.104:0/4186682390 >> v2:172.21.15.171:3300/0 conn(0x7ff9f806dce0 0x7ff9f814de90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:05.619 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.615+0000 7ff9fe032640 1 -- 172.21.15.104:0/4186682390 >> v1:172.21.15.171:6789/0 conn(0x7ff9f80aee40 legacy=0x7ff9f80b29b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:05.619 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.615+0000 7ff9fe032640 1 -- 172.21.15.104:0/4186682390 --> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff9f806e200 con 0x7ff9f80b4910 2024-03-28T14:08:05.620 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.615+0000 7f0273b77640 1 -- start start 2024-03-28T14:08:05.620 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.615+0000 7f0273b77640 1 --2- >> v2:172.21.15.171:3300/0 conn(0x7f0264033920 0x7f0264036540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:05.620 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.615+0000 7f0273b77640 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f0264033d00 0x7f0264035560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:05.621 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.615+0000 7f0273b77640 1 --2- >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f02640c8600 0x7f02640cbb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:05.621 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.615+0000 7f0273b77640 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 -- 0x7f0264036e00 con 0x7f0264033d00 2024-03-28T14:08:05.621 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.615+0000 7f0273b77640 1 -- --> v2:172.21.15.171:3300/0 -- mon_getmap magic: 0 -- 0x7f0264029fb0 con 0x7f0264033920 2024-03-28T14:08:05.621 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.615+0000 7f0273b77640 1 -- --> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] -- mon_getmap magic: 0 -- 0x7f0264035ed0 con 0x7f02640c8600 2024-03-28T14:08:05.621 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.615+0000 7f0271112640 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f0264033d00 0x7f0264035560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:05.621 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.615+0000 7f0271913640 1 --2- >> v2:172.21.15.171:3300/0 conn(0x7f0264033920 0x7f0264036540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:05.621 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.615+0000 7f0271112640 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f0264033d00 0x7f0264035560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:42556/0 (socket says 172.21.15.104:42556) 2024-03-28T14:08:05.621 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.615+0000 7f0271112640 1 -- 172.21.15.104:0/995860086 learned_addr learned my addr 172.21.15.104:0/995860086 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-03-28T14:08:05.622 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.619+0000 7f6b7b7fe640 1 --2- 172.21.15.104:0/149477101 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6b7c0add50 0x7f6b7c14de20 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f6b70009870 tx=0x7f6b7001c3f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=726ffd51e301ed55 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:08:05.622 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.619+0000 7f6b7affd640 1 -- 172.21.15.104:0/149477101 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b700b1070 con 0x7f6b7c0add50 2024-03-28T14:08:05.622 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.619+0000 7f6b7affd640 1 -- 172.21.15.104:0/149477101 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6b700ac630 con 0x7f6b7c0add50 2024-03-28T14:08:05.622 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.619+0000 7f6b7affd640 1 -- 172.21.15.104:0/149477101 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b700a8050 con 0x7f6b7c0add50 2024-03-28T14:08:05.622 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.619+0000 7f0271112640 1 -- 172.21.15.104:0/995860086 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f02640c8600 msgr2=0x7f02640cbb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T14:08:05.622 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.619+0000 7f0272114640 1 --2- 172.21.15.104:0/995860086 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f02640c8600 0x7f02640cbb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:05.622 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.619+0000 7f0271112640 1 --2- 172.21.15.104:0/995860086 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f02640c8600 0x7f02640cbb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:05.623 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.619+0000 7f6b814be640 1 -- 172.21.15.104:0/149477101 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6b7c0add50 msgr2=0x7f6b7c14de20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:05.623 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.619+0000 7f6b814be640 1 --2- 172.21.15.104:0/149477101 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6b7c0add50 0x7f6b7c14de20 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f6b70009870 tx=0x7f6b7001c3f0 comp rx=0 tx=0).stop 2024-03-28T14:08:05.623 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.619+0000 7f0271112640 1 -- 172.21.15.104:0/995860086 >> v2:172.21.15.171:3300/0 conn(0x7f0264033920 msgr2=0x7f0264036540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T14:08:05.623 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.619+0000 7f0271112640 1 --2- 172.21.15.104:0/995860086 >> v2:172.21.15.171:3300/0 conn(0x7f0264033920 0x7f0264036540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:05.623 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.619+0000 7f0271112640 1 -- 172.21.15.104:0/995860086 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f02640361b0 con 0x7f0264033d00 2024-03-28T14:08:05.623 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.619+0000 7f0271913640 1 --2- 172.21.15.104:0/995860086 >> v2:172.21.15.171:3300/0 conn(0x7f0264033920 0x7f0264036540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T14:08:05.623 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.619+0000 7f0272114640 1 --2- 172.21.15.104:0/995860086 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f02640c8600 0x7f02640cbb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T14:08:05.624 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.619+0000 7ff9fe032640 1 --2- 172.21.15.104:0/4186682390 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7ff9f80b4910 0x7ff9f8146b20 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7ff9f0009e50 tx=0x7ff9f001c9a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=15c073a9c4d1c51b server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:08:05.624 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.619+0000 7f6b814be640 1 -- 172.21.15.104:0/149477101 shutdown_connections 2024-03-28T14:08:05.624 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.619+0000 7f6b814be640 1 --2- 172.21.15.104:0/149477101 >> v2:172.21.15.171:3300/0 conn(0x7f6b7c14e360 0x7f6b7c14e810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:05.624 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.619+0000 7f6b814be640 1 --2- 172.21.15.104:0/149477101 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6b7c0add50 0x7f6b7c14de20 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:05.624 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.619+0000 7f6b814be640 1 -- 172.21.15.104:0/149477101 >> 172.21.15.104:0/149477101 conn(0x7f6b7c06c530 msgr2=0x7f6b7c06c940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T14:08:05.625 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.619+0000 7ff9fd030640 1 -- 172.21.15.104:0/4186682390 <== mon.2 v2:172.21.15.171:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9f00a56f0 con 0x7ff9f80b4910 2024-03-28T14:08:05.625 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.619+0000 7ff9fd030640 1 -- 172.21.15.104:0/4186682390 <== mon.2 v2:172.21.15.171:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff9f00a5890 con 0x7ff9f80b4910 2024-03-28T14:08:05.625 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.619+0000 7f6b814be640 1 -- 172.21.15.104:0/149477101 shutdown_connections 2024-03-28T14:08:05.625 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.619+0000 7f6b814be640 1 -- 172.21.15.104:0/149477101 wait complete. 2024-03-28T14:08:05.626 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.619+0000 7ffa00296640 1 -- 172.21.15.104:0/4186682390 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7ff9f80b4910 msgr2=0x7ff9f8146b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:05.626 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.619+0000 7ffa00296640 1 --2- 172.21.15.104:0/4186682390 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7ff9f80b4910 0x7ff9f8146b20 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7ff9f0009e50 tx=0x7ff9f001c9a0 comp rx=0 tx=0).stop 2024-03-28T14:08:05.626 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.619+0000 7ffa00296640 1 -- 172.21.15.104:0/4186682390 shutdown_connections 2024-03-28T14:08:05.626 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.619+0000 7ffa00296640 1 --2- 172.21.15.104:0/4186682390 >> v2:172.21.15.171:3300/0 conn(0x7ff9f806dce0 0x7ff9f814de90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:05.626 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.619+0000 7ffa00296640 1 --2- 172.21.15.104:0/4186682390 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7ff9f80b4910 0x7ff9f8146b20 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:05.626 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.619+0000 7ffa00296640 1 -- 172.21.15.104:0/4186682390 >> 172.21.15.104:0/4186682390 conn(0x7ff9f80b57a0 msgr2=0x7ff9f80b5bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T14:08:05.626 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.619+0000 7ffa00296640 1 -- 172.21.15.104:0/4186682390 shutdown_connections 2024-03-28T14:08:05.626 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.619+0000 7ffa00296640 1 -- 172.21.15.104:0/4186682390 wait complete. 2024-03-28T14:08:05.627 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.623+0000 7f0271112640 1 --2- 172.21.15.104:0/995860086 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0264033d00 0x7f0264035560 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f026403b5b0 tx=0x7f025c01c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=92f69da2adf61ac3 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:08:05.627 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.623+0000 7ffa00296640 1 Processor -- start 2024-03-28T14:08:05.627 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.623+0000 7ffa00296640 1 -- start start 2024-03-28T14:08:05.627 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.623+0000 7f0270911640 1 -- 172.21.15.104:0/995860086 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f025c0b0070 con 0x7f0264033d00 2024-03-28T14:08:05.628 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.623+0000 7f6b814be640 1 Processor -- start 2024-03-28T14:08:05.628 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.623+0000 7f0270911640 1 -- 172.21.15.104:0/995860086 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f025c0ab600 con 0x7f0264033d00 2024-03-28T14:08:05.628 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.623+0000 7f6b814be640 1 -- start start 2024-03-28T14:08:05.629 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.623+0000 7ffa00296640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff9f806dce0 0x7ff9f8143d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:05.629 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.623+0000 7ffa00296640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7ff9f80aee40 0x7ff9f81419a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:05.629 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.623+0000 7ffa00296640 1 --2- >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7ff9f80b4910 0x7ff9f8141ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:05.629 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.623+0000 7ffa00296640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7ff9f80add50 con 0x7ff9f806dce0 2024-03-28T14:08:05.629 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.623+0000 7ffa00296640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7ff9f80ae120 con 0x7ff9f80aee40 2024-03-28T14:08:05.629 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.623+0000 7ffa00296640 1 -- --> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] -- mon_getmap magic: 0 -- 0x7ff9f80aded0 con 0x7ff9f80b4910 2024-03-28T14:08:05.630 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.623+0000 7ff9fe032640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff9f806dce0 0x7ff9f8143d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:05.630 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.623+0000 7ff9fe032640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff9f806dce0 0x7ff9f8143d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:42566/0 (socket says 172.21.15.104:42566) 2024-03-28T14:08:05.630 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.623+0000 7ff9fe032640 1 -- 172.21.15.104:0/385213865 learned_addr learned my addr 172.21.15.104:0/385213865 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-03-28T14:08:05.630 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.623+0000 7ff9fd831640 1 --2- 172.21.15.104:0/385213865 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7ff9f80aee40 0x7ff9f81419a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:05.630 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.623+0000 7ff9fe833640 1 --2- 172.21.15.104:0/385213865 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7ff9f80b4910 0x7ff9f8141ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:05.630 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.623+0000 7ff9fe032640 1 -- 172.21.15.104:0/385213865 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7ff9f80b4910 msgr2=0x7ff9f8141ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:05.630 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.623+0000 7ff9fe032640 1 --2- 172.21.15.104:0/385213865 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7ff9f80b4910 0x7ff9f8141ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:05.630 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.623+0000 7ff9fe032640 1 -- 172.21.15.104:0/385213865 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7ff9f80aee40 msgr2=0x7ff9f81419a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:05.630 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.623+0000 7ff9fe032640 1 --2- 172.21.15.104:0/385213865 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7ff9f80aee40 0x7ff9f81419a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:05.630 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.623+0000 7ff9fe032640 1 -- 172.21.15.104:0/385213865 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff9f81414c0 con 0x7ff9f806dce0 2024-03-28T14:08:05.631 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.623+0000 7f6b814be640 1 --2- >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f6b7c0add50 0x7f6b7c070a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:05.631 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.623+0000 7f6b814be640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f6b7c0b8c00 0x7f6b7c09c440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:05.631 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.623+0000 7f6b814be640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6b7c14e360 0x7f6b7c09d210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:05.631 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.623+0000 7f6b814be640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f6b7c0aeff0 con 0x7f6b7c14e360 2024-03-28T14:08:05.631 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.623+0000 7f6b814be640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f6b7c0b5290 con 0x7f6b7c0b8c00 2024-03-28T14:08:05.631 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.623+0000 7f6b814be640 1 -- --> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] -- mon_getmap magic: 0 -- 0x7f6b7c0af170 con 0x7f6b7c0add50 2024-03-28T14:08:05.631 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.623+0000 7f6b80cbd640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6b7c14e360 0x7f6b7c09d210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:05.631 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.623+0000 7f6b80cbd640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6b7c14e360 0x7f6b7c09d210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:42568/0 (socket says 172.21.15.104:42568) 2024-03-28T14:08:05.631 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.623+0000 7f6b80cbd640 1 -- 172.21.15.104:0/2660765173 learned_addr learned my addr 172.21.15.104:0/2660765173 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-03-28T14:08:05.632 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.623+0000 7f6b7b7fe640 1 --2- 172.21.15.104:0/2660765173 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f6b7c0b8c00 0x7f6b7c09c440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:05.632 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.623+0000 7f6b7bfff640 1 --2- 172.21.15.104:0/2660765173 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f6b7c0add50 0x7f6b7c070a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:05.632 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.623+0000 7f0273b77640 1 -- 172.21.15.104:0/995860086 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0264033d00 msgr2=0x7f0264035560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:05.632 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.623+0000 7f0273b77640 1 --2- 172.21.15.104:0/995860086 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0264033d00 0x7f0264035560 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f026403b5b0 tx=0x7f025c01c520 comp rx=0 tx=0).stop 2024-03-28T14:08:05.632 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.623+0000 7f0273b77640 1 -- 172.21.15.104:0/995860086 shutdown_connections 2024-03-28T14:08:05.632 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.623+0000 7f0273b77640 1 --2- 172.21.15.104:0/995860086 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f02640c8600 0x7f02640cbb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:05.632 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.623+0000 7f0273b77640 1 --2- 172.21.15.104:0/995860086 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0264033d00 0x7f0264035560 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:05.632 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.623+0000 7f0273b77640 1 --2- 172.21.15.104:0/995860086 >> v2:172.21.15.171:3300/0 conn(0x7f0264033920 0x7f0264036540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:05.633 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.623+0000 7f0273b77640 1 -- 172.21.15.104:0/995860086 >> 172.21.15.104:0/995860086 conn(0x7f026403a860 msgr2=0x7f026403ac70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T14:08:05.633 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.623+0000 7f6b80cbd640 1 -- 172.21.15.104:0/2660765173 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f6b7c0add50 msgr2=0x7f6b7c070a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:05.633 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.623+0000 7f6b80cbd640 1 --2- 172.21.15.104:0/2660765173 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f6b7c0add50 0x7f6b7c070a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:05.633 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.623+0000 7f6b80cbd640 1 -- 172.21.15.104:0/2660765173 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f6b7c0b8c00 msgr2=0x7f6b7c09c440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:05.633 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.623+0000 7f6b80cbd640 1 --2- 172.21.15.104:0/2660765173 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f6b7c0b8c00 0x7f6b7c09c440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:05.633 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.627+0000 7f6b80cbd640 1 -- 172.21.15.104:0/2660765173 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6b7c09caa0 con 0x7f6b7c14e360 2024-03-28T14:08:05.633 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.623+0000 7f0273b77640 1 -- 172.21.15.104:0/995860086 shutdown_connections 2024-03-28T14:08:05.634 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.627+0000 7f6b80cbd640 1 --2- 172.21.15.104:0/2660765173 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6b7c14e360 0x7f6b7c09d210 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f6b6c00b430 tx=0x7f6b6c00b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:08:05.634 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.627+0000 7ff9fe032640 1 --2- 172.21.15.104:0/385213865 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff9f806dce0 0x7ff9f8143d20 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7ff9f001a5d0 tx=0x7ff9f0019740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:08:05.634 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.627+0000 7ff9eeffd640 1 -- 172.21.15.104:0/385213865 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9f00bc020 con 0x7ff9f806dce0 2024-03-28T14:08:05.634 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.627+0000 7ff9eeffd640 1 -- 172.21.15.104:0/385213865 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff9f0007c50 con 0x7ff9f806dce0 2024-03-28T14:08:05.635 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.627+0000 7ff9eeffd640 1 -- 172.21.15.104:0/385213865 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9f00b8470 con 0x7ff9f806dce0 2024-03-28T14:08:05.635 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.627+0000 7ffa00296640 1 -- 172.21.15.104:0/385213865 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff9f8072940 con 0x7ff9f806dce0 2024-03-28T14:08:05.635 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.627+0000 7ffa00296640 1 -- 172.21.15.104:0/385213865 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff9f8143160 con 0x7ff9f806dce0 2024-03-28T14:08:05.635 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.627+0000 7f0273b77640 1 -- 172.21.15.104:0/995860086 wait complete. 2024-03-28T14:08:05.635 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.627+0000 7f0273b77640 1 Processor -- start 2024-03-28T14:08:05.636 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.627+0000 7f0273b77640 1 -- start start 2024-03-28T14:08:05.636 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.627+0000 7f0273b77640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0264033920 0x7f026403caa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:05.636 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.627+0000 7f0273b77640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f02640c8600 0x7f02640c5a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:05.636 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.627+0000 7f0273b77640 1 --2- >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f02641751b0 0x7f026416b8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:05.636 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.627+0000 7f0273b77640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f0264035c20 con 0x7f0264033920 2024-03-28T14:08:05.636 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.627+0000 7f0273b77640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f0264035aa0 con 0x7f02640c8600 2024-03-28T14:08:05.636 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.627+0000 7f0273b77640 1 -- --> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] -- mon_getmap magic: 0 -- 0x7f02640d4eb0 con 0x7f02641751b0 2024-03-28T14:08:05.636 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.627+0000 7f0271112640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f02640c8600 0x7f02640c5a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:05.636 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.627+0000 7f0271112640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f02640c8600 0x7f02640c5a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.171:3300/0 says I am v2:172.21.15.104:53656/0 (socket says 172.21.15.104:53656) 2024-03-28T14:08:05.636 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.627+0000 7f0271112640 1 -- 172.21.15.104:0/736062044 learned_addr learned my addr 172.21.15.104:0/736062044 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-03-28T14:08:05.636 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.627+0000 7f0272114640 1 --2- 172.21.15.104:0/736062044 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f02641751b0 0x7f026416b8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:05.636 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.627+0000 7f0271913640 1 --2- 172.21.15.104:0/736062044 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0264033920 0x7f026403caa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:05.636 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.631+0000 7f0271913640 1 -- 172.21.15.104:0/736062044 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f02641751b0 msgr2=0x7f026416b8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:05.637 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.627+0000 7ff9eeffd640 1 -- 172.21.15.104:0/385213865 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) ==== 87803+0+0 (secure 0 0 0) 0x7ff9f00bf020 con 0x7ff9f806dce0 2024-03-28T14:08:05.637 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.631+0000 7ff9eeffd640 1 --2- 172.21.15.104:0/385213865 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7ff9e4068160 0x7ff9e406a620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:05.637 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.631+0000 7ff9eeffd640 1 -- 172.21.15.104:0/385213865 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7ff9f0121540 con 0x7ff9f806dce0 2024-03-28T14:08:05.637 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.631+0000 7f0271913640 1 --2- 172.21.15.104:0/736062044 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f02641751b0 0x7f026416b8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:05.637 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.631+0000 7f0271913640 1 -- 172.21.15.104:0/736062044 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f02640c8600 msgr2=0x7f02640c5a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:05.637 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.631+0000 7f0271913640 1 --2- 172.21.15.104:0/736062044 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f02640c8600 0x7f02640c5a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:05.637 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.631+0000 7f0271913640 1 -- 172.21.15.104:0/736062044 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f02640c5f50 con 0x7f0264033920 2024-03-28T14:08:05.637 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.631+0000 7f0271112640 1 --2- 172.21.15.104:0/736062044 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f02640c8600 0x7f02640c5a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T14:08:05.638 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.631+0000 7ff9fd831640 1 --2- 172.21.15.104:0/385213865 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7ff9e4068160 0x7ff9e406a620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:05.638 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.631+0000 7f0271913640 1 --2- 172.21.15.104:0/736062044 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0264033920 0x7f026403caa0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f026c04fde0 tx=0x7f026c069120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:08:05.638 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.631+0000 7ff9fd831640 1 --2- 172.21.15.104:0/385213865 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7ff9e4068160 0x7ff9e406a620 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7ff9f8144c60 tx=0x7ff9e8009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:08:05.639 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.631+0000 7f0272114640 1 --2- 172.21.15.104:0/736062044 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f02641751b0 0x7f026416b8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T14:08:05.639 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.635+0000 7f6b78ff9640 1 -- 172.21.15.104:0/2660765173 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b6c015020 con 0x7f6b7c14e360 2024-03-28T14:08:05.639 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.635+0000 7f6b814be640 1 -- 172.21.15.104:0/2660765173 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6b7c09cd80 con 0x7f6b7c14e360 2024-03-28T14:08:05.639 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.635+0000 7f6b814be640 1 -- 172.21.15.104:0/2660765173 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6b7c0720b0 con 0x7f6b7c14e360 2024-03-28T14:08:05.640 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.635+0000 7f6b78ff9640 1 -- 172.21.15.104:0/2660765173 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6b6c011830 con 0x7f6b7c14e360 2024-03-28T14:08:05.640 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.635+0000 7f6b78ff9640 1 -- 172.21.15.104:0/2660765173 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b6c011c20 con 0x7f6b7c14e360 2024-03-28T14:08:05.640 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.635+0000 7f6b814be640 1 -- 172.21.15.104:0/2660765173 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6b5c0052a0 con 0x7f6b7c14e360 2024-03-28T14:08:05.640 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.635+0000 7ffa00296640 1 -- 172.21.15.104:0/385213865 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff9e00052a0 con 0x7ff9f806dce0 2024-03-28T14:08:05.640 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.635+0000 7f6b78ff9640 1 -- 172.21.15.104:0/2660765173 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) ==== 87803+0+0 (secure 0 0 0) 0x7f6b6c018020 con 0x7f6b7c14e360 2024-03-28T14:08:05.640 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.639+0000 7f6b78ff9640 1 --2- 172.21.15.104:0/2660765173 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f6b68068040 0x7f6b6806a500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:05.641 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.639+0000 7f02627fc640 1 -- 172.21.15.104:0/736062044 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f026c094470 con 0x7f0264033920 2024-03-28T14:08:05.641 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.639+0000 7f02627fc640 1 -- 172.21.15.104:0/736062044 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f026c08f070 con 0x7f0264033920 2024-03-28T14:08:05.641 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.639+0000 7f6b7bfff640 1 --2- 172.21.15.104:0/2660765173 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f6b68068040 0x7f6b6806a500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:05.642 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.639+0000 7f02627fc640 1 -- 172.21.15.104:0/736062044 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f026c094890 con 0x7f0264033920 2024-03-28T14:08:05.642 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.639+0000 7f0273b77640 1 -- 172.21.15.104:0/736062044 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f02640d7750 con 0x7f0264033920 2024-03-28T14:08:05.642 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.639+0000 7f6b78ff9640 1 -- 172.21.15.104:0/2660765173 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f6b6c08a060 con 0x7f6b7c14e360 2024-03-28T14:08:05.642 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.639+0000 7f6b7bfff640 1 --2- 172.21.15.104:0/2660765173 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f6b68068040 0x7f6b6806a500 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f6b7c0b8940 tx=0x7f6b74009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:08:05.643 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.639+0000 7f0273b77640 1 -- 172.21.15.104:0/736062044 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f026403d1d0 con 0x7f0264033920 2024-03-28T14:08:05.643 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.639+0000 7f0273b77640 1 -- 172.21.15.104:0/736062044 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0264039750 con 0x7f0264033920 2024-03-28T14:08:05.645 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.643+0000 7f6b78ff9640 1 -- 172.21.15.104:0/2660765173 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179117 (secure 0 0 0) 0x7f6b6c055740 con 0x7f6b7c14e360 2024-03-28T14:08:05.651 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.651+0000 7ff9eeffd640 1 -- 172.21.15.104:0/385213865 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179117 (secure 0 0 0) 0x7ff9f00c6030 con 0x7ff9f806dce0 2024-03-28T14:08:05.654 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.651+0000 7f02627fc640 1 -- 172.21.15.104:0/736062044 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) ==== 87803+0+0 (secure 0 0 0) 0x7f026c094a30 con 0x7f0264033920 2024-03-28T14:08:05.654 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.651+0000 7f02627fc640 1 --2- 172.21.15.104:0/736062044 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f0250068010 0x7f025006a4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:05.654 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.651+0000 7f02627fc640 1 -- 172.21.15.104:0/736062044 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f026c101740 con 0x7f0264033920 2024-03-28T14:08:05.654 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.651+0000 7f0271112640 1 --2- 172.21.15.104:0/736062044 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f0250068010 0x7f025006a4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:05.655 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.655+0000 7f0271112640 1 --2- 172.21.15.104:0/736062044 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f0250068010 0x7f025006a4d0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f025c000c10 tx=0x7f025c007f90 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:08:05.658 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.655+0000 7efebffff640 1 Processor -- start 2024-03-28T14:08:05.658 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.655+0000 7efebffff640 1 -- start start 2024-03-28T14:08:05.659 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.655+0000 7efebffff640 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7efec0144840 0x7efec0141000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:05.659 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.655+0000 7efebffff640 1 --2- >> v2:172.21.15.171:3300/0 conn(0x7efec00b4480 0x7efec0141540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:05.659 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.655+0000 7efebffff640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efec00b6dc0 con 0x7efec00b6050 2024-03-28T14:08:05.659 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.655+0000 7efebffff640 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 -- 0x7efec00b49f0 con 0x7efec0144840 2024-03-28T14:08:05.659 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.655+0000 7efebffff640 1 -- --> v2:172.21.15.171:3300/0 -- mon_getmap magic: 0 -- 0x7efec00b4cd0 con 0x7efec00b4480 2024-03-28T14:08:05.659 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.655+0000 7efebe7fc640 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7efec0144840 0x7efec0141000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:05.660 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.655+0000 7efebe7fc640 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7efec0144840 0x7efec0141000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:42586/0 (socket says 172.21.15.104:42586) 2024-03-28T14:08:05.660 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.655+0000 7efebe7fc640 1 -- 172.21.15.104:0/900237546 learned_addr learned my addr 172.21.15.104:0/900237546 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-03-28T14:08:05.660 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.655+0000 7efebe7fc640 1 -- 172.21.15.104:0/900237546 >> v2:172.21.15.171:3300/0 conn(0x7efec00b4480 msgr2=0x7efec0141540 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-28T14:08:05.660 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.655+0000 7efebe7fc640 1 --2- 172.21.15.104:0/900237546 >> v2:172.21.15.171:3300/0 conn(0x7efec00b4480 0x7efec0141540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:05.660 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.655+0000 7efebe7fc640 1 -- 172.21.15.104:0/900237546 >> v1:172.21.15.171:6789/0 conn(0x7efec00b6050 legacy=0x7efec0144130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T14:08:05.660 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.655+0000 7efebe7fc640 1 -- 172.21.15.104:0/900237546 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efec00ba1f0 con 0x7efec0144840 2024-03-28T14:08:05.661 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.659+0000 7efebe7fc640 1 --2- 172.21.15.104:0/900237546 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7efec0144840 0x7efec0141000 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7efec01482f0 tx=0x7efeb001e860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=996f2bba3d857c57 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:08:05.664 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.663+0000 7f02627fc640 1 -- 172.21.15.104:0/736062044 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179117 (secure 0 0 0) 0x7f02640c5f50 con 0x7f0264033920 2024-03-28T14:08:05.666 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.663+0000 7efebdffb640 1 -- 172.21.15.104:0/900237546 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efeb0004230 con 0x7efec0144840 2024-03-28T14:08:05.666 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.663+0000 7efebdffb640 1 -- 172.21.15.104:0/900237546 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efeb00043d0 con 0x7efec0144840 2024-03-28T14:08:05.666 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.663+0000 7efebffff640 1 -- 172.21.15.104:0/900237546 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7efec0144840 msgr2=0x7efec0141000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:05.667 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.663+0000 7efebffff640 1 --2- 172.21.15.104:0/900237546 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7efec0144840 0x7efec0141000 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7efec01482f0 tx=0x7efeb001e860 comp rx=0 tx=0).stop 2024-03-28T14:08:05.667 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.663+0000 7efebffff640 1 -- 172.21.15.104:0/900237546 shutdown_connections 2024-03-28T14:08:05.667 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.663+0000 7efebffff640 1 --2- 172.21.15.104:0/900237546 >> v2:172.21.15.171:3300/0 conn(0x7efec00b4480 0x7efec0141540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:05.667 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.663+0000 7efebffff640 1 --2- 172.21.15.104:0/900237546 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7efec0144840 0x7efec0141000 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:05.667 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.663+0000 7efebffff640 1 -- 172.21.15.104:0/900237546 >> 172.21.15.104:0/900237546 conn(0x7efec00b9450 msgr2=0x7efec00b9830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T14:08:05.675 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.671+0000 7efebffff640 1 -- 172.21.15.104:0/900237546 shutdown_connections 2024-03-28T14:08:05.675 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.671+0000 7efebffff640 1 -- 172.21.15.104:0/900237546 wait complete. 2024-03-28T14:08:05.676 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.675+0000 7efebffff640 1 Processor -- start 2024-03-28T14:08:05.678 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.675+0000 7efebffff640 1 -- start start 2024-03-28T14:08:05.678 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.675+0000 7efebffff640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7efec00b4480 0x7efec0146b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:05.679 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.675+0000 7efebffff640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7efec00b6050 0x7efec01e4050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:05.679 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.675+0000 7efebffff640 1 --2- >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7efec0144840 0x7efec01e4590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:05.679 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.675+0000 7efebffff640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7efec0150920 con 0x7efec00b6050 2024-03-28T14:08:05.679 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.675+0000 7efebffff640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7efec01507a0 con 0x7efec00b4480 2024-03-28T14:08:05.679 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.675+0000 7efebffff640 1 -- --> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] -- mon_getmap magic: 0 -- 0x7efec0150aa0 con 0x7efec0144840 2024-03-28T14:08:05.679 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.675+0000 7efebe7fc640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7efec00b6050 0x7efec01e4050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:05.679 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.675+0000 7efebe7fc640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7efec00b6050 0x7efec01e4050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:42596/0 (socket says 172.21.15.104:42596) 2024-03-28T14:08:05.679 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.675+0000 7efebe7fc640 1 -- 172.21.15.104:0/1593155518 learned_addr learned my addr 172.21.15.104:0/1593155518 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-03-28T14:08:05.679 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.675+0000 7efebf7fe640 1 --2- 172.21.15.104:0/1593155518 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7efec0144840 0x7efec01e4590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:05.679 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.675+0000 7efebeffd640 1 --2- 172.21.15.104:0/1593155518 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7efec00b4480 0x7efec0146b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:05.679 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.675+0000 7efebe7fc640 1 -- 172.21.15.104:0/1593155518 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7efec0144840 msgr2=0x7efec01e4590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:05.680 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.675+0000 7efebe7fc640 1 --2- 172.21.15.104:0/1593155518 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7efec0144840 0x7efec01e4590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:05.680 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.675+0000 7efebe7fc640 1 -- 172.21.15.104:0/1593155518 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7efec00b4480 msgr2=0x7efec0146b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:05.680 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.675+0000 7efebe7fc640 1 --2- 172.21.15.104:0/1593155518 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7efec00b4480 0x7efec0146b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:05.680 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.675+0000 7efebe7fc640 1 -- 172.21.15.104:0/1593155518 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efec01472e0 con 0x7efec00b6050 2024-03-28T14:08:05.681 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.675+0000 7efebe7fc640 1 --2- 172.21.15.104:0/1593155518 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7efec00b6050 0x7efec01e4050 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7efeb001c490 tx=0x7efeb001b600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:08:05.681 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.679+0000 7efeb77fe640 1 -- 172.21.15.104:0/1593155518 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efeb00c0070 con 0x7efec00b6050 2024-03-28T14:08:05.681 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.679+0000 7efeb77fe640 1 -- 172.21.15.104:0/1593155518 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efeb00a8b00 con 0x7efec00b6050 2024-03-28T14:08:05.681 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.679+0000 7efeb77fe640 1 -- 172.21.15.104:0/1593155518 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efeb0006de0 con 0x7efec00b6050 2024-03-28T14:08:05.681 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.679+0000 7efebffff640 1 -- 172.21.15.104:0/1593155518 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efec00b06e0 con 0x7efec00b6050 2024-03-28T14:08:05.681 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.679+0000 7efebffff640 1 -- 172.21.15.104:0/1593155518 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efec014c3f0 con 0x7efec00b6050 2024-03-28T14:08:05.681 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.679+0000 7efeb77fe640 1 -- 172.21.15.104:0/1593155518 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) ==== 87803+0+0 (secure 0 0 0) 0x7efeb0003620 con 0x7efec00b6050 2024-03-28T14:08:05.682 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.679+0000 7efeb77fe640 1 --2- 172.21.15.104:0/1593155518 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7efea0067fc0 0x7efea006a480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:05.682 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.679+0000 7efeb77fe640 1 -- 172.21.15.104:0/1593155518 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7efeb0120d80 con 0x7efec00b6050 2024-03-28T14:08:05.682 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.679+0000 7efebeffd640 1 --2- 172.21.15.104:0/1593155518 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7efea0067fc0 0x7efea006a480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:05.684 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.675+0000 7f3db8fdb640 1 Processor -- start 2024-03-28T14:08:05.684 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.675+0000 7f3db8fdb640 1 -- start start 2024-03-28T14:08:05.684 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.679+0000 7f3db8fdb640 1 --2- >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f3db40add50 0x7f3db4157a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:05.684 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.679+0000 7f3db8fdb640 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f3db4157fc0 0x7f3db4158430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:05.685 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.679+0000 7f3db8fdb640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3db40ae600 con 0x7f3db40b8c50 2024-03-28T14:08:05.685 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.679+0000 7f3db8fdb640 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 -- 0x7f3db40b9030 con 0x7f3db4157fc0 2024-03-28T14:08:05.685 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.679+0000 7f3db8fdb640 1 -- --> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] -- mon_getmap magic: 0 -- 0x7f3db40b0100 con 0x7f3db40add50 2024-03-28T14:08:05.685 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.679+0000 7f3db3fff640 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f3db4157fc0 0x7f3db4158430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:05.685 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.679+0000 7f3db3fff640 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f3db4157fc0 0x7f3db4158430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:42606/0 (socket says 172.21.15.104:42606) 2024-03-28T14:08:05.685 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.683+0000 7efebeffd640 1 --2- 172.21.15.104:0/1593155518 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7efea0067fc0 0x7efea006a480 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7efea8004e30 tx=0x7efea8009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:08:05.686 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.679+0000 7f3db3fff640 1 -- 172.21.15.104:0/1032397037 learned_addr learned my addr 172.21.15.104:0/1032397037 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-03-28T14:08:05.686 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.679+0000 7f3db2ffd640 1 --2- 172.21.15.104:0/1032397037 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f3db40add50 0x7f3db4157a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:05.686 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.683+0000 7f3db3fff640 1 -- 172.21.15.104:0/1032397037 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f3db40add50 msgr2=0x7f3db4157a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T14:08:05.686 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.683+0000 7f3db3fff640 1 --2- 172.21.15.104:0/1032397037 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f3db40add50 0x7f3db4157a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:05.686 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.683+0000 7f3db3fff640 1 -- 172.21.15.104:0/1032397037 >> v1:172.21.15.171:6789/0 conn(0x7f3db40b8c50 legacy=0x7f3db408b5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:05.687 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.683+0000 7f3db3fff640 1 -- 172.21.15.104:0/1032397037 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3db40b03e0 con 0x7f3db4157fc0 2024-03-28T14:08:05.687 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.687+0000 7f3db3fff640 1 --2- 172.21.15.104:0/1032397037 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3db4157fc0 0x7f3db4158430 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f3dac0083f0 tx=0x7f3dac024d20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2f95a29f537d2d8 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:08:05.688 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.687+0000 7efebffff640 1 -- 172.21.15.104:0/1593155518 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efec0068d50 con 0x7efec00b6050 2024-03-28T14:08:05.696 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.695+0000 7f3db27fc640 1 -- 172.21.15.104:0/1032397037 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3dac0b7070 con 0x7f3db4157fc0 2024-03-28T14:08:05.696 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.695+0000 7f3db27fc640 1 -- 172.21.15.104:0/1032397037 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3dac0ae040 con 0x7f3db4157fc0 2024-03-28T14:08:05.696 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.695+0000 7f3db27fc640 1 -- 172.21.15.104:0/1032397037 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3dac003ba0 con 0x7f3db4157fc0 2024-03-28T14:08:05.697 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.695+0000 7efeb77fe640 1 -- 172.21.15.104:0/1593155518 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179117 (secure 0 0 0) 0x7efeb00185a0 con 0x7efec00b6050 2024-03-28T14:08:05.702 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.695+0000 7f3db8fdb640 1 -- 172.21.15.104:0/1032397037 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3db4157fc0 msgr2=0x7f3db4158430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:05.702 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.695+0000 7f3db8fdb640 1 --2- 172.21.15.104:0/1032397037 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3db4157fc0 0x7f3db4158430 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f3dac0083f0 tx=0x7f3dac024d20 comp rx=0 tx=0).stop 2024-03-28T14:08:05.702 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.695+0000 7f3db8fdb640 1 -- 172.21.15.104:0/1032397037 shutdown_connections 2024-03-28T14:08:05.702 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.695+0000 7f3db8fdb640 1 --2- 172.21.15.104:0/1032397037 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3db4157fc0 0x7f3db4158430 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:05.703 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.695+0000 7f3db8fdb640 1 --2- 172.21.15.104:0/1032397037 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f3db40add50 0x7f3db4157a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:05.703 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.695+0000 7f3db8fdb640 1 -- 172.21.15.104:0/1032397037 >> 172.21.15.104:0/1032397037 conn(0x7f3db406c530 msgr2=0x7f3db406c940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T14:08:05.703 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.695+0000 7f3db8fdb640 1 -- 172.21.15.104:0/1032397037 shutdown_connections 2024-03-28T14:08:05.703 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.695+0000 7f3db8fdb640 1 -- 172.21.15.104:0/1032397037 wait complete. 2024-03-28T14:08:05.703 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.699+0000 7f3db8fdb640 1 Processor -- start 2024-03-28T14:08:05.703 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.699+0000 7f3db8fdb640 1 -- start start 2024-03-28T14:08:05.703 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.699+0000 7f3db8fdb640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3db40add50 0x7f3db40718f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:05.703 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.699+0000 7f3db8fdb640 1 --2- >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f3db40b8c50 0x7f3db41f0910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:05.703 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.699+0000 7f3db8fdb640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f3db4157fc0 0x7f3db41f0e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:05.703 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.699+0000 7f3db8fdb640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f3db4162110 con 0x7f3db40add50 2024-03-28T14:08:05.703 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.699+0000 7f3db8fdb640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f3db4161f90 con 0x7f3db4157fc0 2024-03-28T14:08:05.703 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.699+0000 7f3db8fdb640 1 -- --> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] -- mon_getmap magic: 0 -- 0x7f3db4162290 con 0x7f3db40b8c50 2024-03-28T14:08:05.703 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.699+0000 7f3db37fe640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3db40add50 0x7f3db40718f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:05.704 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.699+0000 7f3db37fe640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3db40add50 0x7f3db40718f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:42612/0 (socket says 172.21.15.104:42612) 2024-03-28T14:08:05.704 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.699+0000 7f3db37fe640 1 -- 172.21.15.104:0/423286494 learned_addr learned my addr 172.21.15.104:0/423286494 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-03-28T14:08:05.704 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.699+0000 7f3db2ffd640 1 --2- 172.21.15.104:0/423286494 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f3db40b8c50 0x7f3db41f0910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:05.704 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.699+0000 7f3db3fff640 1 --2- 172.21.15.104:0/423286494 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f3db4157fc0 0x7f3db41f0e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:05.704 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.699+0000 7f3db37fe640 1 -- 172.21.15.104:0/423286494 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f3db40b8c50 msgr2=0x7f3db41f0910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:05.704 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.699+0000 7f3db37fe640 1 --2- 172.21.15.104:0/423286494 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f3db40b8c50 0x7f3db41f0910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:05.704 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.699+0000 7f3db37fe640 1 -- 172.21.15.104:0/423286494 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f3db4157fc0 msgr2=0x7f3db41f0e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:05.704 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.699+0000 7f3db37fe640 1 --2- 172.21.15.104:0/423286494 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f3db4157fc0 0x7f3db41f0e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:05.704 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.699+0000 7f3db37fe640 1 -- 172.21.15.104:0/423286494 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3db41522f0 con 0x7f3db40add50 2024-03-28T14:08:05.704 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.699+0000 7f3db37fe640 1 --2- 172.21.15.104:0/423286494 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3db40add50 0x7f3db40718f0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f3da800c7d0 tx=0x7f3da800cc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:08:05.705 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.699+0000 7f3d9bfff640 1 -- 172.21.15.104:0/423286494 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3da8017070 con 0x7f3db40add50 2024-03-28T14:08:05.705 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.699+0000 7f3d9bfff640 1 -- 172.21.15.104:0/423286494 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3da8005480 con 0x7f3db40add50 2024-03-28T14:08:05.705 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.699+0000 7f3d9bfff640 1 -- 172.21.15.104:0/423286494 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3da800a860 con 0x7f3db40add50 2024-03-28T14:08:05.705 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.699+0000 7f3db8fdb640 1 -- 172.21.15.104:0/423286494 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3db41570f0 con 0x7f3db40add50 2024-03-28T14:08:05.705 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.699+0000 7f3db8fdb640 1 -- 172.21.15.104:0/423286494 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3db4152010 con 0x7f3db40add50 2024-03-28T14:08:05.707 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.703+0000 7f3d9bfff640 1 -- 172.21.15.104:0/423286494 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) ==== 87803+0+0 (secure 0 0 0) 0x7f3da801a070 con 0x7f3db40add50 2024-03-28T14:08:05.707 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.703+0000 7f3d9bfff640 1 --2- 172.21.15.104:0/423286494 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f3da00680c0 0x7f3da006a580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:05.707 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.707+0000 7f3d9bfff640 1 -- 172.21.15.104:0/423286494 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f3da8085ad0 con 0x7f3db40add50 2024-03-28T14:08:05.708 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.707+0000 7f3db2ffd640 1 --2- 172.21.15.104:0/423286494 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f3da00680c0 0x7f3da006a580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:05.708 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.707+0000 7f3db2ffd640 1 --2- 172.21.15.104:0/423286494 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f3da00680c0 0x7f3da006a580 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f3db406ca60 tx=0x7f3da400e040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:08:05.713 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.711+0000 7f30bf0e7640 1 Processor -- start 2024-03-28T14:08:05.714 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.711+0000 7f3db8fdb640 1 -- 172.21.15.104:0/423286494 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3d940052a0 con 0x7f3db40add50 2024-03-28T14:08:05.716 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.711+0000 7f30bf0e7640 1 -- start start 2024-03-28T14:08:05.716 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.711+0000 7f30bf0e7640 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f30b80b91a0 0x7f30b8146c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:05.716 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.711+0000 7f30bf0e7640 1 --2- >> v2:172.21.15.171:3300/0 conn(0x7f30b81471c0 0x7f30b81443a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:05.716 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.711+0000 7f30bf0e7640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f30b80b9580 con 0x7f30b80af990 2024-03-28T14:08:05.716 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.711+0000 7f30bf0e7640 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 -- 0x7f30b80ae310 con 0x7f30b80b91a0 2024-03-28T14:08:05.716 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.711+0000 7f30bf0e7640 1 -- --> v2:172.21.15.171:3300/0 -- mon_getmap magic: 0 -- 0x7f30b80ade80 con 0x7f30b81471c0 2024-03-28T14:08:05.716 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.711+0000 7f30be0e5640 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f30b80b91a0 0x7f30b8146c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:05.716 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.711+0000 7f30be0e5640 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f30b80b91a0 0x7f30b8146c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:42620/0 (socket says 172.21.15.104:42620) 2024-03-28T14:08:05.716 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.711+0000 7f30be0e5640 1 -- 172.21.15.104:0/1369609176 learned_addr learned my addr 172.21.15.104:0/1369609176 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-03-28T14:08:05.717 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.715+0000 7f30be8e6640 1 --2- 172.21.15.104:0/1369609176 >> v2:172.21.15.171:3300/0 conn(0x7f30b81471c0 0x7f30b81443a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:05.717 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.715+0000 7f30be0e5640 1 -- 172.21.15.104:0/1369609176 >> v2:172.21.15.171:3300/0 conn(0x7f30b81471c0 msgr2=0x7f30b81443a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T14:08:05.717 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.715+0000 7f30be0e5640 1 --2- 172.21.15.104:0/1369609176 >> v2:172.21.15.171:3300/0 conn(0x7f30b81471c0 0x7f30b81443a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:05.718 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.715+0000 7f30be0e5640 1 -- 172.21.15.104:0/1369609176 >> v1:172.21.15.171:6789/0 conn(0x7f30b80af990 legacy=0x7f30b8143c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T14:08:05.718 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.715+0000 7f30be0e5640 1 -- 172.21.15.104:0/1369609176 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f30b80b5340 con 0x7f30b80b91a0 2024-03-28T14:08:05.719 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.715+0000 7f30be0e5640 1 --2- 172.21.15.104:0/1369609176 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f30b80b91a0 0x7f30b8146c80 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f30b80ae000 tx=0x7f30b401e8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a4c0e1525310cde3 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:08:05.724 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.723+0000 7f3d9bfff640 1 -- 172.21.15.104:0/423286494 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179117 (secure 0 0 0) 0x7f3db40ae600 con 0x7f3db40add50 2024-03-28T14:08:05.739 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.735+0000 7f30bd0e3640 1 -- 172.21.15.104:0/1369609176 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f30b40ad070 con 0x7f30b80b91a0 2024-03-28T14:08:05.744 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.743+0000 7f30bd0e3640 1 -- 172.21.15.104:0/1369609176 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f30b40a7990 con 0x7f30b80b91a0 2024-03-28T14:08:05.744 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.743+0000 7f30bd0e3640 1 -- 172.21.15.104:0/1369609176 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f30b40a7da0 con 0x7f30b80b91a0 2024-03-28T14:08:05.744 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.743+0000 7f30bf0e7640 1 -- 172.21.15.104:0/1369609176 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f30b80b91a0 msgr2=0x7f30b8146c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:05.745 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.743+0000 7f30bf0e7640 1 --2- 172.21.15.104:0/1369609176 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f30b80b91a0 0x7f30b8146c80 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f30b80ae000 tx=0x7f30b401e8c0 comp rx=0 tx=0).stop 2024-03-28T14:08:05.745 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.743+0000 7f30bf0e7640 1 -- 172.21.15.104:0/1369609176 shutdown_connections 2024-03-28T14:08:05.745 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.743+0000 7f30bf0e7640 1 --2- 172.21.15.104:0/1369609176 >> v2:172.21.15.171:3300/0 conn(0x7f30b81471c0 0x7f30b81443a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:05.745 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.743+0000 7f30bf0e7640 1 --2- 172.21.15.104:0/1369609176 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f30b80b91a0 0x7f30b8146c80 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:05.745 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.743+0000 7f30bf0e7640 1 -- 172.21.15.104:0/1369609176 >> 172.21.15.104:0/1369609176 conn(0x7f30b806c530 msgr2=0x7f30b806c940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T14:08:05.746 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.743+0000 7f30bf0e7640 1 -- 172.21.15.104:0/1369609176 shutdown_connections 2024-03-28T14:08:05.747 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.743+0000 7f30bf0e7640 1 -- 172.21.15.104:0/1369609176 wait complete. 2024-03-28T14:08:05.748 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.747+0000 7f30bf0e7640 1 Processor -- start 2024-03-28T14:08:05.748 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.747+0000 7f30bf0e7640 1 -- start start 2024-03-28T14:08:05.749 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.747+0000 7f30bf0e7640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f30b80af990 0x7f30b8135930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:05.749 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.747+0000 7f30bf0e7640 1 --2- >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f30b80b91a0 0x7f30b81388c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:05.750 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.747+0000 7f30bf0e7640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f30b81471c0 0x7f30b8138e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:05.750 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.747+0000 7f30bf0e7640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f30b81507d0 con 0x7f30b81471c0 2024-03-28T14:08:05.750 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.747+0000 7f30bf0e7640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f30b809edd0 con 0x7f30b80af990 2024-03-28T14:08:05.750 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.747+0000 7f30bf0e7640 1 -- --> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] -- mon_getmap magic: 0 -- 0x7f30b8150950 con 0x7f30b80b91a0 2024-03-28T14:08:05.750 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.747+0000 7f30be8e6640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f30b81471c0 0x7f30b8138e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:05.751 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.747+0000 7f30be8e6640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f30b81471c0 0x7f30b8138e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:42630/0 (socket says 172.21.15.104:42630) 2024-03-28T14:08:05.751 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.747+0000 7f30be8e6640 1 -- 172.21.15.104:0/2960680170 learned_addr learned my addr 172.21.15.104:0/2960680170 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-03-28T14:08:05.751 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.747+0000 7f30be0e5640 1 --2- 172.21.15.104:0/2960680170 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f30b80af990 0x7f30b8135930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:05.751 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.747+0000 7f30bd8e4640 1 --2- 172.21.15.104:0/2960680170 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f30b80b91a0 0x7f30b81388c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:05.751 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.747+0000 7f30be8e6640 1 -- 172.21.15.104:0/2960680170 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f30b80b91a0 msgr2=0x7f30b81388c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:05.751 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.747+0000 7f30be8e6640 1 --2- 172.21.15.104:0/2960680170 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f30b80b91a0 0x7f30b81388c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:05.751 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.747+0000 7f30be8e6640 1 -- 172.21.15.104:0/2960680170 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f30b80af990 msgr2=0x7f30b8135930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:05.751 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.747+0000 7f30be8e6640 1 --2- 172.21.15.104:0/2960680170 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f30b80af990 0x7f30b8135930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:05.751 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.747+0000 7f30be8e6640 1 -- 172.21.15.104:0/2960680170 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f30b8132940 con 0x7f30b81471c0 2024-03-28T14:08:05.751 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.747+0000 7f30be8e6640 1 --2- 172.21.15.104:0/2960680170 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f30b81471c0 0x7f30b8138e00 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f30b000c6e0 tx=0x7f30b000cba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:08:05.754 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.751+0000 7f30aeffd640 1 -- 172.21.15.104:0/2960680170 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f30b0023070 con 0x7f30b81471c0 2024-03-28T14:08:05.754 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.751+0000 7f30bf0e7640 1 -- 172.21.15.104:0/2960680170 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f30b81317a0 con 0x7f30b81471c0 2024-03-28T14:08:05.755 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.751+0000 7f30bf0e7640 1 -- 172.21.15.104:0/2960680170 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f30b8132660 con 0x7f30b81471c0 2024-03-28T14:08:05.755 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.751+0000 7f30aeffd640 1 -- 172.21.15.104:0/2960680170 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f30b000b590 con 0x7f30b81471c0 2024-03-28T14:08:05.755 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.751+0000 7f30aeffd640 1 -- 172.21.15.104:0/2960680170 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f30b000b980 con 0x7f30b81471c0 2024-03-28T14:08:05.758 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.755+0000 7f30aeffd640 1 -- 172.21.15.104:0/2960680170 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) ==== 87803+0+0 (secure 0 0 0) 0x7f30b0026070 con 0x7f30b81471c0 2024-03-28T14:08:05.758 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.755+0000 7f30aeffd640 1 --2- 172.21.15.104:0/2960680170 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f30a4068210 0x7f30a406a6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:05.758 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.755+0000 7f30aeffd640 1 -- 172.21.15.104:0/2960680170 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f30b008eb10 con 0x7f30b81471c0 2024-03-28T14:08:05.758 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.755+0000 7f30be0e5640 1 --2- 172.21.15.104:0/2960680170 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f30a4068210 0x7f30a406a6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:05.758 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.755+0000 7f30be0e5640 1 --2- 172.21.15.104:0/2960680170 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f30a4068210 0x7f30a406a6d0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f30b40b2000 tx=0x7f30b400a040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:08:05.763 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.763+0000 7f30bf0e7640 1 -- 172.21.15.104:0/2960680170 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f30a00052a0 con 0x7f30b81471c0 2024-03-28T14:08:05.773 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.771+0000 7f30aeffd640 1 -- 172.21.15.104:0/2960680170 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179117 (secure 0 0 0) 0x7f30b005ece0 con 0x7f30b81471c0 2024-03-28T14:08:05.776 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.775+0000 7f2c96324640 1 Processor -- start 2024-03-28T14:08:05.776 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.775+0000 7f2c96324640 1 -- start start 2024-03-28T14:08:05.779 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.779+0000 7f2c96324640 1 --2- >> v2:172.21.15.171:3300/0 conn(0x7f2c90071b60 0x7f2c9013e1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:05.780 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.779+0000 7f2c96324640 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f2c900b62c0 0x7f2c9013e730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:05.780 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.779+0000 7f2c8f7fe640 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f2c90071f40 0x7f2c90144490 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:47342/0 (socket says 172.21.15.104:47342) 2024-03-28T14:08:05.780 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.779+0000 7f2c8f7fe640 1 -- 172.21.15.104:0/1357415056 learned_addr learned my addr 172.21.15.104:0/1357415056 (peer_addr_for_me v1:172.21.15.104:0/0) 2024-03-28T14:08:05.780 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.779+0000 7f2c96324640 1 -- 172.21.15.104:0/1357415056 --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2c900708d0 con 0x7f2c90071f40 2024-03-28T14:08:05.780 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.779+0000 7f2c96324640 1 -- 172.21.15.104:0/1357415056 --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 -- 0x7f2c900b0bf0 con 0x7f2c900b62c0 2024-03-28T14:08:05.780 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.779+0000 7f2c96324640 1 -- 172.21.15.104:0/1357415056 --> v2:172.21.15.171:3300/0 -- mon_getmap magic: 0 -- 0x7f2c900b0e40 con 0x7f2c90071b60 2024-03-28T14:08:05.781 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.779+0000 7f2c8ffff640 1 --2- 172.21.15.104:0/1357415056 >> v2:172.21.15.171:3300/0 conn(0x7f2c90071b60 0x7f2c9013e1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:05.781 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.779+0000 7f2c948c1640 1 --2- 172.21.15.104:0/1357415056 >> v2:172.21.15.104:3300/0 conn(0x7f2c900b62c0 0x7f2c9013e730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:05.781 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.779+0000 7f2c8ffff640 1 -- 172.21.15.104:0/1357415056 >> v2:172.21.15.104:3300/0 conn(0x7f2c900b62c0 msgr2=0x7f2c9013e730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T14:08:05.781 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.779+0000 7f2c8ffff640 1 --2- 172.21.15.104:0/1357415056 >> v2:172.21.15.104:3300/0 conn(0x7f2c900b62c0 0x7f2c9013e730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:05.781 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.779+0000 7f2c8ffff640 1 -- 172.21.15.104:0/1357415056 >> v1:172.21.15.104:6789/0 conn(0x7f2c90071f40 legacy=0x7f2c90144490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:05.781 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.779+0000 7f2c8ffff640 1 -- 172.21.15.104:0/1357415056 --> v2:172.21.15.171:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2c90070b70 con 0x7f2c90071b60 2024-03-28T14:08:05.784 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.783+0000 7f2c8ffff640 1 --2- 172.21.15.104:0/1357415056 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f2c90071b60 0x7f2c9013e1f0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f2c900b0d70 tx=0x7f2c8401ec50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=277ffbade8c9761d server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:08:05.785 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.783+0000 7f2c8effd640 1 -- 172.21.15.104:0/1357415056 <== mon.1 v2:172.21.15.171:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c840b5070 con 0x7f2c90071b60 2024-03-28T14:08:05.785 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.783+0000 7f2c8effd640 1 -- 172.21.15.104:0/1357415056 <== mon.1 v2:172.21.15.171:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2c840a8950 con 0x7f2c90071b60 2024-03-28T14:08:05.785 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.783+0000 7f2c8effd640 1 -- 172.21.15.104:0/1357415056 <== mon.1 v2:172.21.15.171:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c840a8d20 con 0x7f2c90071b60 2024-03-28T14:08:05.785 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.783+0000 7f2c96324640 1 -- 172.21.15.104:0/1357415056 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f2c90071b60 msgr2=0x7f2c9013e1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:05.786 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.783+0000 7f2c96324640 1 --2- 172.21.15.104:0/1357415056 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f2c90071b60 0x7f2c9013e1f0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f2c900b0d70 tx=0x7f2c8401ec50 comp rx=0 tx=0).stop 2024-03-28T14:08:05.786 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.783+0000 7f2c96324640 1 -- 172.21.15.104:0/1357415056 shutdown_connections 2024-03-28T14:08:05.786 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.783+0000 7f2c96324640 1 --2- 172.21.15.104:0/1357415056 >> v2:172.21.15.104:3300/0 conn(0x7f2c900b62c0 0x7f2c9013e730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:05.786 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.783+0000 7f2c96324640 1 --2- 172.21.15.104:0/1357415056 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f2c90071b60 0x7f2c9013e1f0 secure :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f2c900b0d70 tx=0x7f2c8401ec50 comp rx=0 tx=0).stop 2024-03-28T14:08:05.786 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.783+0000 7f2c96324640 1 -- 172.21.15.104:0/1357415056 >> 172.21.15.104:0/1357415056 conn(0x7f2c900b9d60 msgr2=0x7f2c900ba170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T14:08:05.799 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.799+0000 7f2c96324640 1 -- 172.21.15.104:0/1357415056 shutdown_connections 2024-03-28T14:08:05.799 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.799+0000 7f2c96324640 1 -- 172.21.15.104:0/1357415056 wait complete. 2024-03-28T14:08:05.801 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.799+0000 7f2c96324640 1 Processor -- start 2024-03-28T14:08:05.801 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.799+0000 7f2c96324640 1 -- start start 2024-03-28T14:08:05.802 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.799+0000 7f2c96324640 1 --2- >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f2c90071f40 0x7f2c90141700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:05.802 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.799+0000 7f2c96324640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2c900b62c0 0x7f2c901d93c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:05.802 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.799+0000 7f2c96324640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f2c90145b50 0x7f2c901461d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:05.803 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.799+0000 7f2c96324640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f2c90144d20 con 0x7f2c900b62c0 2024-03-28T14:08:05.803 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.799+0000 7f2c96324640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f2c90144ba0 con 0x7f2c90145b50 2024-03-28T14:08:05.803 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.799+0000 7f2c96324640 1 -- --> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] -- mon_getmap magic: 0 -- 0x7f2c900b3d20 con 0x7f2c90071f40 2024-03-28T14:08:05.803 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.799+0000 7f2c8f7fe640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2c900b62c0 0x7f2c901d93c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:05.803 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.799+0000 7f2c8f7fe640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2c900b62c0 0x7f2c901d93c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:42656/0 (socket says 172.21.15.104:42656) 2024-03-28T14:08:05.803 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.799+0000 7f2c8f7fe640 1 -- 172.21.15.104:0/2598888051 learned_addr learned my addr 172.21.15.104:0/2598888051 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-03-28T14:08:05.803 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.799+0000 7f2c948c1640 1 --2- 172.21.15.104:0/2598888051 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f2c90145b50 0x7f2c901461d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:05.803 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.799+0000 7f2c8ffff640 1 --2- 172.21.15.104:0/2598888051 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f2c90071f40 0x7f2c90141700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:05.804 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.799+0000 7f2c8f7fe640 1 -- 172.21.15.104:0/2598888051 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f2c90071f40 msgr2=0x7f2c90141700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:05.804 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.799+0000 7f2c8f7fe640 1 --2- 172.21.15.104:0/2598888051 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f2c90071f40 0x7f2c90141700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:05.804 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.799+0000 7f2c8f7fe640 1 -- 172.21.15.104:0/2598888051 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f2c90145b50 msgr2=0x7f2c901461d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:05.804 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.799+0000 7f2c8f7fe640 1 --2- 172.21.15.104:0/2598888051 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f2c90145b50 0x7f2c901461d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:05.804 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.799+0000 7f2c8f7fe640 1 -- 172.21.15.104:0/2598888051 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2c90147a60 con 0x7f2c900b62c0 2024-03-28T14:08:05.804 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.803+0000 7f2c8f7fe640 1 --2- 172.21.15.104:0/2598888051 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2c900b62c0 0x7f2c901d93c0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f2c8000c430 tx=0x7f2c8000c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:08:05.811 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.811+0000 7f2c8cff9640 1 -- 172.21.15.104:0/2598888051 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c80017070 con 0x7f2c900b62c0 2024-03-28T14:08:05.811 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.811+0000 7f2c8cff9640 1 -- 172.21.15.104:0/2598888051 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2c80005cf0 con 0x7f2c900b62c0 2024-03-28T14:08:05.811 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.811+0000 7f2c96324640 1 -- 172.21.15.104:0/2598888051 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2c901d9900 con 0x7f2c900b62c0 2024-03-28T14:08:05.811 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.811+0000 7f2c96324640 1 -- 172.21.15.104:0/2598888051 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2c901408e0 con 0x7f2c900b62c0 2024-03-28T14:08:05.823 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.819+0000 7f2c8cff9640 1 -- 172.21.15.104:0/2598888051 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c800072f0 con 0x7f2c900b62c0 2024-03-28T14:08:05.823 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.819+0000 7f2c8cff9640 1 -- 172.21.15.104:0/2598888051 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) ==== 87803+0+0 (secure 0 0 0) 0x7f2c8001a070 con 0x7f2c900b62c0 2024-03-28T14:08:05.823 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.819+0000 7f2c8cff9640 1 --2- 172.21.15.104:0/2598888051 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f2c7c0680c0 0x7f2c7c06a580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:05.823 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.819+0000 7f2c8cff9640 1 -- 172.21.15.104:0/2598888051 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f2c8008a060 con 0x7f2c900b62c0 2024-03-28T14:08:05.823 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.819+0000 7f2c8ffff640 1 --2- 172.21.15.104:0/2598888051 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f2c7c0680c0 0x7f2c7c06a580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:05.823 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.819+0000 7f2c8ffff640 1 --2- 172.21.15.104:0/2598888051 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f2c7c0680c0 0x7f2c7c06a580 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f2c840b2000 tx=0x7f2c84008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:08:05.835 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.835+0000 7f2c96324640 1 -- 172.21.15.104:0/2598888051 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2c700052a0 con 0x7f2c900b62c0 2024-03-28T14:08:05.845 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.843+0000 7f2c8cff9640 1 -- 172.21.15.104:0/2598888051 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179117 (secure 0 0 0) 0x7f2c90147a60 con 0x7f2c900b62c0 2024-03-28T14:08:05.912 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.911+0000 7f315e3aa640 1 -- 172.21.15.104:0/1258104020 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) -- 0x7f3158146280 con 0x7f315806dce0 2024-03-28T14:08:05.913 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.911+0000 7f3154ff9640 1 -- 172.21.15.104:0/1258104020 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f31500f2dc0 con 0x7f315806dce0 2024-03-28T14:08:05.916 INFO:teuthology.orchestra.run.smithi104.stdout:55834574854 2024-03-28T14:08:05.919 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.915+0000 7f313e7fc640 1 -- 172.21.15.104:0/1258104020 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f3144067fc0 msgr2=0x7f314406a480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:05.919 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.915+0000 7f313e7fc640 1 --2- 172.21.15.104:0/1258104020 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f3144067fc0 0x7f314406a480 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f314c006fd0 tx=0x7f314c008040 comp rx=0 tx=0).stop 2024-03-28T14:08:05.919 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.915+0000 7f313e7fc640 1 -- 172.21.15.104:0/1258104020 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f315806dce0 msgr2=0x7f315813a710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:05.919 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.915+0000 7f313e7fc640 1 --2- 172.21.15.104:0/1258104020 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f315806dce0 0x7f315813a710 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f315001d3d0 tx=0x7f315001bac0 comp rx=0 tx=0).stop 2024-03-28T14:08:05.919 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.915+0000 7f313e7fc640 1 -- 172.21.15.104:0/1258104020 shutdown_connections 2024-03-28T14:08:05.919 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.915+0000 7f313e7fc640 1 --2- 172.21.15.104:0/1258104020 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f31581dfae0 0x7f31581341b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:05.919 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.915+0000 7f313e7fc640 1 --2- 172.21.15.104:0/1258104020 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f31580aee40 0x7f31581df5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:05.919 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.915+0000 7f313e7fc640 1 --2- 172.21.15.104:0/1258104020 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f3144067fc0 0x7f314406a480 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:05.919 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.915+0000 7f313e7fc640 1 --2- 172.21.15.104:0/1258104020 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f315806dce0 0x7f315813a710 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:05.919 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.915+0000 7f313e7fc640 1 -- 172.21.15.104:0/1258104020 >> 172.21.15.104:0/1258104020 conn(0x7f31580b57a0 msgr2=0x7f31580b3df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T14:08:05.927 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.927+0000 7f313e7fc640 1 -- 172.21.15.104:0/1258104020 shutdown_connections 2024-03-28T14:08:05.927 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.927+0000 7f313e7fc640 1 -- 172.21.15.104:0/1258104020 wait complete. 2024-03-28T14:08:05.954 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574854 got 55834574854 for osd.5 2024-03-28T14:08:05.955 DEBUG:teuthology.parallel:result is None 2024-03-28T14:08:05.967 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.967+0000 7f0273b77640 1 -- 172.21.15.104:0/736062044 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) -- 0x7f0264039470 con 0x7f0264033920 2024-03-28T14:08:05.968 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.967+0000 7f02627fc640 1 -- 172.21.15.104:0/736062044 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f026c0d2060 con 0x7f0264033920 2024-03-28T14:08:05.968 INFO:teuthology.orchestra.run.smithi104.stdout:55834574856 2024-03-28T14:08:05.971 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.971+0000 7f0273b77640 1 -- 172.21.15.104:0/736062044 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f0250068010 msgr2=0x7f025006a4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:05.971 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.971+0000 7f0273b77640 1 --2- 172.21.15.104:0/736062044 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f0250068010 0x7f025006a4d0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f025c000c10 tx=0x7f025c007f90 comp rx=0 tx=0).stop 2024-03-28T14:08:05.972 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.971+0000 7f0273b77640 1 -- 172.21.15.104:0/736062044 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0264033920 msgr2=0x7f026403caa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:05.972 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.971+0000 7f0273b77640 1 --2- 172.21.15.104:0/736062044 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0264033920 0x7f026403caa0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f026c04fde0 tx=0x7f026c069120 comp rx=0 tx=0).stop 2024-03-28T14:08:05.972 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.971+0000 7f0273b77640 1 -- 172.21.15.104:0/736062044 shutdown_connections 2024-03-28T14:08:05.972 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.971+0000 7f0273b77640 1 --2- 172.21.15.104:0/736062044 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f02641751b0 0x7f026416b8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:05.972 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.971+0000 7f0273b77640 1 --2- 172.21.15.104:0/736062044 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f02640c8600 0x7f02640c5a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:05.972 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.971+0000 7f0273b77640 1 --2- 172.21.15.104:0/736062044 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f0250068010 0x7f025006a4d0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:05.972 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.971+0000 7f0273b77640 1 --2- 172.21.15.104:0/736062044 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0264033920 0x7f026403caa0 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:05.972 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.971+0000 7f0273b77640 1 -- 172.21.15.104:0/736062044 >> 172.21.15.104:0/736062044 conn(0x7f026403a860 msgr2=0x7f02640d21e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T14:08:05.972 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.971+0000 7f0273b77640 1 -- 172.21.15.104:0/736062044 shutdown_connections 2024-03-28T14:08:05.973 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.971+0000 7f6b814be640 1 -- 172.21.15.104:0/2660765173 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) -- 0x7f6b5c005860 con 0x7f6b7c14e360 2024-03-28T14:08:05.973 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.971+0000 7f6b78ff9640 1 -- 172.21.15.104:0/2660765173 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f6b6c059a40 con 0x7f6b7c14e360 2024-03-28T14:08:05.973 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.971+0000 7f0273b77640 1 -- 172.21.15.104:0/736062044 wait complete. 2024-03-28T14:08:05.973 INFO:teuthology.orchestra.run.smithi104.stdout:55834574855 2024-03-28T14:08:05.992 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.991+0000 7f6b627fc640 1 -- 172.21.15.104:0/2660765173 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f6b68068040 msgr2=0x7f6b6806a500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:05.992 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.991+0000 7f6b627fc640 1 --2- 172.21.15.104:0/2660765173 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f6b68068040 0x7f6b6806a500 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f6b7c0b8940 tx=0x7f6b74009040 comp rx=0 tx=0).stop 2024-03-28T14:08:05.992 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.991+0000 7f6b627fc640 1 -- 172.21.15.104:0/2660765173 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6b7c14e360 msgr2=0x7f6b7c09d210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:05.992 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.991+0000 7f6b627fc640 1 --2- 172.21.15.104:0/2660765173 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6b7c14e360 0x7f6b7c09d210 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f6b6c00b430 tx=0x7f6b6c00b8f0 comp rx=0 tx=0).stop 2024-03-28T14:08:05.992 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.991+0000 7f6b627fc640 1 -- 172.21.15.104:0/2660765173 shutdown_connections 2024-03-28T14:08:05.992 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.991+0000 7f6b627fc640 1 --2- 172.21.15.104:0/2660765173 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f6b68068040 0x7f6b6806a500 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:05.992 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.991+0000 7f6b627fc640 1 --2- 172.21.15.104:0/2660765173 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6b7c14e360 0x7f6b7c09d210 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:05.992 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.991+0000 7f6b627fc640 1 --2- 172.21.15.104:0/2660765173 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f6b7c0b8c00 0x7f6b7c09c440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:05.992 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.991+0000 7f6b627fc640 1 --2- 172.21.15.104:0/2660765173 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f6b7c0add50 0x7f6b7c070a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:05.992 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.991+0000 7f6b627fc640 1 -- 172.21.15.104:0/2660765173 >> 172.21.15.104:0/2660765173 conn(0x7f6b7c06c530 msgr2=0x7f6b7c14bb20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T14:08:05.993 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.991+0000 7f6b627fc640 1 -- 172.21.15.104:0/2660765173 shutdown_connections 2024-03-28T14:08:05.993 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:05.991+0000 7f6b627fc640 1 -- 172.21.15.104:0/2660765173 wait complete. 2024-03-28T14:08:05.997 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574856 got 55834574856 for osd.0 2024-03-28T14:08:05.997 DEBUG:teuthology.parallel:result is None 2024-03-28T14:08:06.003 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:06.003+0000 7ffa00296640 1 -- 172.21.15.104:0/385213865 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) -- 0x7ff9e0005590 con 0x7ff9f806dce0 2024-03-28T14:08:06.004 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:06.003+0000 7ff9eeffd640 1 -- 172.21.15.104:0/385213865 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7ff9f0016620 con 0x7ff9f806dce0 2024-03-28T14:08:06.004 INFO:teuthology.orchestra.run.smithi104.stdout:55834574853 2024-03-28T14:08:06.007 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:06.003+0000 7ffa00296640 1 -- 172.21.15.104:0/385213865 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7ff9e4068160 msgr2=0x7ff9e406a620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:06.007 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:06.003+0000 7ffa00296640 1 --2- 172.21.15.104:0/385213865 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7ff9e4068160 0x7ff9e406a620 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7ff9f8144c60 tx=0x7ff9e8009040 comp rx=0 tx=0).stop 2024-03-28T14:08:06.007 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:06.007+0000 7ffa00296640 1 -- 172.21.15.104:0/385213865 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff9f806dce0 msgr2=0x7ff9f8143d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:06.007 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:06.007+0000 7ffa00296640 1 --2- 172.21.15.104:0/385213865 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff9f806dce0 0x7ff9f8143d20 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7ff9f001a5d0 tx=0x7ff9f0019740 comp rx=0 tx=0).stop 2024-03-28T14:08:06.008 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:06.007+0000 7ffa00296640 1 -- 172.21.15.104:0/385213865 shutdown_connections 2024-03-28T14:08:06.008 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:06.007+0000 7ffa00296640 1 --2- 172.21.15.104:0/385213865 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7ff9f80b4910 0x7ff9f8141ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:06.008 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:06.007+0000 7ffa00296640 1 --2- 172.21.15.104:0/385213865 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7ff9f80aee40 0x7ff9f81419a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:06.008 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:06.007+0000 7ffa00296640 1 --2- 172.21.15.104:0/385213865 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7ff9e4068160 0x7ff9e406a620 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:06.008 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:06.007+0000 7ffa00296640 1 --2- 172.21.15.104:0/385213865 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff9f806dce0 0x7ff9f8143d20 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:06.008 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:06.007+0000 7ffa00296640 1 -- 172.21.15.104:0/385213865 >> 172.21.15.104:0/385213865 conn(0x7ff9f80b57a0 msgr2=0x7ff9f80b3df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T14:08:06.008 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:06.007+0000 7ffa00296640 1 -- 172.21.15.104:0/385213865 shutdown_connections 2024-03-28T14:08:06.008 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:06.007+0000 7ffa00296640 1 -- 172.21.15.104:0/385213865 wait complete. 2024-03-28T14:08:06.019 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574855 got 55834574855 for osd.2 2024-03-28T14:08:06.019 DEBUG:teuthology.parallel:result is None 2024-03-28T14:08:06.023 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:06.023+0000 7efebffff640 1 -- 172.21.15.104:0/1593155518 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) -- 0x7efec0068d50 con 0x7efec00b6050 2024-03-28T14:08:06.023 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:06.023+0000 7efeb77fe640 1 -- 172.21.15.104:0/1593155518 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7efeb00f16e0 con 0x7efec00b6050 2024-03-28T14:08:06.023 INFO:teuthology.orchestra.run.smithi104.stdout:55834574856 2024-03-28T14:08:06.027 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:06.023+0000 7efebffff640 1 -- 172.21.15.104:0/1593155518 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7efea0067fc0 msgr2=0x7efea006a480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:06.027 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:06.023+0000 7efebffff640 1 --2- 172.21.15.104:0/1593155518 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7efea0067fc0 0x7efea006a480 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7efea8004e30 tx=0x7efea8009040 comp rx=0 tx=0).stop 2024-03-28T14:08:06.027 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:06.023+0000 7efebffff640 1 -- 172.21.15.104:0/1593155518 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7efec00b6050 msgr2=0x7efec01e4050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:06.027 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:06.023+0000 7efebffff640 1 --2- 172.21.15.104:0/1593155518 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7efec00b6050 0x7efec01e4050 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7efeb001c490 tx=0x7efeb001b600 comp rx=0 tx=0).stop 2024-03-28T14:08:06.027 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:06.023+0000 7efebffff640 1 -- 172.21.15.104:0/1593155518 shutdown_connections 2024-03-28T14:08:06.027 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:06.023+0000 7efebffff640 1 --2- 172.21.15.104:0/1593155518 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7efec0144840 0x7efec01e4590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:06.027 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:06.023+0000 7efebffff640 1 --2- 172.21.15.104:0/1593155518 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7efea0067fc0 0x7efea006a480 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:06.027 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:06.023+0000 7efebffff640 1 --2- 172.21.15.104:0/1593155518 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7efec00b6050 0x7efec01e4050 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:06.027 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:06.023+0000 7efebffff640 1 --2- 172.21.15.104:0/1593155518 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7efec00b4480 0x7efec0146b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:06.027 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:06.023+0000 7efebffff640 1 -- 172.21.15.104:0/1593155518 >> 172.21.15.104:0/1593155518 conn(0x7efec00b9450 msgr2=0x7efec00b7250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T14:08:06.028 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:06.023+0000 7efebffff640 1 -- 172.21.15.104:0/1593155518 shutdown_connections 2024-03-28T14:08:06.028 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:06.023+0000 7efebffff640 1 -- 172.21.15.104:0/1593155518 wait complete. 2024-03-28T14:08:06.030 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574853 got 55834574853 for osd.1 2024-03-28T14:08:06.030 DEBUG:teuthology.parallel:result is None 2024-03-28T14:08:06.046 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:06.043+0000 7f3db8fdb640 1 -- 172.21.15.104:0/423286494 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) -- 0x7f3d94005590 con 0x7f3db40add50 2024-03-28T14:08:06.047 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574856 got 55834574856 for osd.6 2024-03-28T14:08:06.048 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:06.043+0000 7f3d9bfff640 1 -- 172.21.15.104:0/423286494 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f3da8055ca0 con 0x7f3db40add50 2024-03-28T14:08:06.048 INFO:teuthology.orchestra.run.smithi104.stdout:55834574855 2024-03-28T14:08:06.048 DEBUG:teuthology.parallel:result is None 2024-03-28T14:08:06.054 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:06.047+0000 7f3db8fdb640 1 -- 172.21.15.104:0/423286494 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f3da00680c0 msgr2=0x7f3da006a580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:06.054 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:06.047+0000 7f3db8fdb640 1 --2- 172.21.15.104:0/423286494 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f3da00680c0 0x7f3da006a580 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f3db406ca60 tx=0x7f3da400e040 comp rx=0 tx=0).stop 2024-03-28T14:08:06.054 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:06.047+0000 7f3db8fdb640 1 -- 172.21.15.104:0/423286494 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3db40add50 msgr2=0x7f3db40718f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:06.055 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:06.047+0000 7f3db8fdb640 1 --2- 172.21.15.104:0/423286494 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3db40add50 0x7f3db40718f0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f3da800c7d0 tx=0x7f3da800cc90 comp rx=0 tx=0).stop 2024-03-28T14:08:06.055 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:06.047+0000 7f3db8fdb640 1 -- 172.21.15.104:0/423286494 shutdown_connections 2024-03-28T14:08:06.055 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:06.047+0000 7f3db8fdb640 1 --2- 172.21.15.104:0/423286494 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f3db4157fc0 0x7f3db41f0e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:06.055 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:06.047+0000 7f3db8fdb640 1 --2- 172.21.15.104:0/423286494 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f3db40b8c50 0x7f3db41f0910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:06.055 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:06.047+0000 7f3db8fdb640 1 --2- 172.21.15.104:0/423286494 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f3da00680c0 0x7f3da006a580 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:06.055 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:06.047+0000 7f3db8fdb640 1 --2- 172.21.15.104:0/423286494 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3db40add50 0x7f3db40718f0 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:06.055 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:06.047+0000 7f3db8fdb640 1 -- 172.21.15.104:0/423286494 >> 172.21.15.104:0/423286494 conn(0x7f3db406c530 msgr2=0x7f3db40b7bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T14:08:06.055 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:06.047+0000 7f3db8fdb640 1 -- 172.21.15.104:0/423286494 shutdown_connections 2024-03-28T14:08:06.055 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:06.047+0000 7f3db8fdb640 1 -- 172.21.15.104:0/423286494 wait complete. 2024-03-28T14:08:06.063 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:06.063+0000 7f30bf0e7640 1 -- 172.21.15.104:0/2960680170 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) -- 0x7f30a0005590 con 0x7f30b81471c0 2024-03-28T14:08:06.064 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:06.063+0000 7f30aeffd640 1 -- 172.21.15.104:0/2960680170 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f30b0062fe0 con 0x7f30b81471c0 2024-03-28T14:08:06.064 INFO:teuthology.orchestra.run.smithi104.stdout:55834574855 2024-03-28T14:08:06.066 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:06.063+0000 7f30bf0e7640 1 -- 172.21.15.104:0/2960680170 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f30a4068210 msgr2=0x7f30a406a6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:06.066 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:06.063+0000 7f30bf0e7640 1 --2- 172.21.15.104:0/2960680170 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f30a4068210 0x7f30a406a6d0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f30b40b2000 tx=0x7f30b400a040 comp rx=0 tx=0).stop 2024-03-28T14:08:06.066 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:06.063+0000 7f30bf0e7640 1 -- 172.21.15.104:0/2960680170 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f30b81471c0 msgr2=0x7f30b8138e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:06.066 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:06.063+0000 7f30bf0e7640 1 --2- 172.21.15.104:0/2960680170 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f30b81471c0 0x7f30b8138e00 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f30b000c6e0 tx=0x7f30b000cba0 comp rx=0 tx=0).stop 2024-03-28T14:08:06.066 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:06.063+0000 7f30bf0e7640 1 -- 172.21.15.104:0/2960680170 shutdown_connections 2024-03-28T14:08:06.067 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:06.063+0000 7f30bf0e7640 1 --2- 172.21.15.104:0/2960680170 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f30a4068210 0x7f30a406a6d0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:06.067 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:06.063+0000 7f30bf0e7640 1 --2- 172.21.15.104:0/2960680170 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f30b81471c0 0x7f30b8138e00 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:06.067 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:06.063+0000 7f30bf0e7640 1 --2- 172.21.15.104:0/2960680170 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f30b80b91a0 0x7f30b81388c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:06.067 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:06.063+0000 7f30bf0e7640 1 --2- 172.21.15.104:0/2960680170 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f30b80af990 0x7f30b8135930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:06.067 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:06.063+0000 7f30bf0e7640 1 -- 172.21.15.104:0/2960680170 >> 172.21.15.104:0/2960680170 conn(0x7f30b806c530 msgr2=0x7f30b80b5540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T14:08:06.067 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:06.063+0000 7f30bf0e7640 1 -- 172.21.15.104:0/2960680170 shutdown_connections 2024-03-28T14:08:06.067 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:06.063+0000 7f30bf0e7640 1 -- 172.21.15.104:0/2960680170 wait complete. 2024-03-28T14:08:06.068 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574855 got 55834574855 for osd.4 2024-03-28T14:08:06.069 DEBUG:teuthology.parallel:result is None 2024-03-28T14:08:06.081 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574855 got 55834574855 for osd.7 2024-03-28T14:08:06.081 DEBUG:teuthology.parallel:result is None 2024-03-28T14:08:06.141 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:06.139+0000 7f2c96324640 1 -- 172.21.15.104:0/2598888051 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) -- 0x7f2c70005860 con 0x7f2c900b62c0 2024-03-28T14:08:06.141 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:06.139+0000 7f2c8cff9640 1 -- 172.21.15.104:0/2598888051 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f2c80055850 con 0x7f2c900b62c0 2024-03-28T14:08:06.141 INFO:teuthology.orchestra.run.smithi104.stdout:55834574857 2024-03-28T14:08:06.143 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:06.139+0000 7f2c96324640 1 -- 172.21.15.104:0/2598888051 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f2c7c0680c0 msgr2=0x7f2c7c06a580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:06.143 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:06.139+0000 7f2c96324640 1 --2- 172.21.15.104:0/2598888051 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f2c7c0680c0 0x7f2c7c06a580 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f2c840b2000 tx=0x7f2c84008040 comp rx=0 tx=0).stop 2024-03-28T14:08:06.143 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:06.139+0000 7f2c96324640 1 -- 172.21.15.104:0/2598888051 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2c900b62c0 msgr2=0x7f2c901d93c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:06.143 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:06.139+0000 7f2c96324640 1 --2- 172.21.15.104:0/2598888051 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2c900b62c0 0x7f2c901d93c0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f2c8000c430 tx=0x7f2c8000c8f0 comp rx=0 tx=0).stop 2024-03-28T14:08:06.143 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:06.139+0000 7f2c96324640 1 -- 172.21.15.104:0/2598888051 shutdown_connections 2024-03-28T14:08:06.143 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:06.139+0000 7f2c96324640 1 --2- 172.21.15.104:0/2598888051 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f2c90145b50 0x7f2c901461d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:06.143 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:06.139+0000 7f2c96324640 1 --2- 172.21.15.104:0/2598888051 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f2c7c0680c0 0x7f2c7c06a580 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:06.144 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:06.139+0000 7f2c96324640 1 --2- 172.21.15.104:0/2598888051 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2c900b62c0 0x7f2c901d93c0 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:06.144 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:06.139+0000 7f2c96324640 1 --2- 172.21.15.104:0/2598888051 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f2c90071f40 0x7f2c90141700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:06.144 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:06.139+0000 7f2c96324640 1 -- 172.21.15.104:0/2598888051 >> 172.21.15.104:0/2598888051 conn(0x7f2c900b9d60 msgr2=0x7f2c9013c3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T14:08:06.144 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:06.139+0000 7f2c96324640 1 -- 172.21.15.104:0/2598888051 shutdown_connections 2024-03-28T14:08:06.144 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:06.139+0000 7f2c96324640 1 -- 172.21.15.104:0/2598888051 wait complete. 2024-03-28T14:08:06.164 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574857 got 55834574857 for osd.3 2024-03-28T14:08:06.165 DEBUG:teuthology.parallel:result is None 2024-03-28T14:08:06.165 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-03-28T14:08:06.165 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-28T14:08:06.279 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:06.275+0000 7fe6f77dd640 1 Processor -- start 2024-03-28T14:08:06.279 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:06.275+0000 7fe6f77dd640 1 -- start start 2024-03-28T14:08:06.280 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:06.279+0000 7fe6f77dd640 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fe6f0148ef0 0x7fe6f0149380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:06.280 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:06.279+0000 7fe6f77dd640 1 --2- >> v2:172.21.15.171:3300/0 conn(0x7fe6f01498c0 0x7fe6f0150350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:06.281 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:06.279+0000 7fe6f77dd640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe6f00badf0 con 0x7fe6f00baa10 2024-03-28T14:08:06.281 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:06.279+0000 7fe6f77dd640 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 -- 0x7fe6f00b6860 con 0x7fe6f0148ef0 2024-03-28T14:08:06.281 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:06.279+0000 7fe6f77dd640 1 -- --> v2:172.21.15.171:3300/0 -- mon_getmap magic: 0 -- 0x7fe6f0149cd0 con 0x7fe6f01498c0 2024-03-28T14:08:06.281 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:06.279+0000 7fe6f4d78640 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fe6f0148ef0 0x7fe6f0149380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:06.281 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:06.279+0000 7fe6f5579640 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7fe6f00baa10 0x7fe6f009e6b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:47344/0 (socket says 172.21.15.104:47344) 2024-03-28T14:08:06.281 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:06.279+0000 7fe6f5579640 1 -- 172.21.15.104:0/2481007463 learned_addr learned my addr 172.21.15.104:0/2481007463 (peer_addr_for_me v1:172.21.15.104:0/0) 2024-03-28T14:08:06.281 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:06.279+0000 7fe6f4d78640 1 -- 172.21.15.104:0/2481007463 >> v2:172.21.15.171:3300/0 conn(0x7fe6f01498c0 msgr2=0x7fe6f0150350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T14:08:06.281 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:06.279+0000 7fe6f5d7a640 1 --2- 172.21.15.104:0/2481007463 >> v2:172.21.15.171:3300/0 conn(0x7fe6f01498c0 0x7fe6f0150350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:06.281 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:06.279+0000 7fe6e7fff640 1 -- 172.21.15.104:0/2481007463 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2504654601 0 0) 0x7fe6f00badf0 con 0x7fe6f00baa10 2024-03-28T14:08:06.281 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:06.279+0000 7fe6f4d78640 1 --2- 172.21.15.104:0/2481007463 >> v2:172.21.15.171:3300/0 conn(0x7fe6f01498c0 0x7fe6f0150350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:06.281 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:06.279+0000 7fe6f4d78640 1 -- 172.21.15.104:0/2481007463 >> v1:172.21.15.104:6789/0 conn(0x7fe6f00baa10 legacy=0x7fe6f009e6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:06.281 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:06.279+0000 7fe6f4d78640 1 -- 172.21.15.104:0/2481007463 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe6f009d740 con 0x7fe6f0148ef0 2024-03-28T14:08:06.281 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:06.279+0000 7fe6f5d7a640 1 --2- 172.21.15.104:0/2481007463 >> v2:172.21.15.171:3300/0 conn(0x7fe6f01498c0 0x7fe6f0150350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T14:08:06.284 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:06.283+0000 7fe6f4d78640 1 --2- 172.21.15.104:0/2481007463 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe6f0148ef0 0x7fe6f0149380 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fe6e00097c0 tx=0x7fe6e001c3d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4b5584cd89aedfc6 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:08:06.284 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:06.283+0000 7fe6e7fff640 1 -- 172.21.15.104:0/2481007463 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6e00b0070 con 0x7fe6f0148ef0 2024-03-28T14:08:06.284 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:06.283+0000 7fe6e7fff640 1 -- 172.21.15.104:0/2481007463 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe6e00ab600 con 0x7fe6f0148ef0 2024-03-28T14:08:06.284 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:06.283+0000 7fe6e7fff640 1 -- 172.21.15.104:0/2481007463 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6e00ab9d0 con 0x7fe6f0148ef0 2024-03-28T14:08:06.285 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:06.283+0000 7fe6f77dd640 1 -- 172.21.15.104:0/2481007463 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe6f0148ef0 msgr2=0x7fe6f0149380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:06.285 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:06.283+0000 7fe6f77dd640 1 --2- 172.21.15.104:0/2481007463 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe6f0148ef0 0x7fe6f0149380 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fe6e00097c0 tx=0x7fe6e001c3d0 comp rx=0 tx=0).stop 2024-03-28T14:08:06.285 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:06.283+0000 7fe6f77dd640 1 -- 172.21.15.104:0/2481007463 shutdown_connections 2024-03-28T14:08:06.285 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:06.283+0000 7fe6f77dd640 1 --2- 172.21.15.104:0/2481007463 >> v2:172.21.15.171:3300/0 conn(0x7fe6f01498c0 0x7fe6f0150350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:06.285 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:06.283+0000 7fe6f77dd640 1 --2- 172.21.15.104:0/2481007463 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe6f0148ef0 0x7fe6f0149380 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:06.285 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:06.283+0000 7fe6f77dd640 1 -- 172.21.15.104:0/2481007463 >> 172.21.15.104:0/2481007463 conn(0x7fe6f006cd40 msgr2=0x7fe6f00ba0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T14:08:06.285 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:06.283+0000 7fe6f77dd640 1 -- 172.21.15.104:0/2481007463 shutdown_connections 2024-03-28T14:08:06.285 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:06.283+0000 7fe6f77dd640 1 -- 172.21.15.104:0/2481007463 wait complete. 2024-03-28T14:08:06.287 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:06.283+0000 7fe6f77dd640 1 Processor -- start 2024-03-28T14:08:06.287 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:06.283+0000 7fe6f77dd640 1 -- start start 2024-03-28T14:08:06.287 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:06.283+0000 7fe6f77dd640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fe6f00baa10 0x7fe6f01478f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:06.288 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:06.283+0000 7fe6f77dd640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe6f0148ef0 0x7fe6f00b24b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:06.288 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:06.283+0000 7fe6f77dd640 1 --2- >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fe6f01498c0 0x7fe6f0146b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:06.288 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:06.283+0000 7fe6f77dd640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7fe6f00b4aa0 con 0x7fe6f0148ef0 2024-03-28T14:08:06.288 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:06.283+0000 7fe6f77dd640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7fe6f009e190 con 0x7fe6f00baa10 2024-03-28T14:08:06.288 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:06.283+0000 7fe6f77dd640 1 -- --> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] -- mon_getmap magic: 0 -- 0x7fe6f009d370 con 0x7fe6f01498c0 2024-03-28T14:08:06.288 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:06.283+0000 7fe6f4d78640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe6f0148ef0 0x7fe6f00b24b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:06.288 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:06.283+0000 7fe6f4d78640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe6f0148ef0 0x7fe6f00b24b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:42670/0 (socket says 172.21.15.104:42670) 2024-03-28T14:08:06.288 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:06.283+0000 7fe6f4d78640 1 -- 172.21.15.104:0/2860692998 learned_addr learned my addr 172.21.15.104:0/2860692998 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-03-28T14:08:06.288 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:06.283+0000 7fe6f5d7a640 1 --2- 172.21.15.104:0/2860692998 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fe6f01498c0 0x7fe6f0146b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:06.288 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:06.283+0000 7fe6f5579640 1 --2- 172.21.15.104:0/2860692998 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fe6f00baa10 0x7fe6f01478f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:06.288 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:06.283+0000 7fe6f4d78640 1 -- 172.21.15.104:0/2860692998 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fe6f01498c0 msgr2=0x7fe6f0146b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:06.288 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:06.283+0000 7fe6f4d78640 1 --2- 172.21.15.104:0/2860692998 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fe6f01498c0 0x7fe6f0146b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:06.288 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:06.283+0000 7fe6f4d78640 1 -- 172.21.15.104:0/2860692998 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fe6f00baa10 msgr2=0x7fe6f01478f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:06.288 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:06.283+0000 7fe6f4d78640 1 --2- 172.21.15.104:0/2860692998 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fe6f00baa10 0x7fe6f01478f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:06.289 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:06.283+0000 7fe6f4d78640 1 -- 172.21.15.104:0/2860692998 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe6f0148070 con 0x7fe6f0148ef0 2024-03-28T14:08:06.289 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:06.283+0000 7fe6f4d78640 1 --2- 172.21.15.104:0/2860692998 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe6f0148ef0 0x7fe6f00b24b0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fe6e0018fe0 tx=0x7fe6e0019170 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:08:06.289 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:06.283+0000 7fe6e5ffb640 1 -- 172.21.15.104:0/2860692998 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6e00b0040 con 0x7fe6f0148ef0 2024-03-28T14:08:06.289 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:06.287+0000 7fe6e5ffb640 1 -- 172.21.15.104:0/2860692998 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe6e00b43f0 con 0x7fe6f0148ef0 2024-03-28T14:08:06.289 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:06.287+0000 7fe6f77dd640 1 -- 172.21.15.104:0/2860692998 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe6f0145830 con 0x7fe6f0148ef0 2024-03-28T14:08:06.289 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:06.287+0000 7fe6e5ffb640 1 -- 172.21.15.104:0/2860692998 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6e00b47c0 con 0x7fe6f0148ef0 2024-03-28T14:08:06.289 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:06.287+0000 7fe6f77dd640 1 -- 172.21.15.104:0/2860692998 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe6f0145db0 con 0x7fe6f0148ef0 2024-03-28T14:08:06.290 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:06.287+0000 7fe6f77dd640 1 -- 172.21.15.104:0/2860692998 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe6f00687c0 con 0x7fe6f0148ef0 2024-03-28T14:08:06.290 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:06.287+0000 7fe6e5ffb640 1 -- 172.21.15.104:0/2860692998 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) ==== 87803+0+0 (secure 0 0 0) 0x7fe6e00b4960 con 0x7fe6f0148ef0 2024-03-28T14:08:06.290 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:06.287+0000 7fe6e5ffb640 1 --2- 172.21.15.104:0/2860692998 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7fe6d4067fc0 0x7fe6d406a480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:06.290 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:06.287+0000 7fe6e5ffb640 1 -- 172.21.15.104:0/2860692998 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7fe6e01200b0 con 0x7fe6f0148ef0 2024-03-28T14:08:06.290 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:06.287+0000 7fe6f5579640 1 --2- 172.21.15.104:0/2860692998 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7fe6d4067fc0 0x7fe6d406a480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:06.290 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:06.287+0000 7fe6f5579640 1 --2- 172.21.15.104:0/2860692998 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7fe6d4067fc0 0x7fe6d406a480 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fe6ec007fd0 tx=0x7fe6ec007ee0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:08:06.293 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:06.291+0000 7fe6e5ffb640 1 -- 172.21.15.104:0/2860692998 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179117 (secure 0 0 0) 0x7fe6f0148070 con 0x7fe6f0148ef0 2024-03-28T14:08:06.438 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:06.435+0000 7fe6f77dd640 1 -- 172.21.15.104:0/2860692998 --> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fe6f00b2a50 con 0x7fe6d4067fc0 2024-03-28T14:08:06.456 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:06.455+0000 7fe6e5ffb640 1 -- 172.21.15.104:0/2860692998 <== mgr.4100 v2:172.21.15.171:6832/1522752412 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+251836 (secure 0 0 0) 0x7fe6f00b2a50 con 0x7fe6d4067fc0 2024-03-28T14:08:06.458 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-03-28T14:08:06.459 INFO:teuthology.orchestra.run.smithi104.stderr:dumped all 2024-03-28T14:08:06.460 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:06.455+0000 7fe6f77dd640 1 -- 172.21.15.104:0/2860692998 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7fe6d4067fc0 msgr2=0x7fe6d406a480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:06.461 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:06.455+0000 7fe6f77dd640 1 --2- 172.21.15.104:0/2860692998 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7fe6d4067fc0 0x7fe6d406a480 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fe6ec007fd0 tx=0x7fe6ec007ee0 comp rx=0 tx=0).stop 2024-03-28T14:08:06.461 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:06.455+0000 7fe6f77dd640 1 -- 172.21.15.104:0/2860692998 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe6f0148ef0 msgr2=0x7fe6f00b24b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:06.461 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:06.455+0000 7fe6f77dd640 1 --2- 172.21.15.104:0/2860692998 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe6f0148ef0 0x7fe6f00b24b0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fe6e0018fe0 tx=0x7fe6e0019170 comp rx=0 tx=0).stop 2024-03-28T14:08:06.461 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:06.455+0000 7fe6f77dd640 1 -- 172.21.15.104:0/2860692998 shutdown_connections 2024-03-28T14:08:06.461 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:06.455+0000 7fe6f77dd640 1 --2- 172.21.15.104:0/2860692998 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fe6f01498c0 0x7fe6f0146b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:06.461 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:06.455+0000 7fe6f77dd640 1 --2- 172.21.15.104:0/2860692998 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7fe6d4067fc0 0x7fe6d406a480 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:06.461 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:06.455+0000 7fe6f77dd640 1 --2- 172.21.15.104:0/2860692998 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe6f0148ef0 0x7fe6f00b24b0 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:06.461 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:06.455+0000 7fe6f77dd640 1 --2- 172.21.15.104:0/2860692998 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fe6f00baa10 0x7fe6f01478f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:06.461 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:06.455+0000 7fe6f77dd640 1 -- 172.21.15.104:0/2860692998 >> 172.21.15.104:0/2860692998 conn(0x7fe6f006cd40 msgr2=0x7fe6f01448a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T14:08:06.461 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:06.455+0000 7fe6f77dd640 1 -- 172.21.15.104:0/2860692998 shutdown_connections 2024-03-28T14:08:06.461 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:06.455+0000 7fe6f77dd640 1 -- 172.21.15.104:0/2860692998 wait complete. 2024-03-28T14:08:06.474 INFO:teuthology.orchestra.run.smithi104.stdout:{"pg_ready":true,"pg_map":{"version":24,"stamp":"2024-03-28T14:08:05.579087+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219108,"kb_used_data":2920,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755612,"statfs":{"total":773094113280,"available":772869746688,"internally_reserved":0,"allocated":2990080,"data_stored":1790396,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":221236785},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.930250"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-28T14:08:00.835047+0000","last_change":"2024-03-28T14:07:59.730372+0000","last_active":"2024-03-28T14:08:00.835047+0000","last_peered":"2024-03-28T14:08:00.835047+0000","last_clean":"2024-03-28T14:08:00.835047+0000","last_became_active":"2024-03-28T14:07:59.729644+0000","last_became_peered":"2024-03-28T14:07:59.729644+0000","last_unstale":"2024-03-28T14:08:00.835047+0000","last_undegraded":"2024-03-28T14:08:00.835047+0000","last_fullsized":"2024-03-28T14:08:00.835047+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_clean_scrub_stamp":"2024-03-28T14:07:58.641775+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:59:25.862480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-28T14:08:00.696778+0000","last_change":"2024-03-28T14:07:59.705187+0000","last_active":"2024-03-28T14:08:00.696778+0000","last_peered":"2024-03-28T14:08:00.696778+0000","last_clean":"2024-03-28T14:08:00.696778+0000","last_became_active":"2024-03-28T14:07:59.700386+0000","last_became_peered":"2024-03-28T14:07:59.700386+0000","last_unstale":"2024-03-28T14:08:00.696778+0000","last_undegraded":"2024-03-28T14:08:00.696778+0000","last_fullsized":"2024-03-28T14:08:00.696778+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_clean_scrub_stamp":"2024-03-28T14:07:58.641775+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T16:47:04.317583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-28T14:08:01.245347+0000","last_change":"2024-03-28T14:07:59.698753+0000","last_active":"2024-03-28T14:08:01.245347+0000","last_peered":"2024-03-28T14:08:01.245347+0000","last_clean":"2024-03-28T14:08:01.245347+0000","last_became_active":"2024-03-28T14:07:59.698467+0000","last_became_peered":"2024-03-28T14:07:59.698467+0000","last_unstale":"2024-03-28T14:08:01.245347+0000","last_undegraded":"2024-03-28T14:08:01.245347+0000","last_fullsized":"2024-03-28T14:08:01.245347+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_clean_scrub_stamp":"2024-03-28T14:07:58.641775+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:25:39.647896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-28T14:08:00.680934+0000","last_change":"2024-03-28T14:07:59.747773+0000","last_active":"2024-03-28T14:08:00.680934+0000","last_peered":"2024-03-28T14:08:00.680934+0000","last_clean":"2024-03-28T14:08:00.680934+0000","last_became_active":"2024-03-28T14:07:59.745640+0000","last_became_peered":"2024-03-28T14:07:59.745640+0000","last_unstale":"2024-03-28T14:08:00.680934+0000","last_undegraded":"2024-03-28T14:08:00.680934+0000","last_fullsized":"2024-03-28T14:08:00.680934+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_clean_scrub_stamp":"2024-03-28T14:07:58.641775+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:10:31.257059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-28T14:08:00.834115+0000","last_change":"2024-03-28T14:07:59.722923+0000","last_active":"2024-03-28T14:08:00.834115+0000","last_peered":"2024-03-28T14:08:00.834115+0000","last_clean":"2024-03-28T14:08:00.834115+0000","last_became_active":"2024-03-28T14:07:59.711460+0000","last_became_peered":"2024-03-28T14:07:59.711460+0000","last_unstale":"2024-03-28T14:08:00.834115+0000","last_undegraded":"2024-03-28T14:08:00.834115+0000","last_fullsized":"2024-03-28T14:08:00.834115+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_clean_scrub_stamp":"2024-03-28T14:07:58.641775+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T18:51:19.214005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-28T14:08:00.680087+0000","last_change":"2024-03-28T14:07:59.742906+0000","last_active":"2024-03-28T14:08:00.680087+0000","last_peered":"2024-03-28T14:08:00.680087+0000","last_clean":"2024-03-28T14:08:00.680087+0000","last_became_active":"2024-03-28T14:07:59.734609+0000","last_became_peered":"2024-03-28T14:07:59.734609+0000","last_unstale":"2024-03-28T14:08:00.680087+0000","last_undegraded":"2024-03-28T14:08:00.680087+0000","last_fullsized":"2024-03-28T14:08:00.680087+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_clean_scrub_stamp":"2024-03-28T14:07:58.641775+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:41:27.409736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-28T14:08:00.833773+0000","last_change":"2024-03-28T14:07:59.717632+0000","last_active":"2024-03-28T14:08:00.833773+0000","last_peered":"2024-03-28T14:08:00.833773+0000","last_clean":"2024-03-28T14:08:00.833773+0000","last_became_active":"2024-03-28T14:07:59.711894+0000","last_became_peered":"2024-03-28T14:07:59.711894+0000","last_unstale":"2024-03-28T14:08:00.833773+0000","last_undegraded":"2024-03-28T14:08:00.833773+0000","last_fullsized":"2024-03-28T14:08:00.833773+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_clean_scrub_stamp":"2024-03-28T14:07:58.641775+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T16:47:04.317583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-28T14:08:00.680228+0000","last_change":"2024-03-28T14:07:59.740843+0000","last_active":"2024-03-28T14:08:00.680228+0000","last_peered":"2024-03-28T14:08:00.680228+0000","last_clean":"2024-03-28T14:08:00.680228+0000","last_became_active":"2024-03-28T14:07:59.740606+0000","last_became_peered":"2024-03-28T14:07:59.740606+0000","last_unstale":"2024-03-28T14:08:00.680228+0000","last_undegraded":"2024-03-28T14:08:00.680228+0000","last_fullsized":"2024-03-28T14:08:00.680228+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_clean_scrub_stamp":"2024-03-28T14:07:58.641775+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T14:20:33.555524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-28T14:08:00.835049+0000","last_change":"2024-03-28T14:07:59.745208+0000","last_active":"2024-03-28T14:08:00.835049+0000","last_peered":"2024-03-28T14:08:00.835049+0000","last_clean":"2024-03-28T14:08:00.835049+0000","last_became_active":"2024-03-28T14:07:59.744704+0000","last_became_peered":"2024-03-28T14:07:59.744704+0000","last_unstale":"2024-03-28T14:08:00.835049+0000","last_undegraded":"2024-03-28T14:08:00.835049+0000","last_fullsized":"2024-03-28T14:08:00.835049+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_clean_scrub_stamp":"2024-03-28T14:07:58.641775+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:53:55.059948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-28T14:08:00.703336+0000","last_change":"2024-03-28T14:07:59.744713+0000","last_active":"2024-03-28T14:08:00.703336+0000","last_peered":"2024-03-28T14:08:00.703336+0000","last_clean":"2024-03-28T14:08:00.703336+0000","last_became_active":"2024-03-28T14:07:59.744134+0000","last_became_peered":"2024-03-28T14:07:59.744134+0000","last_unstale":"2024-03-28T14:08:00.703336+0000","last_undegraded":"2024-03-28T14:08:00.703336+0000","last_fullsized":"2024-03-28T14:08:00.703336+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_clean_scrub_stamp":"2024-03-28T14:07:58.641775+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:25:39.647896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-28T14:08:00.703286+0000","last_change":"2024-03-28T14:07:59.745769+0000","last_active":"2024-03-28T14:08:00.703286+0000","last_peered":"2024-03-28T14:08:00.703286+0000","last_clean":"2024-03-28T14:08:00.703286+0000","last_became_active":"2024-03-28T14:07:59.745151+0000","last_became_peered":"2024-03-28T14:07:59.745151+0000","last_unstale":"2024-03-28T14:08:00.703286+0000","last_undegraded":"2024-03-28T14:08:00.703286+0000","last_fullsized":"2024-03-28T14:08:00.703286+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_clean_scrub_stamp":"2024-03-28T14:07:58.641775+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:37:29.402787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-28T14:08:01.244022+0000","last_change":"2024-03-28T14:07:59.702599+0000","last_active":"2024-03-28T14:08:01.244022+0000","last_peered":"2024-03-28T14:08:01.244022+0000","last_clean":"2024-03-28T14:08:01.244022+0000","last_became_active":"2024-03-28T14:07:59.701506+0000","last_became_peered":"2024-03-28T14:07:59.701506+0000","last_unstale":"2024-03-28T14:08:01.244022+0000","last_undegraded":"2024-03-28T14:08:01.244022+0000","last_fullsized":"2024-03-28T14:08:01.244022+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_clean_scrub_stamp":"2024-03-28T14:07:58.641775+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T15:40:59.112394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-28T14:08:00.680979+0000","last_change":"2024-03-28T14:07:59.704862+0000","last_active":"2024-03-28T14:08:00.680979+0000","last_peered":"2024-03-28T14:08:00.680979+0000","last_clean":"2024-03-28T14:08:00.680979+0000","last_became_active":"2024-03-28T14:07:59.704461+0000","last_became_peered":"2024-03-28T14:07:59.704461+0000","last_unstale":"2024-03-28T14:08:00.680979+0000","last_undegraded":"2024-03-28T14:08:00.680979+0000","last_fullsized":"2024-03-28T14:08:00.680979+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_clean_scrub_stamp":"2024-03-28T14:07:58.641775+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:37:29.402787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-28T14:08:01.243740+0000","last_change":"2024-03-28T14:07:59.707399+0000","last_active":"2024-03-28T14:08:01.243740+0000","last_peered":"2024-03-28T14:08:01.243740+0000","last_clean":"2024-03-28T14:08:01.243740+0000","last_became_active":"2024-03-28T14:07:59.707057+0000","last_became_peered":"2024-03-28T14:07:59.707057+0000","last_unstale":"2024-03-28T14:08:01.243740+0000","last_undegraded":"2024-03-28T14:08:01.243740+0000","last_fullsized":"2024-03-28T14:08:01.243740+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_clean_scrub_stamp":"2024-03-28T14:07:58.641775+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:54:39.259029+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-28T14:08:00.696269+0000","last_change":"2024-03-28T14:07:59.684910+0000","last_active":"2024-03-28T14:08:00.696269+0000","last_peered":"2024-03-28T14:08:00.696269+0000","last_clean":"2024-03-28T14:08:00.696269+0000","last_became_active":"2024-03-28T14:07:59.684112+0000","last_became_peered":"2024-03-28T14:07:59.684112+0000","last_unstale":"2024-03-28T14:08:00.696269+0000","last_undegraded":"2024-03-28T14:08:00.696269+0000","last_fullsized":"2024-03-28T14:08:00.696269+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_clean_scrub_stamp":"2024-03-28T14:07:58.641775+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:37:29.402787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-28T14:08:00.680001+0000","last_change":"2024-03-28T14:07:59.740138+0000","last_active":"2024-03-28T14:08:00.680001+0000","last_peered":"2024-03-28T14:08:00.680001+0000","last_clean":"2024-03-28T14:08:00.680001+0000","last_became_active":"2024-03-28T14:07:59.739320+0000","last_became_peered":"2024-03-28T14:07:59.739320+0000","last_unstale":"2024-03-28T14:08:00.680001+0000","last_undegraded":"2024-03-28T14:08:00.680001+0000","last_fullsized":"2024-03-28T14:08:00.680001+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_clean_scrub_stamp":"2024-03-28T14:07:58.641775+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:25:39.647896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-28T14:08:00.681671+0000","last_change":"2024-03-28T14:07:59.744839+0000","last_active":"2024-03-28T14:08:00.681671+0000","last_peered":"2024-03-28T14:08:00.681671+0000","last_clean":"2024-03-28T14:08:00.681671+0000","last_became_active":"2024-03-28T14:07:59.739555+0000","last_became_peered":"2024-03-28T14:07:59.739555+0000","last_unstale":"2024-03-28T14:08:00.681671+0000","last_undegraded":"2024-03-28T14:08:00.681671+0000","last_fullsized":"2024-03-28T14:08:00.681671+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_clean_scrub_stamp":"2024-03-28T14:07:58.641775+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:53:55.059948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-28T14:08:01.242371+0000","last_change":"2024-03-28T14:07:59.707740+0000","last_active":"2024-03-28T14:08:01.242371+0000","last_peered":"2024-03-28T14:08:01.242371+0000","last_clean":"2024-03-28T14:08:01.242371+0000","last_became_active":"2024-03-28T14:07:59.702876+0000","last_became_peered":"2024-03-28T14:07:59.702876+0000","last_unstale":"2024-03-28T14:08:01.242371+0000","last_undegraded":"2024-03-28T14:08:01.242371+0000","last_fullsized":"2024-03-28T14:08:01.242371+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_clean_scrub_stamp":"2024-03-28T14:07:58.641775+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T15:40:01.667745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-28T14:08:00.834771+0000","last_change":"2024-03-28T14:07:59.723973+0000","last_active":"2024-03-28T14:08:00.834771+0000","last_peered":"2024-03-28T14:08:00.834771+0000","last_clean":"2024-03-28T14:08:00.834771+0000","last_became_active":"2024-03-28T14:07:59.712162+0000","last_became_peered":"2024-03-28T14:07:59.712162+0000","last_unstale":"2024-03-28T14:08:00.834771+0000","last_undegraded":"2024-03-28T14:08:00.834771+0000","last_fullsized":"2024-03-28T14:08:00.834771+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_clean_scrub_stamp":"2024-03-28T14:07:58.641775+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:51:23.322586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-28T14:08:00.703000+0000","last_change":"2024-03-28T14:07:59.704917+0000","last_active":"2024-03-28T14:08:00.703000+0000","last_peered":"2024-03-28T14:08:00.703000+0000","last_clean":"2024-03-28T14:08:00.703000+0000","last_became_active":"2024-03-28T14:07:59.702369+0000","last_became_peered":"2024-03-28T14:07:59.702369+0000","last_unstale":"2024-03-28T14:08:00.703000+0000","last_undegraded":"2024-03-28T14:08:00.703000+0000","last_fullsized":"2024-03-28T14:08:00.703000+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_clean_scrub_stamp":"2024-03-28T14:07:58.641775+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:24:30.197596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-28T14:08:01.244077+0000","last_change":"2024-03-28T14:07:59.727121+0000","last_active":"2024-03-28T14:08:01.244077+0000","last_peered":"2024-03-28T14:08:01.244077+0000","last_clean":"2024-03-28T14:08:01.244077+0000","last_became_active":"2024-03-28T14:07:59.726783+0000","last_became_peered":"2024-03-28T14:07:59.726783+0000","last_unstale":"2024-03-28T14:08:01.244077+0000","last_undegraded":"2024-03-28T14:08:01.244077+0000","last_fullsized":"2024-03-28T14:08:01.244077+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_clean_scrub_stamp":"2024-03-28T14:07:58.641775+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:10:27.248645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-28T14:08:00.680533+0000","last_change":"2024-03-28T14:07:59.742729+0000","last_active":"2024-03-28T14:08:00.680533+0000","last_peered":"2024-03-28T14:08:00.680533+0000","last_clean":"2024-03-28T14:08:00.680533+0000","last_became_active":"2024-03-28T14:07:59.739727+0000","last_became_peered":"2024-03-28T14:07:59.739727+0000","last_unstale":"2024-03-28T14:08:00.680533+0000","last_undegraded":"2024-03-28T14:08:00.680533+0000","last_fullsized":"2024-03-28T14:08:00.680533+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_clean_scrub_stamp":"2024-03-28T14:07:58.641775+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T17:46:06.253111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-28T14:08:00.681600+0000","last_change":"2024-03-28T14:07:59.704724+0000","last_active":"2024-03-28T14:08:00.681600+0000","last_peered":"2024-03-28T14:08:00.681600+0000","last_clean":"2024-03-28T14:08:00.681600+0000","last_became_active":"2024-03-28T14:07:59.704398+0000","last_became_peered":"2024-03-28T14:07:59.704398+0000","last_unstale":"2024-03-28T14:08:00.681600+0000","last_undegraded":"2024-03-28T14:08:00.681600+0000","last_fullsized":"2024-03-28T14:08:00.681600+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_clean_scrub_stamp":"2024-03-28T14:07:58.641775+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T18:51:19.214005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-28T14:08:00.680302+0000","last_change":"2024-03-28T14:07:59.739100+0000","last_active":"2024-03-28T14:08:00.680302+0000","last_peered":"2024-03-28T14:08:00.680302+0000","last_clean":"2024-03-28T14:08:00.680302+0000","last_became_active":"2024-03-28T14:07:59.733884+0000","last_became_peered":"2024-03-28T14:07:59.733884+0000","last_unstale":"2024-03-28T14:08:00.680302+0000","last_undegraded":"2024-03-28T14:08:00.680302+0000","last_fullsized":"2024-03-28T14:08:00.680302+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_clean_scrub_stamp":"2024-03-28T14:07:58.641775+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T16:47:04.317583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-28T14:08:00.681010+0000","last_change":"2024-03-28T14:07:59.716106+0000","last_active":"2024-03-28T14:08:00.681010+0000","last_peered":"2024-03-28T14:08:00.681010+0000","last_clean":"2024-03-28T14:08:00.681010+0000","last_became_active":"2024-03-28T14:07:59.715747+0000","last_became_peered":"2024-03-28T14:07:59.715747+0000","last_unstale":"2024-03-28T14:08:00.681010+0000","last_undegraded":"2024-03-28T14:08:00.681010+0000","last_fullsized":"2024-03-28T14:08:00.681010+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_clean_scrub_stamp":"2024-03-28T14:07:58.641775+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:25:39.647896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-28T14:08:00.703528+0000","last_change":"2024-03-28T14:07:59.710509+0000","last_active":"2024-03-28T14:08:00.703528+0000","last_peered":"2024-03-28T14:08:00.703528+0000","last_clean":"2024-03-28T14:08:00.703528+0000","last_became_active":"2024-03-28T14:07:59.709767+0000","last_became_peered":"2024-03-28T14:07:59.709767+0000","last_unstale":"2024-03-28T14:08:00.703528+0000","last_undegraded":"2024-03-28T14:08:00.703528+0000","last_fullsized":"2024-03-28T14:08:00.703528+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_clean_scrub_stamp":"2024-03-28T14:07:58.641775+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:41:27.409736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-28T14:08:00.696547+0000","last_change":"2024-03-28T14:07:59.704832+0000","last_active":"2024-03-28T14:08:00.696547+0000","last_peered":"2024-03-28T14:08:00.696547+0000","last_clean":"2024-03-28T14:08:00.696547+0000","last_became_active":"2024-03-28T14:07:59.704546+0000","last_became_peered":"2024-03-28T14:07:59.704546+0000","last_unstale":"2024-03-28T14:08:00.696547+0000","last_undegraded":"2024-03-28T14:08:00.696547+0000","last_fullsized":"2024-03-28T14:08:00.696547+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_clean_scrub_stamp":"2024-03-28T14:07:58.641775+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:41:27.409736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-28T14:08:00.681706+0000","last_change":"2024-03-28T14:07:59.702155+0000","last_active":"2024-03-28T14:08:00.681706+0000","last_peered":"2024-03-28T14:08:00.681706+0000","last_clean":"2024-03-28T14:08:00.681706+0000","last_became_active":"2024-03-28T14:07:59.701799+0000","last_became_peered":"2024-03-28T14:07:59.701799+0000","last_unstale":"2024-03-28T14:08:00.681706+0000","last_undegraded":"2024-03-28T14:08:00.681706+0000","last_fullsized":"2024-03-28T14:08:00.681706+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_clean_scrub_stamp":"2024-03-28T14:07:58.641775+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:10:31.257059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-28T14:08:01.244122+0000","last_change":"2024-03-28T14:07:59.706889+0000","last_active":"2024-03-28T14:08:01.244122+0000","last_peered":"2024-03-28T14:08:01.244122+0000","last_clean":"2024-03-28T14:08:01.244122+0000","last_became_active":"2024-03-28T14:07:59.706631+0000","last_became_peered":"2024-03-28T14:07:59.706631+0000","last_unstale":"2024-03-28T14:08:01.244122+0000","last_undegraded":"2024-03-28T14:08:01.244122+0000","last_fullsized":"2024-03-28T14:08:01.244122+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_clean_scrub_stamp":"2024-03-28T14:07:58.641775+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:57:11.908191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-28T14:08:00.680602+0000","last_change":"2024-03-28T14:07:59.722801+0000","last_active":"2024-03-28T14:08:00.680602+0000","last_peered":"2024-03-28T14:08:00.680602+0000","last_clean":"2024-03-28T14:08:00.680602+0000","last_became_active":"2024-03-28T14:07:59.722132+0000","last_became_peered":"2024-03-28T14:07:59.722132+0000","last_unstale":"2024-03-28T14:08:00.680602+0000","last_undegraded":"2024-03-28T14:08:00.680602+0000","last_fullsized":"2024-03-28T14:08:00.680602+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_clean_scrub_stamp":"2024-03-28T14:07:58.641775+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T18:51:19.214005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-28T14:08:01.242238+0000","last_change":"2024-03-28T14:07:59.740448+0000","last_active":"2024-03-28T14:08:01.242238+0000","last_peered":"2024-03-28T14:08:01.242238+0000","last_clean":"2024-03-28T14:08:01.242238+0000","last_became_active":"2024-03-28T14:07:59.740121+0000","last_became_peered":"2024-03-28T14:07:59.740121+0000","last_unstale":"2024-03-28T14:08:01.242238+0000","last_undegraded":"2024-03-28T14:08:01.242238+0000","last_fullsized":"2024-03-28T14:08:01.242238+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_clean_scrub_stamp":"2024-03-28T14:07:58.641775+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T15:40:59.112394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-28T14:08:00.702471+0000","last_change":"2024-03-28T14:07:59.704155+0000","last_active":"2024-03-28T14:08:00.702471+0000","last_peered":"2024-03-28T14:08:00.702471+0000","last_clean":"2024-03-28T14:08:00.702471+0000","last_became_active":"2024-03-28T14:07:59.702173+0000","last_became_peered":"2024-03-28T14:07:59.702173+0000","last_unstale":"2024-03-28T14:08:00.702471+0000","last_undegraded":"2024-03-28T14:08:00.702471+0000","last_fullsized":"2024-03-28T14:08:00.702471+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_clean_scrub_stamp":"2024-03-28T14:07:58.641775+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T14:20:33.555524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-28T14:08:01.243508+0000","last_change":"2024-03-28T14:07:59.682929+0000","last_active":"2024-03-28T14:08:01.243508+0000","last_peered":"2024-03-28T14:08:01.243508+0000","last_clean":"2024-03-28T14:08:01.243508+0000","last_became_active":"2024-03-28T14:07:59.682238+0000","last_became_peered":"2024-03-28T14:07:59.682238+0000","last_unstale":"2024-03-28T14:08:01.243508+0000","last_undegraded":"2024-03-28T14:08:01.243508+0000","last_fullsized":"2024-03-28T14:08:01.243508+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_clean_scrub_stamp":"2024-03-28T14:07:58.641775+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T17:35:01.163718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"20'1","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-28T14:08:00.681702+0000","last_change":"2024-03-28T14:07:58.672879+0000","last_active":"2024-03-28T14:08:00.681702+0000","last_peered":"2024-03-28T14:08:00.681702+0000","last_clean":"2024-03-28T14:08:00.681702+0000","last_became_active":"2024-03-28T14:07:58.672646+0000","last_became_peered":"2024-03-28T14:07:58.672646+0000","last_unstale":"2024-03-28T14:08:00.681702+0000","last_undegraded":"2024-03-28T14:08:00.681702+0000","last_fullsized":"2024-03-28T14:08:00.681702+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:07:57.631994+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:07:57.631994+0000","last_clean_scrub_stamp":"2024-03-28T14:07:57.631994+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:35:03.722392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-28T14:08:01.244389+0000","last_change":"2024-03-28T14:07:59.715972+0000","last_active":"2024-03-28T14:08:01.244389+0000","last_peered":"2024-03-28T14:08:01.244389+0000","last_clean":"2024-03-28T14:08:01.244389+0000","last_became_active":"2024-03-28T14:07:59.715621+0000","last_became_peered":"2024-03-28T14:07:59.715621+0000","last_unstale":"2024-03-28T14:08:01.244389+0000","last_undegraded":"2024-03-28T14:08:01.244389+0000","last_fullsized":"2024-03-28T14:08:01.244389+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_clean_scrub_stamp":"2024-03-28T14:07:58.641775+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:24:30.197596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1a","version":"20'1","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-28T14:08:00.702735+0000","last_change":"2024-03-28T14:07:58.688093+0000","last_active":"2024-03-28T14:08:00.702735+0000","last_peered":"2024-03-28T14:08:00.702735+0000","last_clean":"2024-03-28T14:08:00.702735+0000","last_became_active":"2024-03-28T14:07:58.687279+0000","last_became_peered":"2024-03-28T14:07:58.687279+0000","last_unstale":"2024-03-28T14:08:00.702735+0000","last_undegraded":"2024-03-28T14:08:00.702735+0000","last_fullsized":"2024-03-28T14:08:00.702735+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:07:57.631994+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:07:57.631994+0000","last_clean_scrub_stamp":"2024-03-28T14:07:57.631994+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T18:00:55.131543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-28T14:08:00.680142+0000","last_change":"2024-03-28T14:07:58.678668+0000","last_active":"2024-03-28T14:08:00.680142+0000","last_peered":"2024-03-28T14:08:00.680142+0000","last_clean":"2024-03-28T14:08:00.680142+0000","last_became_active":"2024-03-28T14:07:58.678292+0000","last_became_peered":"2024-03-28T14:07:58.678292+0000","last_unstale":"2024-03-28T14:08:00.680142+0000","last_undegraded":"2024-03-28T14:08:00.680142+0000","last_fullsized":"2024-03-28T14:08:00.680142+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:07:57.631994+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:07:57.631994+0000","last_clean_scrub_stamp":"2024-03-28T14:07:57.631994+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T18:00:55.131543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-28T14:08:00.680364+0000","last_change":"2024-03-28T14:07:59.703432+0000","last_active":"2024-03-28T14:08:00.680364+0000","last_peered":"2024-03-28T14:08:00.680364+0000","last_clean":"2024-03-28T14:08:00.680364+0000","last_became_active":"2024-03-28T14:07:59.702896+0000","last_became_peered":"2024-03-28T14:07:59.702896+0000","last_unstale":"2024-03-28T14:08:00.680364+0000","last_undegraded":"2024-03-28T14:08:00.680364+0000","last_fullsized":"2024-03-28T14:08:00.680364+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_clean_scrub_stamp":"2024-03-28T14:07:58.641775+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:10:27.248645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-28T14:08:00.680262+0000","last_change":"2024-03-28T14:07:58.703631+0000","last_active":"2024-03-28T14:08:00.680262+0000","last_peered":"2024-03-28T14:08:00.680262+0000","last_clean":"2024-03-28T14:08:00.680262+0000","last_became_active":"2024-03-28T14:07:58.702897+0000","last_became_peered":"2024-03-28T14:07:58.702897+0000","last_unstale":"2024-03-28T14:08:00.680262+0000","last_undegraded":"2024-03-28T14:08:00.680262+0000","last_fullsized":"2024-03-28T14:08:00.680262+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:07:57.631994+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:07:57.631994+0000","last_clean_scrub_stamp":"2024-03-28T14:07:57.631994+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:35:03.722392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-28T14:08:00.696158+0000","last_change":"2024-03-28T14:07:59.706310+0000","last_active":"2024-03-28T14:08:00.696158+0000","last_peered":"2024-03-28T14:08:00.696158+0000","last_clean":"2024-03-28T14:08:00.696158+0000","last_became_active":"2024-03-28T14:07:59.705967+0000","last_became_peered":"2024-03-28T14:07:59.705967+0000","last_unstale":"2024-03-28T14:08:00.696158+0000","last_undegraded":"2024-03-28T14:08:00.696158+0000","last_fullsized":"2024-03-28T14:08:00.696158+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_clean_scrub_stamp":"2024-03-28T14:07:58.641775+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:24:30.197596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-28T14:08:00.703408+0000","last_change":"2024-03-28T14:07:58.702042+0000","last_active":"2024-03-28T14:08:00.703408+0000","last_peered":"2024-03-28T14:08:00.703408+0000","last_clean":"2024-03-28T14:08:00.703408+0000","last_became_active":"2024-03-28T14:07:58.701395+0000","last_became_peered":"2024-03-28T14:07:58.701395+0000","last_unstale":"2024-03-28T14:08:00.703408+0000","last_undegraded":"2024-03-28T14:08:00.703408+0000","last_fullsized":"2024-03-28T14:08:00.703408+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:07:57.631994+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:07:57.631994+0000","last_clean_scrub_stamp":"2024-03-28T14:07:57.631994+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T17:35:00.153938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-28T14:08:00.681271+0000","last_change":"2024-03-28T14:07:59.740904+0000","last_active":"2024-03-28T14:08:00.681271+0000","last_peered":"2024-03-28T14:08:00.681271+0000","last_clean":"2024-03-28T14:08:00.681271+0000","last_became_active":"2024-03-28T14:07:59.740209+0000","last_became_peered":"2024-03-28T14:07:59.740209+0000","last_unstale":"2024-03-28T14:08:00.681271+0000","last_undegraded":"2024-03-28T14:08:00.681271+0000","last_fullsized":"2024-03-28T14:08:00.681271+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_clean_scrub_stamp":"2024-03-28T14:07:58.641775+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:02:48.017536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-28T14:08:00.681223+0000","last_change":"2024-03-28T14:07:58.699895+0000","last_active":"2024-03-28T14:08:00.681223+0000","last_peered":"2024-03-28T14:08:00.681223+0000","last_clean":"2024-03-28T14:08:00.681223+0000","last_became_active":"2024-03-28T14:07:58.698823+0000","last_became_peered":"2024-03-28T14:07:58.698823+0000","last_unstale":"2024-03-28T14:08:00.681223+0000","last_undegraded":"2024-03-28T14:08:00.681223+0000","last_fullsized":"2024-03-28T14:08:00.681223+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:07:57.631994+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:07:57.631994+0000","last_clean_scrub_stamp":"2024-03-28T14:07:57.631994+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:09:38.264121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-28T14:08:01.245669+0000","last_change":"2024-03-28T14:07:59.744137+0000","last_active":"2024-03-28T14:08:01.245669+0000","last_peered":"2024-03-28T14:08:01.245669+0000","last_clean":"2024-03-28T14:08:01.245669+0000","last_became_active":"2024-03-28T14:07:59.743854+0000","last_became_peered":"2024-03-28T14:07:59.743854+0000","last_unstale":"2024-03-28T14:08:01.245669+0000","last_undegraded":"2024-03-28T14:08:01.245669+0000","last_fullsized":"2024-03-28T14:08:01.245669+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_clean_scrub_stamp":"2024-03-28T14:07:58.641775+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:41:27.409736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"20'3","reported_seq":17,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-28T14:08:00.696618+0000","last_change":"2024-03-28T14:07:58.703242+0000","last_active":"2024-03-28T14:08:00.696618+0000","last_peered":"2024-03-28T14:08:00.696618+0000","last_clean":"2024-03-28T14:08:00.696618+0000","last_became_active":"2024-03-28T14:07:58.702970+0000","last_became_peered":"2024-03-28T14:07:58.702970+0000","last_unstale":"2024-03-28T14:08:00.696618+0000","last_undegraded":"2024-03-28T14:08:00.696618+0000","last_fullsized":"2024-03-28T14:08:00.696618+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:07:57.631994+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:07:57.631994+0000","last_clean_scrub_stamp":"2024-03-28T14:07:57.631994+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T15:40:58.102613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-28T14:08:01.244046+0000","last_change":"2024-03-28T14:07:59.708551+0000","last_active":"2024-03-28T14:08:01.244046+0000","last_peered":"2024-03-28T14:08:01.244046+0000","last_clean":"2024-03-28T14:08:01.244046+0000","last_became_active":"2024-03-28T14:07:59.706665+0000","last_became_peered":"2024-03-28T14:07:59.706665+0000","last_unstale":"2024-03-28T14:08:01.244046+0000","last_undegraded":"2024-03-28T14:08:01.244046+0000","last_fullsized":"2024-03-28T14:08:01.244046+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_clean_scrub_stamp":"2024-03-28T14:07:58.641775+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T15:40:01.667745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.14","version":"20'6","reported_seq":20,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-28T14:08:00.702870+0000","last_change":"2024-03-28T14:07:58.680343+0000","last_active":"2024-03-28T14:08:00.702870+0000","last_peered":"2024-03-28T14:08:00.702870+0000","last_clean":"2024-03-28T14:08:00.702870+0000","last_became_active":"2024-03-28T14:07:58.679621+0000","last_became_peered":"2024-03-28T14:07:58.679621+0000","last_unstale":"2024-03-28T14:08:00.702870+0000","last_undegraded":"2024-03-28T14:08:00.702870+0000","last_fullsized":"2024-03-28T14:08:00.702870+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:07:57.631994+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:07:57.631994+0000","last_clean_scrub_stamp":"2024-03-28T14:07:57.631994+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:30:19.088624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-28T14:08:00.834351+0000","last_change":"2024-03-28T14:07:59.714064+0000","last_active":"2024-03-28T14:08:00.834351+0000","last_peered":"2024-03-28T14:08:00.834351+0000","last_clean":"2024-03-28T14:08:00.834351+0000","last_became_active":"2024-03-28T14:07:59.710619+0000","last_became_peered":"2024-03-28T14:07:59.710619+0000","last_unstale":"2024-03-28T14:08:00.834351+0000","last_undegraded":"2024-03-28T14:08:00.834351+0000","last_fullsized":"2024-03-28T14:08:00.834351+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_clean_scrub_stamp":"2024-03-28T14:07:58.641775+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:34:47.572488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"20'2","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-28T14:08:01.244515+0000","last_change":"2024-03-28T14:07:58.684715+0000","last_active":"2024-03-28T14:08:01.244515+0000","last_peered":"2024-03-28T14:08:01.244515+0000","last_clean":"2024-03-28T14:08:01.244515+0000","last_became_active":"2024-03-28T14:07:58.684314+0000","last_became_peered":"2024-03-28T14:07:58.684314+0000","last_unstale":"2024-03-28T14:08:01.244515+0000","last_undegraded":"2024-03-28T14:08:01.244515+0000","last_fullsized":"2024-03-28T14:08:01.244515+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:07:57.631994+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:07:57.631994+0000","last_clean_scrub_stamp":"2024-03-28T14:07:57.631994+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T17:34:09.030115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-28T14:08:01.243877+0000","last_change":"2024-03-28T14:07:59.730071+0000","last_active":"2024-03-28T14:08:01.243877+0000","last_peered":"2024-03-28T14:08:01.243877+0000","last_clean":"2024-03-28T14:08:01.243877+0000","last_became_active":"2024-03-28T14:07:59.729794+0000","last_became_peered":"2024-03-28T14:07:59.729794+0000","last_unstale":"2024-03-28T14:08:01.243877+0000","last_undegraded":"2024-03-28T14:08:01.243877+0000","last_fullsized":"2024-03-28T14:08:01.243877+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_clean_scrub_stamp":"2024-03-28T14:07:58.641775+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:38:43.102577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-28T14:08:00.680825+0000","last_change":"2024-03-28T14:07:58.704245+0000","last_active":"2024-03-28T14:08:00.680825+0000","last_peered":"2024-03-28T14:08:00.680825+0000","last_clean":"2024-03-28T14:08:00.680825+0000","last_became_active":"2024-03-28T14:07:58.703603+0000","last_became_peered":"2024-03-28T14:07:58.703603+0000","last_unstale":"2024-03-28T14:08:00.680825+0000","last_undegraded":"2024-03-28T14:08:00.680825+0000","last_fullsized":"2024-03-28T14:08:00.680825+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:07:57.631994+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:07:57.631994+0000","last_clean_scrub_stamp":"2024-03-28T14:07:57.631994+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T17:35:00.153938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-28T14:08:00.834163+0000","last_change":"2024-03-28T14:07:59.718822+0000","last_active":"2024-03-28T14:08:00.834163+0000","last_peered":"2024-03-28T14:08:00.834163+0000","last_clean":"2024-03-28T14:08:00.834163+0000","last_became_active":"2024-03-28T14:07:59.718058+0000","last_became_peered":"2024-03-28T14:07:59.718058+0000","last_unstale":"2024-03-28T14:08:00.834163+0000","last_undegraded":"2024-03-28T14:08:00.834163+0000","last_fullsized":"2024-03-28T14:08:00.834163+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_clean_scrub_stamp":"2024-03-28T14:07:58.641775+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:10:31.257059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-28T14:08:00.702420+0000","last_change":"2024-03-28T14:07:58.689870+0000","last_active":"2024-03-28T14:08:00.702420+0000","last_peered":"2024-03-28T14:08:00.702420+0000","last_clean":"2024-03-28T14:08:00.702420+0000","last_became_active":"2024-03-28T14:07:58.689121+0000","last_became_peered":"2024-03-28T14:07:58.689121+0000","last_unstale":"2024-03-28T14:08:00.702420+0000","last_undegraded":"2024-03-28T14:08:00.702420+0000","last_fullsized":"2024-03-28T14:08:00.702420+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:07:57.631994+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:07:57.631994+0000","last_clean_scrub_stamp":"2024-03-28T14:07:57.631994+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:35:03.722392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-28T14:08:00.702531+0000","last_change":"2024-03-28T14:07:59.712485+0000","last_active":"2024-03-28T14:08:00.702531+0000","last_peered":"2024-03-28T14:08:00.702531+0000","last_clean":"2024-03-28T14:08:00.702531+0000","last_became_active":"2024-03-28T14:07:59.704056+0000","last_became_peered":"2024-03-28T14:07:59.704056+0000","last_unstale":"2024-03-28T14:08:00.702531+0000","last_undegraded":"2024-03-28T14:08:00.702531+0000","last_fullsized":"2024-03-28T14:08:00.702531+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_clean_scrub_stamp":"2024-03-28T14:07:58.641775+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:10:27.248645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"20'8","reported_seq":22,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-28T14:08:01.242042+0000","last_change":"2024-03-28T14:07:58.667252+0000","last_active":"2024-03-28T14:08:01.242042+0000","last_peered":"2024-03-28T14:08:01.242042+0000","last_clean":"2024-03-28T14:08:01.242042+0000","last_became_active":"2024-03-28T14:07:58.666837+0000","last_became_peered":"2024-03-28T14:07:58.666837+0000","last_unstale":"2024-03-28T14:08:01.242042+0000","last_undegraded":"2024-03-28T14:08:01.242042+0000","last_fullsized":"2024-03-28T14:08:01.242042+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:07:57.631994+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:07:57.631994+0000","last_clean_scrub_stamp":"2024-03-28T14:07:57.631994+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:30:19.088624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-28T14:08:00.833905+0000","last_change":"2024-03-28T14:07:59.719433+0000","last_active":"2024-03-28T14:08:00.833905+0000","last_peered":"2024-03-28T14:08:00.833905+0000","last_clean":"2024-03-28T14:08:00.833905+0000","last_became_active":"2024-03-28T14:07:59.718644+0000","last_became_peered":"2024-03-28T14:07:59.718644+0000","last_unstale":"2024-03-28T14:08:00.833905+0000","last_undegraded":"2024-03-28T14:08:00.833905+0000","last_fullsized":"2024-03-28T14:08:00.833905+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_clean_scrub_stamp":"2024-03-28T14:07:58.641775+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T17:46:06.253111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"20'1","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-28T14:08:01.243993+0000","last_change":"2024-03-28T14:07:58.674598+0000","last_active":"2024-03-28T14:08:01.243993+0000","last_peered":"2024-03-28T14:08:01.243993+0000","last_clean":"2024-03-28T14:08:01.243993+0000","last_became_active":"2024-03-28T14:07:58.673894+0000","last_became_peered":"2024-03-28T14:07:58.673894+0000","last_unstale":"2024-03-28T14:08:01.243993+0000","last_undegraded":"2024-03-28T14:08:01.243993+0000","last_fullsized":"2024-03-28T14:08:01.243993+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:07:57.631994+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:07:57.631994+0000","last_clean_scrub_stamp":"2024-03-28T14:07:57.631994+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:32:35.921987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-28T14:08:00.834711+0000","last_change":"2024-03-28T14:07:59.725643+0000","last_active":"2024-03-28T14:08:00.834711+0000","last_peered":"2024-03-28T14:08:00.834711+0000","last_clean":"2024-03-28T14:08:00.834711+0000","last_became_active":"2024-03-28T14:07:59.725019+0000","last_became_peered":"2024-03-28T14:07:59.725019+0000","last_unstale":"2024-03-28T14:08:00.834711+0000","last_undegraded":"2024-03-28T14:08:00.834711+0000","last_fullsized":"2024-03-28T14:08:00.834711+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_clean_scrub_stamp":"2024-03-28T14:07:58.641775+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:02:48.017536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"20'2","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-28T14:08:01.242483+0000","last_change":"2024-03-28T14:07:58.663444+0000","last_active":"2024-03-28T14:08:01.242483+0000","last_peered":"2024-03-28T14:08:01.242483+0000","last_clean":"2024-03-28T14:08:01.242483+0000","last_became_active":"2024-03-28T14:07:58.663001+0000","last_became_peered":"2024-03-28T14:07:58.663001+0000","last_unstale":"2024-03-28T14:08:01.242483+0000","last_undegraded":"2024-03-28T14:08:01.242483+0000","last_fullsized":"2024-03-28T14:08:01.242483+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:07:57.631994+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:07:57.631994+0000","last_clean_scrub_stamp":"2024-03-28T14:07:57.631994+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:35:03.722392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-28T14:08:00.835294+0000","last_change":"2024-03-28T14:07:59.731950+0000","last_active":"2024-03-28T14:08:00.835294+0000","last_peered":"2024-03-28T14:08:00.835294+0000","last_clean":"2024-03-28T14:08:00.835294+0000","last_became_active":"2024-03-28T14:07:59.731208+0000","last_became_peered":"2024-03-28T14:07:59.731208+0000","last_unstale":"2024-03-28T14:08:00.835294+0000","last_undegraded":"2024-03-28T14:08:00.835294+0000","last_fullsized":"2024-03-28T14:08:00.835294+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_clean_scrub_stamp":"2024-03-28T14:07:58.641775+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T14:17:34.720686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"20'1","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-28T14:08:00.696647+0000","last_change":"2024-03-28T14:07:58.684273+0000","last_active":"2024-03-28T14:08:00.696647+0000","last_peered":"2024-03-28T14:08:00.696647+0000","last_clean":"2024-03-28T14:08:00.696647+0000","last_became_active":"2024-03-28T14:07:58.683886+0000","last_became_peered":"2024-03-28T14:07:58.683886+0000","last_unstale":"2024-03-28T14:08:00.696647+0000","last_undegraded":"2024-03-28T14:08:00.696647+0000","last_fullsized":"2024-03-28T14:08:00.696647+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:07:57.631994+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:07:57.631994+0000","last_clean_scrub_stamp":"2024-03-28T14:07:57.631994+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T15:40:00.657964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-28T14:08:00.681505+0000","last_change":"2024-03-28T14:07:59.739696+0000","last_active":"2024-03-28T14:08:00.681505+0000","last_peered":"2024-03-28T14:08:00.681505+0000","last_clean":"2024-03-28T14:08:00.681505+0000","last_became_active":"2024-03-28T14:07:59.734736+0000","last_became_peered":"2024-03-28T14:07:59.734736+0000","last_unstale":"2024-03-28T14:08:00.681505+0000","last_undegraded":"2024-03-28T14:08:00.681505+0000","last_fullsized":"2024-03-28T14:08:00.681505+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_clean_scrub_stamp":"2024-03-28T14:07:58.641775+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:51:23.322586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-28T14:08:01.242313+0000","last_change":"2024-03-28T14:07:58.704650+0000","last_active":"2024-03-28T14:08:01.242313+0000","last_peered":"2024-03-28T14:08:01.242313+0000","last_clean":"2024-03-28T14:08:01.242313+0000","last_became_active":"2024-03-28T14:07:58.704305+0000","last_became_peered":"2024-03-28T14:07:58.704305+0000","last_unstale":"2024-03-28T14:08:01.242313+0000","last_undegraded":"2024-03-28T14:08:01.242313+0000","last_fullsized":"2024-03-28T14:08:01.242313+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:07:57.631994+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:07:57.631994+0000","last_clean_scrub_stamp":"2024-03-28T14:07:57.631994+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T17:35:00.153938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-28T14:08:00.834345+0000","last_change":"2024-03-28T14:07:59.745242+0000","last_active":"2024-03-28T14:08:00.834345+0000","last_peered":"2024-03-28T14:08:00.834345+0000","last_clean":"2024-03-28T14:08:00.834345+0000","last_became_active":"2024-03-28T14:07:59.744734+0000","last_became_peered":"2024-03-28T14:07:59.744734+0000","last_unstale":"2024-03-28T14:08:00.834345+0000","last_undegraded":"2024-03-28T14:08:00.834345+0000","last_fullsized":"2024-03-28T14:08:00.834345+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_clean_scrub_stamp":"2024-03-28T14:07:58.641775+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:49:58.161511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":170,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-28T14:08:00.833744+0000","last_change":"2024-03-28T14:07:49.234448+0000","last_active":"2024-03-28T14:08:00.833744+0000","last_peered":"2024-03-28T14:08:00.833744+0000","last_clean":"2024-03-28T14:08:00.833744+0000","last_became_active":"2024-03-28T14:07:49.233895+0000","last_became_peered":"2024-03-28T14:07:49.233895+0000","last_unstale":"2024-03-28T14:08:00.833744+0000","last_undegraded":"2024-03-28T14:08:00.833744+0000","last_fullsized":"2024-03-28T14:08:00.833744+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:07:48.069157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:07:48.069157+0000","last_clean_scrub_stamp":"2024-03-28T14:07:48.069157+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:11:39.977557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-28T14:08:00.680569+0000","last_change":"2024-03-28T14:07:58.699326+0000","last_active":"2024-03-28T14:08:00.680569+0000","last_peered":"2024-03-28T14:08:00.680569+0000","last_clean":"2024-03-28T14:08:00.680569+0000","last_became_active":"2024-03-28T14:07:58.698258+0000","last_became_peered":"2024-03-28T14:07:58.698258+0000","last_unstale":"2024-03-28T14:08:00.680569+0000","last_undegraded":"2024-03-28T14:08:00.680569+0000","last_fullsized":"2024-03-28T14:08:00.680569+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:07:57.631994+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:07:57.631994+0000","last_clean_scrub_stamp":"2024-03-28T14:07:57.631994+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T18:00:55.131543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-28T14:08:00.681381+0000","last_change":"2024-03-28T14:07:59.744015+0000","last_active":"2024-03-28T14:08:00.681381+0000","last_peered":"2024-03-28T14:08:00.681381+0000","last_clean":"2024-03-28T14:08:00.681381+0000","last_became_active":"2024-03-28T14:07:59.743723+0000","last_became_peered":"2024-03-28T14:07:59.743723+0000","last_unstale":"2024-03-28T14:08:00.681381+0000","last_undegraded":"2024-03-28T14:08:00.681381+0000","last_fullsized":"2024-03-28T14:08:00.681381+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_clean_scrub_stamp":"2024-03-28T14:07:58.641775+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T16:47:04.317583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-28T14:08:00.833592+0000","last_change":"2024-03-28T14:07:58.688278+0000","last_active":"2024-03-28T14:08:00.833592+0000","last_peered":"2024-03-28T14:08:00.833592+0000","last_clean":"2024-03-28T14:08:00.833592+0000","last_became_active":"2024-03-28T14:07:58.687442+0000","last_became_peered":"2024-03-28T14:07:58.687442+0000","last_unstale":"2024-03-28T14:08:00.833592+0000","last_undegraded":"2024-03-28T14:08:00.833592+0000","last_fullsized":"2024-03-28T14:08:00.833592+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:07:57.631994+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:07:57.631994+0000","last_clean_scrub_stamp":"2024-03-28T14:07:57.631994+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T17:35:00.153938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-28T14:08:01.243711+0000","last_change":"2024-03-28T14:07:59.715990+0000","last_active":"2024-03-28T14:08:01.243711+0000","last_peered":"2024-03-28T14:08:01.243711+0000","last_clean":"2024-03-28T14:08:01.243711+0000","last_became_active":"2024-03-28T14:07:59.715627+0000","last_became_peered":"2024-03-28T14:07:59.715627+0000","last_unstale":"2024-03-28T14:08:01.243711+0000","last_undegraded":"2024-03-28T14:08:01.243711+0000","last_fullsized":"2024-03-28T14:08:01.243711+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_clean_scrub_stamp":"2024-03-28T14:07:58.641775+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T18:46:53.473825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-28T14:08:01.242294+0000","last_change":"2024-03-28T14:07:58.663475+0000","last_active":"2024-03-28T14:08:01.242294+0000","last_peered":"2024-03-28T14:08:01.242294+0000","last_clean":"2024-03-28T14:08:01.242294+0000","last_became_active":"2024-03-28T14:07:58.662967+0000","last_became_peered":"2024-03-28T14:07:58.662967+0000","last_unstale":"2024-03-28T14:08:01.242294+0000","last_undegraded":"2024-03-28T14:08:01.242294+0000","last_fullsized":"2024-03-28T14:08:01.242294+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:07:57.631994+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:07:57.631994+0000","last_clean_scrub_stamp":"2024-03-28T14:07:57.631994+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T18:00:55.131543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-28T14:08:00.696122+0000","last_change":"2024-03-28T14:07:59.693645+0000","last_active":"2024-03-28T14:08:00.696122+0000","last_peered":"2024-03-28T14:08:00.696122+0000","last_clean":"2024-03-28T14:08:00.696122+0000","last_became_active":"2024-03-28T14:07:59.693346+0000","last_became_peered":"2024-03-28T14:07:59.693346+0000","last_unstale":"2024-03-28T14:08:00.696122+0000","last_undegraded":"2024-03-28T14:08:00.696122+0000","last_fullsized":"2024-03-28T14:08:00.696122+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_clean_scrub_stamp":"2024-03-28T14:07:58.641775+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:57:11.908191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-28T14:08:00.680856+0000","last_change":"2024-03-28T14:07:58.702868+0000","last_active":"2024-03-28T14:08:00.680856+0000","last_peered":"2024-03-28T14:08:00.680856+0000","last_clean":"2024-03-28T14:08:00.680856+0000","last_became_active":"2024-03-28T14:07:58.700099+0000","last_became_peered":"2024-03-28T14:07:58.700099+0000","last_unstale":"2024-03-28T14:08:00.680856+0000","last_undegraded":"2024-03-28T14:08:00.680856+0000","last_fullsized":"2024-03-28T14:08:00.680856+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:07:57.631994+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:07:57.631994+0000","last_clean_scrub_stamp":"2024-03-28T14:07:57.631994+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:30:19.088624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-28T14:08:01.243699+0000","last_change":"2024-03-28T14:07:59.707739+0000","last_active":"2024-03-28T14:08:01.243699+0000","last_peered":"2024-03-28T14:08:01.243699+0000","last_clean":"2024-03-28T14:08:01.243699+0000","last_became_active":"2024-03-28T14:07:59.705252+0000","last_became_peered":"2024-03-28T14:07:59.705252+0000","last_unstale":"2024-03-28T14:08:01.243699+0000","last_undegraded":"2024-03-28T14:08:01.243699+0000","last_fullsized":"2024-03-28T14:08:01.243699+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_clean_scrub_stamp":"2024-03-28T14:07:58.641775+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:09:39.273902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-28T14:08:00.696496+0000","last_change":"2024-03-28T14:07:58.669940+0000","last_active":"2024-03-28T14:08:00.696496+0000","last_peered":"2024-03-28T14:08:00.696496+0000","last_clean":"2024-03-28T14:08:00.696496+0000","last_became_active":"2024-03-28T14:07:58.669568+0000","last_became_peered":"2024-03-28T14:07:58.669568+0000","last_unstale":"2024-03-28T14:08:00.696496+0000","last_undegraded":"2024-03-28T14:08:00.696496+0000","last_fullsized":"2024-03-28T14:08:00.696496+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:07:57.631994+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:07:57.631994+0000","last_clean_scrub_stamp":"2024-03-28T14:07:57.631994+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:09:38.264121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-28T14:08:00.681619+0000","last_change":"2024-03-28T14:07:59.742899+0000","last_active":"2024-03-28T14:08:00.681619+0000","last_peered":"2024-03-28T14:08:00.681619+0000","last_clean":"2024-03-28T14:08:00.681619+0000","last_became_active":"2024-03-28T14:07:59.740075+0000","last_became_peered":"2024-03-28T14:07:59.740075+0000","last_unstale":"2024-03-28T14:08:00.681619+0000","last_undegraded":"2024-03-28T14:08:00.681619+0000","last_fullsized":"2024-03-28T14:08:00.681619+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_clean_scrub_stamp":"2024-03-28T14:07:58.641775+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:59:25.862480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-28T14:08:00.834925+0000","last_change":"2024-03-28T14:07:58.687874+0000","last_active":"2024-03-28T14:08:00.834925+0000","last_peered":"2024-03-28T14:08:00.834925+0000","last_clean":"2024-03-28T14:08:00.834925+0000","last_became_active":"2024-03-28T14:07:58.686656+0000","last_became_peered":"2024-03-28T14:07:58.686656+0000","last_unstale":"2024-03-28T14:08:00.834925+0000","last_undegraded":"2024-03-28T14:08:00.834925+0000","last_fullsized":"2024-03-28T14:08:00.834925+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:07:57.631994+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:07:57.631994+0000","last_clean_scrub_stamp":"2024-03-28T14:07:57.631994+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T15:40:00.657964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-28T14:08:00.696528+0000","last_change":"2024-03-28T14:07:59.700465+0000","last_active":"2024-03-28T14:08:00.696528+0000","last_peered":"2024-03-28T14:08:00.696528+0000","last_clean":"2024-03-28T14:08:00.696528+0000","last_became_active":"2024-03-28T14:07:59.699705+0000","last_became_peered":"2024-03-28T14:07:59.699705+0000","last_unstale":"2024-03-28T14:08:00.696528+0000","last_undegraded":"2024-03-28T14:08:00.696528+0000","last_fullsized":"2024-03-28T14:08:00.696528+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_clean_scrub_stamp":"2024-03-28T14:07:58.641775+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:25:39.647896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"21'1","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-28T14:08:00.700925+0000","last_change":"2024-03-28T14:07:58.681340+0000","last_active":"2024-03-28T14:08:00.700925+0000","last_peered":"2024-03-28T14:08:00.700925+0000","last_clean":"2024-03-28T14:08:00.700925+0000","last_became_active":"2024-03-28T14:07:58.681006+0000","last_became_peered":"2024-03-28T14:07:58.681006+0000","last_unstale":"2024-03-28T14:08:00.700925+0000","last_undegraded":"2024-03-28T14:08:00.700925+0000","last_fullsized":"2024-03-28T14:08:00.700925+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:07:57.631994+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:07:57.631994+0000","last_clean_scrub_stamp":"2024-03-28T14:07:57.631994+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:04:40.185372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-28T14:08:00.681402+0000","last_change":"2024-03-28T14:07:59.727008+0000","last_active":"2024-03-28T14:08:00.681402+0000","last_peered":"2024-03-28T14:08:00.681402+0000","last_clean":"2024-03-28T14:08:00.681402+0000","last_became_active":"2024-03-28T14:07:59.726702+0000","last_became_peered":"2024-03-28T14:07:59.726702+0000","last_unstale":"2024-03-28T14:08:00.681402+0000","last_undegraded":"2024-03-28T14:08:00.681402+0000","last_fullsized":"2024-03-28T14:08:00.681402+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_clean_scrub_stamp":"2024-03-28T14:07:58.641775+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T17:46:06.253111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"20'2","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-28T14:08:00.703440+0000","last_change":"2024-03-28T14:07:58.685500+0000","last_active":"2024-03-28T14:08:00.703440+0000","last_peered":"2024-03-28T14:08:00.703440+0000","last_clean":"2024-03-28T14:08:00.703440+0000","last_became_active":"2024-03-28T14:07:58.684781+0000","last_became_peered":"2024-03-28T14:07:58.684781+0000","last_unstale":"2024-03-28T14:08:00.703440+0000","last_undegraded":"2024-03-28T14:08:00.703440+0000","last_fullsized":"2024-03-28T14:08:00.703440+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:07:57.631994+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:07:57.631994+0000","last_clean_scrub_stamp":"2024-03-28T14:07:57.631994+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:09:38.264121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-28T14:08:01.244918+0000","last_change":"2024-03-28T14:07:59.707407+0000","last_active":"2024-03-28T14:08:01.244918+0000","last_peered":"2024-03-28T14:08:01.244918+0000","last_clean":"2024-03-28T14:08:01.244918+0000","last_became_active":"2024-03-28T14:07:59.707103+0000","last_became_peered":"2024-03-28T14:07:59.707103+0000","last_unstale":"2024-03-28T14:08:01.244918+0000","last_undegraded":"2024-03-28T14:08:01.244918+0000","last_fullsized":"2024-03-28T14:08:01.244918+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_clean_scrub_stamp":"2024-03-28T14:07:58.641775+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:37:29.402787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-28T14:08:00.833570+0000","last_change":"2024-03-28T14:07:58.682456+0000","last_active":"2024-03-28T14:08:00.833570+0000","last_peered":"2024-03-28T14:08:00.833570+0000","last_clean":"2024-03-28T14:08:00.833570+0000","last_became_active":"2024-03-28T14:07:58.678987+0000","last_became_peered":"2024-03-28T14:07:58.678987+0000","last_unstale":"2024-03-28T14:08:00.833570+0000","last_undegraded":"2024-03-28T14:08:00.833570+0000","last_fullsized":"2024-03-28T14:08:00.833570+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:07:57.631994+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:07:57.631994+0000","last_clean_scrub_stamp":"2024-03-28T14:07:57.631994+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:30:19.088624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-28T14:08:01.243764+0000","last_change":"2024-03-28T14:07:59.697567+0000","last_active":"2024-03-28T14:08:01.243764+0000","last_peered":"2024-03-28T14:08:01.243764+0000","last_clean":"2024-03-28T14:08:01.243764+0000","last_became_active":"2024-03-28T14:07:59.693975+0000","last_became_peered":"2024-03-28T14:07:59.693975+0000","last_unstale":"2024-03-28T14:08:01.243764+0000","last_undegraded":"2024-03-28T14:08:01.243764+0000","last_fullsized":"2024-03-28T14:08:01.243764+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_clean_scrub_stamp":"2024-03-28T14:07:58.641775+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:04:41.195153+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-28T14:08:00.696246+0000","last_change":"2024-03-28T14:07:58.680918+0000","last_active":"2024-03-28T14:08:00.696246+0000","last_peered":"2024-03-28T14:08:00.696246+0000","last_clean":"2024-03-28T14:08:00.696246+0000","last_became_active":"2024-03-28T14:07:58.680548+0000","last_became_peered":"2024-03-28T14:07:58.680548+0000","last_unstale":"2024-03-28T14:08:00.696246+0000","last_undegraded":"2024-03-28T14:08:00.696246+0000","last_fullsized":"2024-03-28T14:08:00.696246+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:07:57.631994+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:07:57.631994+0000","last_clean_scrub_stamp":"2024-03-28T14:07:57.631994+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:30:19.088624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-28T14:08:00.680594+0000","last_change":"2024-03-28T14:07:59.702172+0000","last_active":"2024-03-28T14:08:00.680594+0000","last_peered":"2024-03-28T14:08:00.680594+0000","last_clean":"2024-03-28T14:08:00.680594+0000","last_became_active":"2024-03-28T14:07:59.701709+0000","last_became_peered":"2024-03-28T14:07:59.701709+0000","last_unstale":"2024-03-28T14:08:00.680594+0000","last_undegraded":"2024-03-28T14:08:00.680594+0000","last_fullsized":"2024-03-28T14:08:00.680594+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_clean_scrub_stamp":"2024-03-28T14:07:58.641775+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:57:11.908191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-28T14:08:00.696422+0000","last_change":"2024-03-28T14:07:58.672681+0000","last_active":"2024-03-28T14:08:00.696422+0000","last_peered":"2024-03-28T14:08:00.696422+0000","last_clean":"2024-03-28T14:08:00.696422+0000","last_became_active":"2024-03-28T14:07:58.672094+0000","last_became_peered":"2024-03-28T14:07:58.672094+0000","last_unstale":"2024-03-28T14:08:00.696422+0000","last_undegraded":"2024-03-28T14:08:00.696422+0000","last_fullsized":"2024-03-28T14:08:00.696422+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:07:57.631994+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:07:57.631994+0000","last_clean_scrub_stamp":"2024-03-28T14:07:57.631994+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T17:35:00.153938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-28T14:08:00.681203+0000","last_change":"2024-03-28T14:07:59.705251+0000","last_active":"2024-03-28T14:08:00.681203+0000","last_peered":"2024-03-28T14:08:00.681203+0000","last_clean":"2024-03-28T14:08:00.681203+0000","last_became_active":"2024-03-28T14:07:59.703366+0000","last_became_peered":"2024-03-28T14:07:59.703366+0000","last_unstale":"2024-03-28T14:08:00.681203+0000","last_undegraded":"2024-03-28T14:08:00.681203+0000","last_fullsized":"2024-03-28T14:08:00.681203+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_clean_scrub_stamp":"2024-03-28T14:07:58.641775+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:41:27.409736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"20'2","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-28T14:08:00.834672+0000","last_change":"2024-03-28T14:07:58.688811+0000","last_active":"2024-03-28T14:08:00.834672+0000","last_peered":"2024-03-28T14:08:00.834672+0000","last_clean":"2024-03-28T14:08:00.834672+0000","last_became_active":"2024-03-28T14:07:58.688054+0000","last_became_peered":"2024-03-28T14:07:58.688054+0000","last_unstale":"2024-03-28T14:08:00.834672+0000","last_undegraded":"2024-03-28T14:08:00.834672+0000","last_fullsized":"2024-03-28T14:08:00.834672+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:07:57.631994+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:07:57.631994+0000","last_clean_scrub_stamp":"2024-03-28T14:07:57.631994+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:04:40.185372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-28T14:08:00.702555+0000","last_change":"2024-03-28T14:07:59.709224+0000","last_active":"2024-03-28T14:08:00.702555+0000","last_peered":"2024-03-28T14:08:00.702555+0000","last_clean":"2024-03-28T14:08:00.702555+0000","last_became_active":"2024-03-28T14:07:59.704202+0000","last_became_peered":"2024-03-28T14:07:59.704202+0000","last_unstale":"2024-03-28T14:08:00.702555+0000","last_undegraded":"2024-03-28T14:08:00.702555+0000","last_fullsized":"2024-03-28T14:08:00.702555+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_clean_scrub_stamp":"2024-03-28T14:07:58.641775+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:57:11.908191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-28T14:08:00.681258+0000","last_change":"2024-03-28T14:07:59.739995+0000","last_active":"2024-03-28T14:08:00.681258+0000","last_peered":"2024-03-28T14:08:00.681258+0000","last_clean":"2024-03-28T14:08:00.681258+0000","last_became_active":"2024-03-28T14:07:59.739369+0000","last_became_peered":"2024-03-28T14:07:59.739369+0000","last_unstale":"2024-03-28T14:08:00.681258+0000","last_undegraded":"2024-03-28T14:08:00.681258+0000","last_fullsized":"2024-03-28T14:08:00.681258+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_clean_scrub_stamp":"2024-03-28T14:07:58.641775+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:34:47.572488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"20'2","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-28T14:08:00.834148+0000","last_change":"2024-03-28T14:07:58.705499+0000","last_active":"2024-03-28T14:08:00.834148+0000","last_peered":"2024-03-28T14:08:00.834148+0000","last_clean":"2024-03-28T14:08:00.834148+0000","last_became_active":"2024-03-28T14:07:58.704848+0000","last_became_peered":"2024-03-28T14:07:58.704848+0000","last_unstale":"2024-03-28T14:08:00.834148+0000","last_undegraded":"2024-03-28T14:08:00.834148+0000","last_fullsized":"2024-03-28T14:08:00.834148+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:07:57.631994+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:07:57.631994+0000","last_clean_scrub_stamp":"2024-03-28T14:07:57.631994+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:09:38.264121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-28T14:08:00.680957+0000","last_change":"2024-03-28T14:07:59.740931+0000","last_active":"2024-03-28T14:08:00.680957+0000","last_peered":"2024-03-28T14:08:00.680957+0000","last_clean":"2024-03-28T14:08:00.680957+0000","last_became_active":"2024-03-28T14:07:59.740255+0000","last_became_peered":"2024-03-28T14:07:59.740255+0000","last_unstale":"2024-03-28T14:08:00.680957+0000","last_undegraded":"2024-03-28T14:08:00.680957+0000","last_fullsized":"2024-03-28T14:08:00.680957+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_clean_scrub_stamp":"2024-03-28T14:07:58.641775+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:49:58.161511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"20'2","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-28T14:08:00.834791+0000","last_change":"2024-03-28T14:07:58.693440+0000","last_active":"2024-03-28T14:08:00.834791+0000","last_peered":"2024-03-28T14:08:00.834791+0000","last_clean":"2024-03-28T14:08:00.834791+0000","last_became_active":"2024-03-28T14:07:58.692096+0000","last_became_peered":"2024-03-28T14:07:58.692096+0000","last_unstale":"2024-03-28T14:08:00.834791+0000","last_undegraded":"2024-03-28T14:08:00.834791+0000","last_fullsized":"2024-03-28T14:08:00.834791+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:07:57.631994+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:07:57.631994+0000","last_clean_scrub_stamp":"2024-03-28T14:07:57.631994+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T15:40:58.102613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-28T14:08:01.244001+0000","last_change":"2024-03-28T14:07:59.744398+0000","last_active":"2024-03-28T14:08:01.244001+0000","last_peered":"2024-03-28T14:08:01.244001+0000","last_clean":"2024-03-28T14:08:01.244001+0000","last_became_active":"2024-03-28T14:07:59.744127+0000","last_became_peered":"2024-03-28T14:07:59.744127+0000","last_unstale":"2024-03-28T14:08:01.244001+0000","last_undegraded":"2024-03-28T14:08:01.244001+0000","last_fullsized":"2024-03-28T14:08:01.244001+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_clean_scrub_stamp":"2024-03-28T14:07:58.641775+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T14:20:33.555524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"20'1","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-28T14:08:00.681112+0000","last_change":"2024-03-28T14:07:58.671168+0000","last_active":"2024-03-28T14:08:00.681112+0000","last_peered":"2024-03-28T14:08:00.681112+0000","last_clean":"2024-03-28T14:08:00.681112+0000","last_became_active":"2024-03-28T14:07:58.670691+0000","last_became_peered":"2024-03-28T14:07:58.670691+0000","last_unstale":"2024-03-28T14:08:00.681112+0000","last_undegraded":"2024-03-28T14:08:00.681112+0000","last_fullsized":"2024-03-28T14:08:00.681112+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:07:57.631994+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:07:57.631994+0000","last_clean_scrub_stamp":"2024-03-28T14:07:57.631994+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T17:34:09.030115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-28T14:08:00.680644+0000","last_change":"2024-03-28T14:07:59.727344+0000","last_active":"2024-03-28T14:08:00.680644+0000","last_peered":"2024-03-28T14:08:00.680644+0000","last_clean":"2024-03-28T14:08:00.680644+0000","last_became_active":"2024-03-28T14:07:59.727023+0000","last_became_peered":"2024-03-28T14:07:59.727023+0000","last_unstale":"2024-03-28T14:08:00.680644+0000","last_undegraded":"2024-03-28T14:08:00.680644+0000","last_fullsized":"2024-03-28T14:08:00.680644+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_clean_scrub_stamp":"2024-03-28T14:07:58.641775+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:24:30.197596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"20'4","reported_seq":18,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-28T14:08:01.245810+0000","last_change":"2024-03-28T14:07:58.676958+0000","last_active":"2024-03-28T14:08:01.245810+0000","last_peered":"2024-03-28T14:08:01.245810+0000","last_clean":"2024-03-28T14:08:01.245810+0000","last_became_active":"2024-03-28T14:07:58.676611+0000","last_became_peered":"2024-03-28T14:07:58.676611+0000","last_unstale":"2024-03-28T14:08:01.245810+0000","last_undegraded":"2024-03-28T14:08:01.245810+0000","last_fullsized":"2024-03-28T14:08:01.245810+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:07:57.631994+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:07:57.631994+0000","last_clean_scrub_stamp":"2024-03-28T14:07:57.631994+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:59:04.674426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574855,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27820,"kb_used_data":796,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344020,"statfs":{"total":96636764160,"available":96608276480,"internally_reserved":0,"allocated":815104,"data_stored":666555,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574856,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27248,"kb_used_data":224,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344592,"statfs":{"total":96636764160,"available":96608862208,"internally_reserved":0,"allocated":229376,"data_stored":77217,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574854,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27240,"kb_used_data":216,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344600,"statfs":{"total":96636764160,"available":96608870400,"internally_reserved":0,"allocated":221184,"data_stored":75695,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574855,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27248,"kb_used_data":224,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344592,"statfs":{"total":96636764160,"available":96608862208,"internally_reserved":0,"allocated":229376,"data_stored":75731,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574857,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27248,"kb_used_data":224,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344592,"statfs":{"total":96636764160,"available":96608862208,"internally_reserved":0,"allocated":229376,"data_stored":77217,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574855,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27236,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344604,"statfs":{"total":96636764160,"available":96608874496,"internally_reserved":0,"allocated":217088,"data_stored":75605,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574853,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27244,"kb_used_data":220,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344596,"statfs":{"total":96636764160,"available":96608866304,"internally_reserved":0,"allocated":225280,"data_stored":75765,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574856,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27824,"kb_used_data":804,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344016,"statfs":{"total":96636764160,"available":96608272384,"internally_reserved":0,"allocated":823296,"data_stored":666611,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-28T14:08:06.478 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-28T14:08:06.624 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:06.623+0000 7face319d640 1 Processor -- start 2024-03-28T14:08:06.624 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:06.623+0000 7face319d640 1 -- start start 2024-03-28T14:08:06.626 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:06.623+0000 7face319d640 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7facdc06d970 0x7facdc13f3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:06.626 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:06.623+0000 7face319d640 1 --2- >> v2:172.21.15.171:3300/0 conn(0x7facdc0732a0 0x7facdc13f920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:06.626 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:06.623+0000 7face319d640 1 --2- >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7facdc0b5460 0x7facdc0b0c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:06.626 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:06.623+0000 7face319d640 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 -- 0x7facdc0b02a0 con 0x7facdc06d970 2024-03-28T14:08:06.626 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:06.623+0000 7face319d640 1 -- --> v2:172.21.15.171:3300/0 -- mon_getmap magic: 0 -- 0x7facdc06dd50 con 0x7facdc0732a0 2024-03-28T14:08:06.626 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:06.623+0000 7face319d640 1 -- --> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] -- mon_getmap magic: 0 -- 0x7facdc0b2d30 con 0x7facdc0b5460 2024-03-28T14:08:06.626 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:06.623+0000 7face0f39640 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7facdc06d970 0x7facdc13f3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:06.626 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:06.623+0000 7face0f39640 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7facdc06d970 0x7facdc13f3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:34688/0 (socket says 172.21.15.104:34688) 2024-03-28T14:08:06.627 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:06.623+0000 7face0f39640 1 -- 172.21.15.104:0/1934907186 learned_addr learned my addr 172.21.15.104:0/1934907186 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-03-28T14:08:06.627 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:06.623+0000 7facdbfff640 1 --2- 172.21.15.104:0/1934907186 >> v2:172.21.15.171:3300/0 conn(0x7facdc0732a0 0x7facdc13f920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:06.627 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:06.623+0000 7face173a640 1 --2- 172.21.15.104:0/1934907186 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7facdc0b5460 0x7facdc0b0c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:06.627 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:06.623+0000 7face0f39640 1 -- 172.21.15.104:0/1934907186 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7facdc0b5460 msgr2=0x7facdc0b0c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T14:08:06.627 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:06.623+0000 7face0f39640 1 --2- 172.21.15.104:0/1934907186 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7facdc0b5460 0x7facdc0b0c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:06.627 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:06.623+0000 7face0f39640 1 -- 172.21.15.104:0/1934907186 >> v2:172.21.15.171:3300/0 conn(0x7facdc0732a0 msgr2=0x7facdc13f920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T14:08:06.627 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:06.623+0000 7face0f39640 1 --2- 172.21.15.104:0/1934907186 >> v2:172.21.15.171:3300/0 conn(0x7facdc0732a0 0x7facdc13f920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:06.627 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:06.623+0000 7face0f39640 1 -- 172.21.15.104:0/1934907186 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7facdc0b3010 con 0x7facdc06d970 2024-03-28T14:08:06.628 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:06.627+0000 7face0f39640 1 --2- 172.21.15.104:0/1934907186 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7facdc06d970 0x7facdc13f3e0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7facdc073cb0 tx=0x7facc401c460 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8e14704a0760b0ac server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:08:06.628 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:06.627+0000 7facdb7fe640 1 -- 172.21.15.104:0/1934907186 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7facc40ab600 con 0x7facdc06d970 2024-03-28T14:08:06.628 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:06.627+0000 7facdb7fe640 1 -- 172.21.15.104:0/1934907186 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7facc40b0070 con 0x7facdc06d970 2024-03-28T14:08:06.628 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:06.627+0000 7facdb7fe640 1 -- 172.21.15.104:0/1934907186 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7facc40ab9c0 con 0x7facdc06d970 2024-03-28T14:08:06.629 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:06.627+0000 7face319d640 1 -- 172.21.15.104:0/1934907186 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7facdc06d970 msgr2=0x7facdc13f3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:06.629 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:06.627+0000 7face319d640 1 --2- 172.21.15.104:0/1934907186 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7facdc06d970 0x7facdc13f3e0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7facdc073cb0 tx=0x7facc401c460 comp rx=0 tx=0).stop 2024-03-28T14:08:06.629 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:06.627+0000 7face319d640 1 -- 172.21.15.104:0/1934907186 shutdown_connections 2024-03-28T14:08:06.629 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:06.627+0000 7face319d640 1 --2- 172.21.15.104:0/1934907186 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7facdc0b5460 0x7facdc0b0c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:06.629 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:06.627+0000 7face319d640 1 --2- 172.21.15.104:0/1934907186 >> v2:172.21.15.171:3300/0 conn(0x7facdc0732a0 0x7facdc13f920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:06.629 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:06.627+0000 7face319d640 1 --2- 172.21.15.104:0/1934907186 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7facdc06d970 0x7facdc13f3e0 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:06.629 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:06.627+0000 7face319d640 1 -- 172.21.15.104:0/1934907186 >> 172.21.15.104:0/1934907186 conn(0x7facdc0b58a0 msgr2=0x7facdc0b5cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T14:08:06.630 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:06.627+0000 7face319d640 1 -- 172.21.15.104:0/1934907186 shutdown_connections 2024-03-28T14:08:06.630 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:06.627+0000 7face319d640 1 -- 172.21.15.104:0/1934907186 wait complete. 2024-03-28T14:08:06.632 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:06.627+0000 7face319d640 1 Processor -- start 2024-03-28T14:08:06.632 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:06.627+0000 7face319d640 1 -- start start 2024-03-28T14:08:06.632 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:06.627+0000 7face319d640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7facdc06d970 0x7facdc1e17b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:06.632 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:06.627+0000 7face319d640 1 --2- >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7facdc0732a0 0x7facdc1e1cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:06.632 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:06.627+0000 7face319d640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7facdc0b5460 0x7facdc137af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:06.632 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:06.627+0000 7face319d640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7facdc0b4130 con 0x7facdc0b5460 2024-03-28T14:08:06.632 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:06.627+0000 7face319d640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7facdc0b3fb0 con 0x7facdc06d970 2024-03-28T14:08:06.632 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:06.627+0000 7face319d640 1 -- --> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] -- mon_getmap magic: 0 -- 0x7facdc0b42b0 con 0x7facdc0732a0 2024-03-28T14:08:06.632 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:06.627+0000 7face173a640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7facdc0b5460 0x7facdc137af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:06.632 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:06.627+0000 7face173a640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7facdc0b5460 0x7facdc137af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:34694/0 (socket says 172.21.15.104:34694) 2024-03-28T14:08:06.632 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:06.627+0000 7face173a640 1 -- 172.21.15.104:0/306372281 learned_addr learned my addr 172.21.15.104:0/306372281 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-03-28T14:08:06.632 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:06.627+0000 7face173a640 1 -- 172.21.15.104:0/306372281 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7facdc0732a0 msgr2=0x7facdc1e1cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:06.632 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:06.627+0000 7facdbfff640 1 --2- 172.21.15.104:0/306372281 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7facdc0732a0 0x7facdc1e1cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:06.633 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:06.627+0000 7face173a640 1 --2- 172.21.15.104:0/306372281 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7facdc0732a0 0x7facdc1e1cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:06.633 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:06.627+0000 7face173a640 1 -- 172.21.15.104:0/306372281 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7facdc06d970 msgr2=0x7facdc1e17b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:06.633 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:06.627+0000 7face173a640 1 --2- 172.21.15.104:0/306372281 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7facdc06d970 0x7facdc1e17b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:06.633 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:06.627+0000 7face173a640 1 -- 172.21.15.104:0/306372281 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7facdc13fcb0 con 0x7facdc0b5460 2024-03-28T14:08:06.633 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:06.627+0000 7face173a640 1 --2- 172.21.15.104:0/306372281 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7facdc0b5460 0x7facdc137af0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7facd4009fd0 tx=0x7facd400e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:08:06.633 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:06.631+0000 7facd97fa640 1 -- 172.21.15.104:0/306372281 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7facd4014420 con 0x7facdc0b5460 2024-03-28T14:08:06.633 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:06.631+0000 7facd97fa640 1 -- 172.21.15.104:0/306372281 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7facd4019070 con 0x7facdc0b5460 2024-03-28T14:08:06.633 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:06.631+0000 7face319d640 1 -- 172.21.15.104:0/306372281 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7facdc148f50 con 0x7facdc0b5460 2024-03-28T14:08:06.633 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:06.631+0000 7facd97fa640 1 -- 172.21.15.104:0/306372281 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7facd4014800 con 0x7facdc0b5460 2024-03-28T14:08:06.633 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:06.631+0000 7face319d640 1 -- 172.21.15.104:0/306372281 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7facdc147a10 con 0x7facdc0b5460 2024-03-28T14:08:06.634 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:06.631+0000 7face319d640 1 -- 172.21.15.104:0/306372281 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7facdc0687c0 con 0x7facdc0b5460 2024-03-28T14:08:06.634 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:06.631+0000 7facd97fa640 1 -- 172.21.15.104:0/306372281 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) ==== 87803+0+0 (secure 0 0 0) 0x7facd40149a0 con 0x7facdc0b5460 2024-03-28T14:08:06.634 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:06.631+0000 7facd97fa640 1 --2- 172.21.15.104:0/306372281 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7facbc067ff0 0x7facbc06a4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:06.634 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:06.631+0000 7facd97fa640 1 -- 172.21.15.104:0/306372281 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7facd4086c90 con 0x7facdc0b5460 2024-03-28T14:08:06.635 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:06.631+0000 7face0f39640 1 --2- 172.21.15.104:0/306372281 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7facbc067ff0 0x7facbc06a4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:06.638 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:06.631+0000 7face0f39640 1 --2- 172.21.15.104:0/306372281 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7facbc067ff0 0x7facbc06a4b0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7facc40af000 tx=0x7facc4004740 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:08:06.638 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:06.635+0000 7facd97fa640 1 -- 172.21.15.104:0/306372281 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179117 (secure 0 0 0) 0x7facd4057440 con 0x7facdc0b5460 2024-03-28T14:08:06.803 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:06.799+0000 7face319d640 1 -- 172.21.15.104:0/306372281 --> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7facdc147ea0 con 0x7facbc067ff0 2024-03-28T14:08:06.820 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:06.819+0000 7facd97fa640 1 -- 172.21.15.104:0/306372281 <== mgr.4100 v2:172.21.15.171:6832/1522752412 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+251836 (secure 0 0 0) 0x7facdc147ea0 con 0x7facbc067ff0 2024-03-28T14:08:06.823 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-03-28T14:08:06.825 INFO:teuthology.orchestra.run.smithi104.stderr:dumped all 2024-03-28T14:08:06.826 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:06.823+0000 7face319d640 1 -- 172.21.15.104:0/306372281 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7facbc067ff0 msgr2=0x7facbc06a4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:06.826 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:06.823+0000 7face319d640 1 --2- 172.21.15.104:0/306372281 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7facbc067ff0 0x7facbc06a4b0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7facc40af000 tx=0x7facc4004740 comp rx=0 tx=0).stop 2024-03-28T14:08:06.826 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:06.823+0000 7face319d640 1 -- 172.21.15.104:0/306372281 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7facdc0b5460 msgr2=0x7facdc137af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:06.826 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:06.823+0000 7face319d640 1 --2- 172.21.15.104:0/306372281 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7facdc0b5460 0x7facdc137af0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7facd4009fd0 tx=0x7facd400e4b0 comp rx=0 tx=0).stop 2024-03-28T14:08:06.826 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:06.823+0000 7face319d640 1 -- 172.21.15.104:0/306372281 shutdown_connections 2024-03-28T14:08:06.826 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:06.823+0000 7face319d640 1 --2- 172.21.15.104:0/306372281 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7facbc067ff0 0x7facbc06a4b0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:06.827 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:06.823+0000 7face319d640 1 --2- 172.21.15.104:0/306372281 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7facdc0b5460 0x7facdc137af0 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:06.827 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:06.823+0000 7face319d640 1 --2- 172.21.15.104:0/306372281 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7facdc0732a0 0x7facdc1e1cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:06.827 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:06.823+0000 7face319d640 1 --2- 172.21.15.104:0/306372281 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7facdc06d970 0x7facdc1e17b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:06.827 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:06.823+0000 7face319d640 1 -- 172.21.15.104:0/306372281 >> 172.21.15.104:0/306372281 conn(0x7facdc0b58a0 msgr2=0x7facdc145590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T14:08:06.827 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:06.823+0000 7face319d640 1 -- 172.21.15.104:0/306372281 shutdown_connections 2024-03-28T14:08:06.827 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:06.823+0000 7face319d640 1 -- 172.21.15.104:0/306372281 wait complete. 2024-03-28T14:08:06.840 INFO:teuthology.orchestra.run.smithi104.stdout:{"pg_ready":true,"pg_map":{"version":24,"stamp":"2024-03-28T14:08:05.579087+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219108,"kb_used_data":2920,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755612,"statfs":{"total":773094113280,"available":772869746688,"internally_reserved":0,"allocated":2990080,"data_stored":1790396,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":221236785},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.930250"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-28T14:08:00.835047+0000","last_change":"2024-03-28T14:07:59.730372+0000","last_active":"2024-03-28T14:08:00.835047+0000","last_peered":"2024-03-28T14:08:00.835047+0000","last_clean":"2024-03-28T14:08:00.835047+0000","last_became_active":"2024-03-28T14:07:59.729644+0000","last_became_peered":"2024-03-28T14:07:59.729644+0000","last_unstale":"2024-03-28T14:08:00.835047+0000","last_undegraded":"2024-03-28T14:08:00.835047+0000","last_fullsized":"2024-03-28T14:08:00.835047+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_clean_scrub_stamp":"2024-03-28T14:07:58.641775+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:59:25.862480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-28T14:08:00.696778+0000","last_change":"2024-03-28T14:07:59.705187+0000","last_active":"2024-03-28T14:08:00.696778+0000","last_peered":"2024-03-28T14:08:00.696778+0000","last_clean":"2024-03-28T14:08:00.696778+0000","last_became_active":"2024-03-28T14:07:59.700386+0000","last_became_peered":"2024-03-28T14:07:59.700386+0000","last_unstale":"2024-03-28T14:08:00.696778+0000","last_undegraded":"2024-03-28T14:08:00.696778+0000","last_fullsized":"2024-03-28T14:08:00.696778+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_clean_scrub_stamp":"2024-03-28T14:07:58.641775+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T16:47:04.317583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-28T14:08:01.245347+0000","last_change":"2024-03-28T14:07:59.698753+0000","last_active":"2024-03-28T14:08:01.245347+0000","last_peered":"2024-03-28T14:08:01.245347+0000","last_clean":"2024-03-28T14:08:01.245347+0000","last_became_active":"2024-03-28T14:07:59.698467+0000","last_became_peered":"2024-03-28T14:07:59.698467+0000","last_unstale":"2024-03-28T14:08:01.245347+0000","last_undegraded":"2024-03-28T14:08:01.245347+0000","last_fullsized":"2024-03-28T14:08:01.245347+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_clean_scrub_stamp":"2024-03-28T14:07:58.641775+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:25:39.647896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-28T14:08:00.680934+0000","last_change":"2024-03-28T14:07:59.747773+0000","last_active":"2024-03-28T14:08:00.680934+0000","last_peered":"2024-03-28T14:08:00.680934+0000","last_clean":"2024-03-28T14:08:00.680934+0000","last_became_active":"2024-03-28T14:07:59.745640+0000","last_became_peered":"2024-03-28T14:07:59.745640+0000","last_unstale":"2024-03-28T14:08:00.680934+0000","last_undegraded":"2024-03-28T14:08:00.680934+0000","last_fullsized":"2024-03-28T14:08:00.680934+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_clean_scrub_stamp":"2024-03-28T14:07:58.641775+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:10:31.257059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-28T14:08:00.834115+0000","last_change":"2024-03-28T14:07:59.722923+0000","last_active":"2024-03-28T14:08:00.834115+0000","last_peered":"2024-03-28T14:08:00.834115+0000","last_clean":"2024-03-28T14:08:00.834115+0000","last_became_active":"2024-03-28T14:07:59.711460+0000","last_became_peered":"2024-03-28T14:07:59.711460+0000","last_unstale":"2024-03-28T14:08:00.834115+0000","last_undegraded":"2024-03-28T14:08:00.834115+0000","last_fullsized":"2024-03-28T14:08:00.834115+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_clean_scrub_stamp":"2024-03-28T14:07:58.641775+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T18:51:19.214005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-28T14:08:00.680087+0000","last_change":"2024-03-28T14:07:59.742906+0000","last_active":"2024-03-28T14:08:00.680087+0000","last_peered":"2024-03-28T14:08:00.680087+0000","last_clean":"2024-03-28T14:08:00.680087+0000","last_became_active":"2024-03-28T14:07:59.734609+0000","last_became_peered":"2024-03-28T14:07:59.734609+0000","last_unstale":"2024-03-28T14:08:00.680087+0000","last_undegraded":"2024-03-28T14:08:00.680087+0000","last_fullsized":"2024-03-28T14:08:00.680087+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_clean_scrub_stamp":"2024-03-28T14:07:58.641775+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:41:27.409736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-28T14:08:00.833773+0000","last_change":"2024-03-28T14:07:59.717632+0000","last_active":"2024-03-28T14:08:00.833773+0000","last_peered":"2024-03-28T14:08:00.833773+0000","last_clean":"2024-03-28T14:08:00.833773+0000","last_became_active":"2024-03-28T14:07:59.711894+0000","last_became_peered":"2024-03-28T14:07:59.711894+0000","last_unstale":"2024-03-28T14:08:00.833773+0000","last_undegraded":"2024-03-28T14:08:00.833773+0000","last_fullsized":"2024-03-28T14:08:00.833773+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_clean_scrub_stamp":"2024-03-28T14:07:58.641775+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T16:47:04.317583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-28T14:08:00.680228+0000","last_change":"2024-03-28T14:07:59.740843+0000","last_active":"2024-03-28T14:08:00.680228+0000","last_peered":"2024-03-28T14:08:00.680228+0000","last_clean":"2024-03-28T14:08:00.680228+0000","last_became_active":"2024-03-28T14:07:59.740606+0000","last_became_peered":"2024-03-28T14:07:59.740606+0000","last_unstale":"2024-03-28T14:08:00.680228+0000","last_undegraded":"2024-03-28T14:08:00.680228+0000","last_fullsized":"2024-03-28T14:08:00.680228+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_clean_scrub_stamp":"2024-03-28T14:07:58.641775+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T14:20:33.555524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-28T14:08:00.835049+0000","last_change":"2024-03-28T14:07:59.745208+0000","last_active":"2024-03-28T14:08:00.835049+0000","last_peered":"2024-03-28T14:08:00.835049+0000","last_clean":"2024-03-28T14:08:00.835049+0000","last_became_active":"2024-03-28T14:07:59.744704+0000","last_became_peered":"2024-03-28T14:07:59.744704+0000","last_unstale":"2024-03-28T14:08:00.835049+0000","last_undegraded":"2024-03-28T14:08:00.835049+0000","last_fullsized":"2024-03-28T14:08:00.835049+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_clean_scrub_stamp":"2024-03-28T14:07:58.641775+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:53:55.059948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-28T14:08:00.703336+0000","last_change":"2024-03-28T14:07:59.744713+0000","last_active":"2024-03-28T14:08:00.703336+0000","last_peered":"2024-03-28T14:08:00.703336+0000","last_clean":"2024-03-28T14:08:00.703336+0000","last_became_active":"2024-03-28T14:07:59.744134+0000","last_became_peered":"2024-03-28T14:07:59.744134+0000","last_unstale":"2024-03-28T14:08:00.703336+0000","last_undegraded":"2024-03-28T14:08:00.703336+0000","last_fullsized":"2024-03-28T14:08:00.703336+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_clean_scrub_stamp":"2024-03-28T14:07:58.641775+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:25:39.647896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-28T14:08:00.703286+0000","last_change":"2024-03-28T14:07:59.745769+0000","last_active":"2024-03-28T14:08:00.703286+0000","last_peered":"2024-03-28T14:08:00.703286+0000","last_clean":"2024-03-28T14:08:00.703286+0000","last_became_active":"2024-03-28T14:07:59.745151+0000","last_became_peered":"2024-03-28T14:07:59.745151+0000","last_unstale":"2024-03-28T14:08:00.703286+0000","last_undegraded":"2024-03-28T14:08:00.703286+0000","last_fullsized":"2024-03-28T14:08:00.703286+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_clean_scrub_stamp":"2024-03-28T14:07:58.641775+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:37:29.402787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-28T14:08:01.244022+0000","last_change":"2024-03-28T14:07:59.702599+0000","last_active":"2024-03-28T14:08:01.244022+0000","last_peered":"2024-03-28T14:08:01.244022+0000","last_clean":"2024-03-28T14:08:01.244022+0000","last_became_active":"2024-03-28T14:07:59.701506+0000","last_became_peered":"2024-03-28T14:07:59.701506+0000","last_unstale":"2024-03-28T14:08:01.244022+0000","last_undegraded":"2024-03-28T14:08:01.244022+0000","last_fullsized":"2024-03-28T14:08:01.244022+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_clean_scrub_stamp":"2024-03-28T14:07:58.641775+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T15:40:59.112394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-28T14:08:00.680979+0000","last_change":"2024-03-28T14:07:59.704862+0000","last_active":"2024-03-28T14:08:00.680979+0000","last_peered":"2024-03-28T14:08:00.680979+0000","last_clean":"2024-03-28T14:08:00.680979+0000","last_became_active":"2024-03-28T14:07:59.704461+0000","last_became_peered":"2024-03-28T14:07:59.704461+0000","last_unstale":"2024-03-28T14:08:00.680979+0000","last_undegraded":"2024-03-28T14:08:00.680979+0000","last_fullsized":"2024-03-28T14:08:00.680979+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_clean_scrub_stamp":"2024-03-28T14:07:58.641775+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:37:29.402787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-28T14:08:01.243740+0000","last_change":"2024-03-28T14:07:59.707399+0000","last_active":"2024-03-28T14:08:01.243740+0000","last_peered":"2024-03-28T14:08:01.243740+0000","last_clean":"2024-03-28T14:08:01.243740+0000","last_became_active":"2024-03-28T14:07:59.707057+0000","last_became_peered":"2024-03-28T14:07:59.707057+0000","last_unstale":"2024-03-28T14:08:01.243740+0000","last_undegraded":"2024-03-28T14:08:01.243740+0000","last_fullsized":"2024-03-28T14:08:01.243740+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_clean_scrub_stamp":"2024-03-28T14:07:58.641775+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:54:39.259029+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-28T14:08:00.696269+0000","last_change":"2024-03-28T14:07:59.684910+0000","last_active":"2024-03-28T14:08:00.696269+0000","last_peered":"2024-03-28T14:08:00.696269+0000","last_clean":"2024-03-28T14:08:00.696269+0000","last_became_active":"2024-03-28T14:07:59.684112+0000","last_became_peered":"2024-03-28T14:07:59.684112+0000","last_unstale":"2024-03-28T14:08:00.696269+0000","last_undegraded":"2024-03-28T14:08:00.696269+0000","last_fullsized":"2024-03-28T14:08:00.696269+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_clean_scrub_stamp":"2024-03-28T14:07:58.641775+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:37:29.402787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-28T14:08:00.680001+0000","last_change":"2024-03-28T14:07:59.740138+0000","last_active":"2024-03-28T14:08:00.680001+0000","last_peered":"2024-03-28T14:08:00.680001+0000","last_clean":"2024-03-28T14:08:00.680001+0000","last_became_active":"2024-03-28T14:07:59.739320+0000","last_became_peered":"2024-03-28T14:07:59.739320+0000","last_unstale":"2024-03-28T14:08:00.680001+0000","last_undegraded":"2024-03-28T14:08:00.680001+0000","last_fullsized":"2024-03-28T14:08:00.680001+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_clean_scrub_stamp":"2024-03-28T14:07:58.641775+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:25:39.647896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-28T14:08:00.681671+0000","last_change":"2024-03-28T14:07:59.744839+0000","last_active":"2024-03-28T14:08:00.681671+0000","last_peered":"2024-03-28T14:08:00.681671+0000","last_clean":"2024-03-28T14:08:00.681671+0000","last_became_active":"2024-03-28T14:07:59.739555+0000","last_became_peered":"2024-03-28T14:07:59.739555+0000","last_unstale":"2024-03-28T14:08:00.681671+0000","last_undegraded":"2024-03-28T14:08:00.681671+0000","last_fullsized":"2024-03-28T14:08:00.681671+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_clean_scrub_stamp":"2024-03-28T14:07:58.641775+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:53:55.059948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-28T14:08:01.242371+0000","last_change":"2024-03-28T14:07:59.707740+0000","last_active":"2024-03-28T14:08:01.242371+0000","last_peered":"2024-03-28T14:08:01.242371+0000","last_clean":"2024-03-28T14:08:01.242371+0000","last_became_active":"2024-03-28T14:07:59.702876+0000","last_became_peered":"2024-03-28T14:07:59.702876+0000","last_unstale":"2024-03-28T14:08:01.242371+0000","last_undegraded":"2024-03-28T14:08:01.242371+0000","last_fullsized":"2024-03-28T14:08:01.242371+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_clean_scrub_stamp":"2024-03-28T14:07:58.641775+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T15:40:01.667745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-28T14:08:00.834771+0000","last_change":"2024-03-28T14:07:59.723973+0000","last_active":"2024-03-28T14:08:00.834771+0000","last_peered":"2024-03-28T14:08:00.834771+0000","last_clean":"2024-03-28T14:08:00.834771+0000","last_became_active":"2024-03-28T14:07:59.712162+0000","last_became_peered":"2024-03-28T14:07:59.712162+0000","last_unstale":"2024-03-28T14:08:00.834771+0000","last_undegraded":"2024-03-28T14:08:00.834771+0000","last_fullsized":"2024-03-28T14:08:00.834771+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_clean_scrub_stamp":"2024-03-28T14:07:58.641775+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:51:23.322586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-28T14:08:00.703000+0000","last_change":"2024-03-28T14:07:59.704917+0000","last_active":"2024-03-28T14:08:00.703000+0000","last_peered":"2024-03-28T14:08:00.703000+0000","last_clean":"2024-03-28T14:08:00.703000+0000","last_became_active":"2024-03-28T14:07:59.702369+0000","last_became_peered":"2024-03-28T14:07:59.702369+0000","last_unstale":"2024-03-28T14:08:00.703000+0000","last_undegraded":"2024-03-28T14:08:00.703000+0000","last_fullsized":"2024-03-28T14:08:00.703000+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_clean_scrub_stamp":"2024-03-28T14:07:58.641775+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:24:30.197596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-28T14:08:01.244077+0000","last_change":"2024-03-28T14:07:59.727121+0000","last_active":"2024-03-28T14:08:01.244077+0000","last_peered":"2024-03-28T14:08:01.244077+0000","last_clean":"2024-03-28T14:08:01.244077+0000","last_became_active":"2024-03-28T14:07:59.726783+0000","last_became_peered":"2024-03-28T14:07:59.726783+0000","last_unstale":"2024-03-28T14:08:01.244077+0000","last_undegraded":"2024-03-28T14:08:01.244077+0000","last_fullsized":"2024-03-28T14:08:01.244077+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_clean_scrub_stamp":"2024-03-28T14:07:58.641775+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:10:27.248645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-28T14:08:00.680533+0000","last_change":"2024-03-28T14:07:59.742729+0000","last_active":"2024-03-28T14:08:00.680533+0000","last_peered":"2024-03-28T14:08:00.680533+0000","last_clean":"2024-03-28T14:08:00.680533+0000","last_became_active":"2024-03-28T14:07:59.739727+0000","last_became_peered":"2024-03-28T14:07:59.739727+0000","last_unstale":"2024-03-28T14:08:00.680533+0000","last_undegraded":"2024-03-28T14:08:00.680533+0000","last_fullsized":"2024-03-28T14:08:00.680533+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_clean_scrub_stamp":"2024-03-28T14:07:58.641775+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T17:46:06.253111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-28T14:08:00.681600+0000","last_change":"2024-03-28T14:07:59.704724+0000","last_active":"2024-03-28T14:08:00.681600+0000","last_peered":"2024-03-28T14:08:00.681600+0000","last_clean":"2024-03-28T14:08:00.681600+0000","last_became_active":"2024-03-28T14:07:59.704398+0000","last_became_peered":"2024-03-28T14:07:59.704398+0000","last_unstale":"2024-03-28T14:08:00.681600+0000","last_undegraded":"2024-03-28T14:08:00.681600+0000","last_fullsized":"2024-03-28T14:08:00.681600+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_clean_scrub_stamp":"2024-03-28T14:07:58.641775+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T18:51:19.214005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-28T14:08:00.680302+0000","last_change":"2024-03-28T14:07:59.739100+0000","last_active":"2024-03-28T14:08:00.680302+0000","last_peered":"2024-03-28T14:08:00.680302+0000","last_clean":"2024-03-28T14:08:00.680302+0000","last_became_active":"2024-03-28T14:07:59.733884+0000","last_became_peered":"2024-03-28T14:07:59.733884+0000","last_unstale":"2024-03-28T14:08:00.680302+0000","last_undegraded":"2024-03-28T14:08:00.680302+0000","last_fullsized":"2024-03-28T14:08:00.680302+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_clean_scrub_stamp":"2024-03-28T14:07:58.641775+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T16:47:04.317583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-28T14:08:00.681010+0000","last_change":"2024-03-28T14:07:59.716106+0000","last_active":"2024-03-28T14:08:00.681010+0000","last_peered":"2024-03-28T14:08:00.681010+0000","last_clean":"2024-03-28T14:08:00.681010+0000","last_became_active":"2024-03-28T14:07:59.715747+0000","last_became_peered":"2024-03-28T14:07:59.715747+0000","last_unstale":"2024-03-28T14:08:00.681010+0000","last_undegraded":"2024-03-28T14:08:00.681010+0000","last_fullsized":"2024-03-28T14:08:00.681010+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_clean_scrub_stamp":"2024-03-28T14:07:58.641775+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:25:39.647896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-28T14:08:00.703528+0000","last_change":"2024-03-28T14:07:59.710509+0000","last_active":"2024-03-28T14:08:00.703528+0000","last_peered":"2024-03-28T14:08:00.703528+0000","last_clean":"2024-03-28T14:08:00.703528+0000","last_became_active":"2024-03-28T14:07:59.709767+0000","last_became_peered":"2024-03-28T14:07:59.709767+0000","last_unstale":"2024-03-28T14:08:00.703528+0000","last_undegraded":"2024-03-28T14:08:00.703528+0000","last_fullsized":"2024-03-28T14:08:00.703528+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_clean_scrub_stamp":"2024-03-28T14:07:58.641775+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:41:27.409736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-28T14:08:00.696547+0000","last_change":"2024-03-28T14:07:59.704832+0000","last_active":"2024-03-28T14:08:00.696547+0000","last_peered":"2024-03-28T14:08:00.696547+0000","last_clean":"2024-03-28T14:08:00.696547+0000","last_became_active":"2024-03-28T14:07:59.704546+0000","last_became_peered":"2024-03-28T14:07:59.704546+0000","last_unstale":"2024-03-28T14:08:00.696547+0000","last_undegraded":"2024-03-28T14:08:00.696547+0000","last_fullsized":"2024-03-28T14:08:00.696547+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_clean_scrub_stamp":"2024-03-28T14:07:58.641775+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:41:27.409736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-28T14:08:00.681706+0000","last_change":"2024-03-28T14:07:59.702155+0000","last_active":"2024-03-28T14:08:00.681706+0000","last_peered":"2024-03-28T14:08:00.681706+0000","last_clean":"2024-03-28T14:08:00.681706+0000","last_became_active":"2024-03-28T14:07:59.701799+0000","last_became_peered":"2024-03-28T14:07:59.701799+0000","last_unstale":"2024-03-28T14:08:00.681706+0000","last_undegraded":"2024-03-28T14:08:00.681706+0000","last_fullsized":"2024-03-28T14:08:00.681706+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_clean_scrub_stamp":"2024-03-28T14:07:58.641775+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:10:31.257059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-28T14:08:01.244122+0000","last_change":"2024-03-28T14:07:59.706889+0000","last_active":"2024-03-28T14:08:01.244122+0000","last_peered":"2024-03-28T14:08:01.244122+0000","last_clean":"2024-03-28T14:08:01.244122+0000","last_became_active":"2024-03-28T14:07:59.706631+0000","last_became_peered":"2024-03-28T14:07:59.706631+0000","last_unstale":"2024-03-28T14:08:01.244122+0000","last_undegraded":"2024-03-28T14:08:01.244122+0000","last_fullsized":"2024-03-28T14:08:01.244122+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_clean_scrub_stamp":"2024-03-28T14:07:58.641775+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:57:11.908191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-28T14:08:00.680602+0000","last_change":"2024-03-28T14:07:59.722801+0000","last_active":"2024-03-28T14:08:00.680602+0000","last_peered":"2024-03-28T14:08:00.680602+0000","last_clean":"2024-03-28T14:08:00.680602+0000","last_became_active":"2024-03-28T14:07:59.722132+0000","last_became_peered":"2024-03-28T14:07:59.722132+0000","last_unstale":"2024-03-28T14:08:00.680602+0000","last_undegraded":"2024-03-28T14:08:00.680602+0000","last_fullsized":"2024-03-28T14:08:00.680602+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_clean_scrub_stamp":"2024-03-28T14:07:58.641775+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T18:51:19.214005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-28T14:08:01.242238+0000","last_change":"2024-03-28T14:07:59.740448+0000","last_active":"2024-03-28T14:08:01.242238+0000","last_peered":"2024-03-28T14:08:01.242238+0000","last_clean":"2024-03-28T14:08:01.242238+0000","last_became_active":"2024-03-28T14:07:59.740121+0000","last_became_peered":"2024-03-28T14:07:59.740121+0000","last_unstale":"2024-03-28T14:08:01.242238+0000","last_undegraded":"2024-03-28T14:08:01.242238+0000","last_fullsized":"2024-03-28T14:08:01.242238+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_clean_scrub_stamp":"2024-03-28T14:07:58.641775+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T15:40:59.112394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-28T14:08:00.702471+0000","last_change":"2024-03-28T14:07:59.704155+0000","last_active":"2024-03-28T14:08:00.702471+0000","last_peered":"2024-03-28T14:08:00.702471+0000","last_clean":"2024-03-28T14:08:00.702471+0000","last_became_active":"2024-03-28T14:07:59.702173+0000","last_became_peered":"2024-03-28T14:07:59.702173+0000","last_unstale":"2024-03-28T14:08:00.702471+0000","last_undegraded":"2024-03-28T14:08:00.702471+0000","last_fullsized":"2024-03-28T14:08:00.702471+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_clean_scrub_stamp":"2024-03-28T14:07:58.641775+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T14:20:33.555524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-28T14:08:01.243508+0000","last_change":"2024-03-28T14:07:59.682929+0000","last_active":"2024-03-28T14:08:01.243508+0000","last_peered":"2024-03-28T14:08:01.243508+0000","last_clean":"2024-03-28T14:08:01.243508+0000","last_became_active":"2024-03-28T14:07:59.682238+0000","last_became_peered":"2024-03-28T14:07:59.682238+0000","last_unstale":"2024-03-28T14:08:01.243508+0000","last_undegraded":"2024-03-28T14:08:01.243508+0000","last_fullsized":"2024-03-28T14:08:01.243508+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_clean_scrub_stamp":"2024-03-28T14:07:58.641775+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T17:35:01.163718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"20'1","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-28T14:08:00.681702+0000","last_change":"2024-03-28T14:07:58.672879+0000","last_active":"2024-03-28T14:08:00.681702+0000","last_peered":"2024-03-28T14:08:00.681702+0000","last_clean":"2024-03-28T14:08:00.681702+0000","last_became_active":"2024-03-28T14:07:58.672646+0000","last_became_peered":"2024-03-28T14:07:58.672646+0000","last_unstale":"2024-03-28T14:08:00.681702+0000","last_undegraded":"2024-03-28T14:08:00.681702+0000","last_fullsized":"2024-03-28T14:08:00.681702+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:07:57.631994+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:07:57.631994+0000","last_clean_scrub_stamp":"2024-03-28T14:07:57.631994+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:35:03.722392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-28T14:08:01.244389+0000","last_change":"2024-03-28T14:07:59.715972+0000","last_active":"2024-03-28T14:08:01.244389+0000","last_peered":"2024-03-28T14:08:01.244389+0000","last_clean":"2024-03-28T14:08:01.244389+0000","last_became_active":"2024-03-28T14:07:59.715621+0000","last_became_peered":"2024-03-28T14:07:59.715621+0000","last_unstale":"2024-03-28T14:08:01.244389+0000","last_undegraded":"2024-03-28T14:08:01.244389+0000","last_fullsized":"2024-03-28T14:08:01.244389+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_clean_scrub_stamp":"2024-03-28T14:07:58.641775+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:24:30.197596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1a","version":"20'1","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-28T14:08:00.702735+0000","last_change":"2024-03-28T14:07:58.688093+0000","last_active":"2024-03-28T14:08:00.702735+0000","last_peered":"2024-03-28T14:08:00.702735+0000","last_clean":"2024-03-28T14:08:00.702735+0000","last_became_active":"2024-03-28T14:07:58.687279+0000","last_became_peered":"2024-03-28T14:07:58.687279+0000","last_unstale":"2024-03-28T14:08:00.702735+0000","last_undegraded":"2024-03-28T14:08:00.702735+0000","last_fullsized":"2024-03-28T14:08:00.702735+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:07:57.631994+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:07:57.631994+0000","last_clean_scrub_stamp":"2024-03-28T14:07:57.631994+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T18:00:55.131543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-28T14:08:00.680142+0000","last_change":"2024-03-28T14:07:58.678668+0000","last_active":"2024-03-28T14:08:00.680142+0000","last_peered":"2024-03-28T14:08:00.680142+0000","last_clean":"2024-03-28T14:08:00.680142+0000","last_became_active":"2024-03-28T14:07:58.678292+0000","last_became_peered":"2024-03-28T14:07:58.678292+0000","last_unstale":"2024-03-28T14:08:00.680142+0000","last_undegraded":"2024-03-28T14:08:00.680142+0000","last_fullsized":"2024-03-28T14:08:00.680142+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:07:57.631994+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:07:57.631994+0000","last_clean_scrub_stamp":"2024-03-28T14:07:57.631994+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T18:00:55.131543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-28T14:08:00.680364+0000","last_change":"2024-03-28T14:07:59.703432+0000","last_active":"2024-03-28T14:08:00.680364+0000","last_peered":"2024-03-28T14:08:00.680364+0000","last_clean":"2024-03-28T14:08:00.680364+0000","last_became_active":"2024-03-28T14:07:59.702896+0000","last_became_peered":"2024-03-28T14:07:59.702896+0000","last_unstale":"2024-03-28T14:08:00.680364+0000","last_undegraded":"2024-03-28T14:08:00.680364+0000","last_fullsized":"2024-03-28T14:08:00.680364+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_clean_scrub_stamp":"2024-03-28T14:07:58.641775+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:10:27.248645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-28T14:08:00.680262+0000","last_change":"2024-03-28T14:07:58.703631+0000","last_active":"2024-03-28T14:08:00.680262+0000","last_peered":"2024-03-28T14:08:00.680262+0000","last_clean":"2024-03-28T14:08:00.680262+0000","last_became_active":"2024-03-28T14:07:58.702897+0000","last_became_peered":"2024-03-28T14:07:58.702897+0000","last_unstale":"2024-03-28T14:08:00.680262+0000","last_undegraded":"2024-03-28T14:08:00.680262+0000","last_fullsized":"2024-03-28T14:08:00.680262+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:07:57.631994+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:07:57.631994+0000","last_clean_scrub_stamp":"2024-03-28T14:07:57.631994+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:35:03.722392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-28T14:08:00.696158+0000","last_change":"2024-03-28T14:07:59.706310+0000","last_active":"2024-03-28T14:08:00.696158+0000","last_peered":"2024-03-28T14:08:00.696158+0000","last_clean":"2024-03-28T14:08:00.696158+0000","last_became_active":"2024-03-28T14:07:59.705967+0000","last_became_peered":"2024-03-28T14:07:59.705967+0000","last_unstale":"2024-03-28T14:08:00.696158+0000","last_undegraded":"2024-03-28T14:08:00.696158+0000","last_fullsized":"2024-03-28T14:08:00.696158+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_clean_scrub_stamp":"2024-03-28T14:07:58.641775+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:24:30.197596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-28T14:08:00.703408+0000","last_change":"2024-03-28T14:07:58.702042+0000","last_active":"2024-03-28T14:08:00.703408+0000","last_peered":"2024-03-28T14:08:00.703408+0000","last_clean":"2024-03-28T14:08:00.703408+0000","last_became_active":"2024-03-28T14:07:58.701395+0000","last_became_peered":"2024-03-28T14:07:58.701395+0000","last_unstale":"2024-03-28T14:08:00.703408+0000","last_undegraded":"2024-03-28T14:08:00.703408+0000","last_fullsized":"2024-03-28T14:08:00.703408+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:07:57.631994+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:07:57.631994+0000","last_clean_scrub_stamp":"2024-03-28T14:07:57.631994+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T17:35:00.153938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-28T14:08:00.681271+0000","last_change":"2024-03-28T14:07:59.740904+0000","last_active":"2024-03-28T14:08:00.681271+0000","last_peered":"2024-03-28T14:08:00.681271+0000","last_clean":"2024-03-28T14:08:00.681271+0000","last_became_active":"2024-03-28T14:07:59.740209+0000","last_became_peered":"2024-03-28T14:07:59.740209+0000","last_unstale":"2024-03-28T14:08:00.681271+0000","last_undegraded":"2024-03-28T14:08:00.681271+0000","last_fullsized":"2024-03-28T14:08:00.681271+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_clean_scrub_stamp":"2024-03-28T14:07:58.641775+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:02:48.017536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-28T14:08:00.681223+0000","last_change":"2024-03-28T14:07:58.699895+0000","last_active":"2024-03-28T14:08:00.681223+0000","last_peered":"2024-03-28T14:08:00.681223+0000","last_clean":"2024-03-28T14:08:00.681223+0000","last_became_active":"2024-03-28T14:07:58.698823+0000","last_became_peered":"2024-03-28T14:07:58.698823+0000","last_unstale":"2024-03-28T14:08:00.681223+0000","last_undegraded":"2024-03-28T14:08:00.681223+0000","last_fullsized":"2024-03-28T14:08:00.681223+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:07:57.631994+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:07:57.631994+0000","last_clean_scrub_stamp":"2024-03-28T14:07:57.631994+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:09:38.264121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-28T14:08:01.245669+0000","last_change":"2024-03-28T14:07:59.744137+0000","last_active":"2024-03-28T14:08:01.245669+0000","last_peered":"2024-03-28T14:08:01.245669+0000","last_clean":"2024-03-28T14:08:01.245669+0000","last_became_active":"2024-03-28T14:07:59.743854+0000","last_became_peered":"2024-03-28T14:07:59.743854+0000","last_unstale":"2024-03-28T14:08:01.245669+0000","last_undegraded":"2024-03-28T14:08:01.245669+0000","last_fullsized":"2024-03-28T14:08:01.245669+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_clean_scrub_stamp":"2024-03-28T14:07:58.641775+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:41:27.409736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"20'3","reported_seq":17,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-28T14:08:00.696618+0000","last_change":"2024-03-28T14:07:58.703242+0000","last_active":"2024-03-28T14:08:00.696618+0000","last_peered":"2024-03-28T14:08:00.696618+0000","last_clean":"2024-03-28T14:08:00.696618+0000","last_became_active":"2024-03-28T14:07:58.702970+0000","last_became_peered":"2024-03-28T14:07:58.702970+0000","last_unstale":"2024-03-28T14:08:00.696618+0000","last_undegraded":"2024-03-28T14:08:00.696618+0000","last_fullsized":"2024-03-28T14:08:00.696618+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:07:57.631994+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:07:57.631994+0000","last_clean_scrub_stamp":"2024-03-28T14:07:57.631994+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T15:40:58.102613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-28T14:08:01.244046+0000","last_change":"2024-03-28T14:07:59.708551+0000","last_active":"2024-03-28T14:08:01.244046+0000","last_peered":"2024-03-28T14:08:01.244046+0000","last_clean":"2024-03-28T14:08:01.244046+0000","last_became_active":"2024-03-28T14:07:59.706665+0000","last_became_peered":"2024-03-28T14:07:59.706665+0000","last_unstale":"2024-03-28T14:08:01.244046+0000","last_undegraded":"2024-03-28T14:08:01.244046+0000","last_fullsized":"2024-03-28T14:08:01.244046+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_clean_scrub_stamp":"2024-03-28T14:07:58.641775+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T15:40:01.667745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.14","version":"20'6","reported_seq":20,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-28T14:08:00.702870+0000","last_change":"2024-03-28T14:07:58.680343+0000","last_active":"2024-03-28T14:08:00.702870+0000","last_peered":"2024-03-28T14:08:00.702870+0000","last_clean":"2024-03-28T14:08:00.702870+0000","last_became_active":"2024-03-28T14:07:58.679621+0000","last_became_peered":"2024-03-28T14:07:58.679621+0000","last_unstale":"2024-03-28T14:08:00.702870+0000","last_undegraded":"2024-03-28T14:08:00.702870+0000","last_fullsized":"2024-03-28T14:08:00.702870+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:07:57.631994+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:07:57.631994+0000","last_clean_scrub_stamp":"2024-03-28T14:07:57.631994+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:30:19.088624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-28T14:08:00.834351+0000","last_change":"2024-03-28T14:07:59.714064+0000","last_active":"2024-03-28T14:08:00.834351+0000","last_peered":"2024-03-28T14:08:00.834351+0000","last_clean":"2024-03-28T14:08:00.834351+0000","last_became_active":"2024-03-28T14:07:59.710619+0000","last_became_peered":"2024-03-28T14:07:59.710619+0000","last_unstale":"2024-03-28T14:08:00.834351+0000","last_undegraded":"2024-03-28T14:08:00.834351+0000","last_fullsized":"2024-03-28T14:08:00.834351+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_clean_scrub_stamp":"2024-03-28T14:07:58.641775+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:34:47.572488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"20'2","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-28T14:08:01.244515+0000","last_change":"2024-03-28T14:07:58.684715+0000","last_active":"2024-03-28T14:08:01.244515+0000","last_peered":"2024-03-28T14:08:01.244515+0000","last_clean":"2024-03-28T14:08:01.244515+0000","last_became_active":"2024-03-28T14:07:58.684314+0000","last_became_peered":"2024-03-28T14:07:58.684314+0000","last_unstale":"2024-03-28T14:08:01.244515+0000","last_undegraded":"2024-03-28T14:08:01.244515+0000","last_fullsized":"2024-03-28T14:08:01.244515+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:07:57.631994+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:07:57.631994+0000","last_clean_scrub_stamp":"2024-03-28T14:07:57.631994+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T17:34:09.030115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-28T14:08:01.243877+0000","last_change":"2024-03-28T14:07:59.730071+0000","last_active":"2024-03-28T14:08:01.243877+0000","last_peered":"2024-03-28T14:08:01.243877+0000","last_clean":"2024-03-28T14:08:01.243877+0000","last_became_active":"2024-03-28T14:07:59.729794+0000","last_became_peered":"2024-03-28T14:07:59.729794+0000","last_unstale":"2024-03-28T14:08:01.243877+0000","last_undegraded":"2024-03-28T14:08:01.243877+0000","last_fullsized":"2024-03-28T14:08:01.243877+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_clean_scrub_stamp":"2024-03-28T14:07:58.641775+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:38:43.102577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-28T14:08:00.680825+0000","last_change":"2024-03-28T14:07:58.704245+0000","last_active":"2024-03-28T14:08:00.680825+0000","last_peered":"2024-03-28T14:08:00.680825+0000","last_clean":"2024-03-28T14:08:00.680825+0000","last_became_active":"2024-03-28T14:07:58.703603+0000","last_became_peered":"2024-03-28T14:07:58.703603+0000","last_unstale":"2024-03-28T14:08:00.680825+0000","last_undegraded":"2024-03-28T14:08:00.680825+0000","last_fullsized":"2024-03-28T14:08:00.680825+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:07:57.631994+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:07:57.631994+0000","last_clean_scrub_stamp":"2024-03-28T14:07:57.631994+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T17:35:00.153938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-28T14:08:00.834163+0000","last_change":"2024-03-28T14:07:59.718822+0000","last_active":"2024-03-28T14:08:00.834163+0000","last_peered":"2024-03-28T14:08:00.834163+0000","last_clean":"2024-03-28T14:08:00.834163+0000","last_became_active":"2024-03-28T14:07:59.718058+0000","last_became_peered":"2024-03-28T14:07:59.718058+0000","last_unstale":"2024-03-28T14:08:00.834163+0000","last_undegraded":"2024-03-28T14:08:00.834163+0000","last_fullsized":"2024-03-28T14:08:00.834163+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_clean_scrub_stamp":"2024-03-28T14:07:58.641775+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:10:31.257059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-28T14:08:00.702420+0000","last_change":"2024-03-28T14:07:58.689870+0000","last_active":"2024-03-28T14:08:00.702420+0000","last_peered":"2024-03-28T14:08:00.702420+0000","last_clean":"2024-03-28T14:08:00.702420+0000","last_became_active":"2024-03-28T14:07:58.689121+0000","last_became_peered":"2024-03-28T14:07:58.689121+0000","last_unstale":"2024-03-28T14:08:00.702420+0000","last_undegraded":"2024-03-28T14:08:00.702420+0000","last_fullsized":"2024-03-28T14:08:00.702420+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:07:57.631994+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:07:57.631994+0000","last_clean_scrub_stamp":"2024-03-28T14:07:57.631994+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:35:03.722392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-28T14:08:00.702531+0000","last_change":"2024-03-28T14:07:59.712485+0000","last_active":"2024-03-28T14:08:00.702531+0000","last_peered":"2024-03-28T14:08:00.702531+0000","last_clean":"2024-03-28T14:08:00.702531+0000","last_became_active":"2024-03-28T14:07:59.704056+0000","last_became_peered":"2024-03-28T14:07:59.704056+0000","last_unstale":"2024-03-28T14:08:00.702531+0000","last_undegraded":"2024-03-28T14:08:00.702531+0000","last_fullsized":"2024-03-28T14:08:00.702531+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_clean_scrub_stamp":"2024-03-28T14:07:58.641775+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:10:27.248645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"20'8","reported_seq":22,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-28T14:08:01.242042+0000","last_change":"2024-03-28T14:07:58.667252+0000","last_active":"2024-03-28T14:08:01.242042+0000","last_peered":"2024-03-28T14:08:01.242042+0000","last_clean":"2024-03-28T14:08:01.242042+0000","last_became_active":"2024-03-28T14:07:58.666837+0000","last_became_peered":"2024-03-28T14:07:58.666837+0000","last_unstale":"2024-03-28T14:08:01.242042+0000","last_undegraded":"2024-03-28T14:08:01.242042+0000","last_fullsized":"2024-03-28T14:08:01.242042+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:07:57.631994+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:07:57.631994+0000","last_clean_scrub_stamp":"2024-03-28T14:07:57.631994+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:30:19.088624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-28T14:08:00.833905+0000","last_change":"2024-03-28T14:07:59.719433+0000","last_active":"2024-03-28T14:08:00.833905+0000","last_peered":"2024-03-28T14:08:00.833905+0000","last_clean":"2024-03-28T14:08:00.833905+0000","last_became_active":"2024-03-28T14:07:59.718644+0000","last_became_peered":"2024-03-28T14:07:59.718644+0000","last_unstale":"2024-03-28T14:08:00.833905+0000","last_undegraded":"2024-03-28T14:08:00.833905+0000","last_fullsized":"2024-03-28T14:08:00.833905+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_clean_scrub_stamp":"2024-03-28T14:07:58.641775+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T17:46:06.253111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"20'1","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-28T14:08:01.243993+0000","last_change":"2024-03-28T14:07:58.674598+0000","last_active":"2024-03-28T14:08:01.243993+0000","last_peered":"2024-03-28T14:08:01.243993+0000","last_clean":"2024-03-28T14:08:01.243993+0000","last_became_active":"2024-03-28T14:07:58.673894+0000","last_became_peered":"2024-03-28T14:07:58.673894+0000","last_unstale":"2024-03-28T14:08:01.243993+0000","last_undegraded":"2024-03-28T14:08:01.243993+0000","last_fullsized":"2024-03-28T14:08:01.243993+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:07:57.631994+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:07:57.631994+0000","last_clean_scrub_stamp":"2024-03-28T14:07:57.631994+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:32:35.921987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-28T14:08:00.834711+0000","last_change":"2024-03-28T14:07:59.725643+0000","last_active":"2024-03-28T14:08:00.834711+0000","last_peered":"2024-03-28T14:08:00.834711+0000","last_clean":"2024-03-28T14:08:00.834711+0000","last_became_active":"2024-03-28T14:07:59.725019+0000","last_became_peered":"2024-03-28T14:07:59.725019+0000","last_unstale":"2024-03-28T14:08:00.834711+0000","last_undegraded":"2024-03-28T14:08:00.834711+0000","last_fullsized":"2024-03-28T14:08:00.834711+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_clean_scrub_stamp":"2024-03-28T14:07:58.641775+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:02:48.017536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"20'2","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-28T14:08:01.242483+0000","last_change":"2024-03-28T14:07:58.663444+0000","last_active":"2024-03-28T14:08:01.242483+0000","last_peered":"2024-03-28T14:08:01.242483+0000","last_clean":"2024-03-28T14:08:01.242483+0000","last_became_active":"2024-03-28T14:07:58.663001+0000","last_became_peered":"2024-03-28T14:07:58.663001+0000","last_unstale":"2024-03-28T14:08:01.242483+0000","last_undegraded":"2024-03-28T14:08:01.242483+0000","last_fullsized":"2024-03-28T14:08:01.242483+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:07:57.631994+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:07:57.631994+0000","last_clean_scrub_stamp":"2024-03-28T14:07:57.631994+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:35:03.722392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-28T14:08:00.835294+0000","last_change":"2024-03-28T14:07:59.731950+0000","last_active":"2024-03-28T14:08:00.835294+0000","last_peered":"2024-03-28T14:08:00.835294+0000","last_clean":"2024-03-28T14:08:00.835294+0000","last_became_active":"2024-03-28T14:07:59.731208+0000","last_became_peered":"2024-03-28T14:07:59.731208+0000","last_unstale":"2024-03-28T14:08:00.835294+0000","last_undegraded":"2024-03-28T14:08:00.835294+0000","last_fullsized":"2024-03-28T14:08:00.835294+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_clean_scrub_stamp":"2024-03-28T14:07:58.641775+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T14:17:34.720686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"20'1","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-28T14:08:00.696647+0000","last_change":"2024-03-28T14:07:58.684273+0000","last_active":"2024-03-28T14:08:00.696647+0000","last_peered":"2024-03-28T14:08:00.696647+0000","last_clean":"2024-03-28T14:08:00.696647+0000","last_became_active":"2024-03-28T14:07:58.683886+0000","last_became_peered":"2024-03-28T14:07:58.683886+0000","last_unstale":"2024-03-28T14:08:00.696647+0000","last_undegraded":"2024-03-28T14:08:00.696647+0000","last_fullsized":"2024-03-28T14:08:00.696647+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:07:57.631994+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:07:57.631994+0000","last_clean_scrub_stamp":"2024-03-28T14:07:57.631994+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T15:40:00.657964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-28T14:08:00.681505+0000","last_change":"2024-03-28T14:07:59.739696+0000","last_active":"2024-03-28T14:08:00.681505+0000","last_peered":"2024-03-28T14:08:00.681505+0000","last_clean":"2024-03-28T14:08:00.681505+0000","last_became_active":"2024-03-28T14:07:59.734736+0000","last_became_peered":"2024-03-28T14:07:59.734736+0000","last_unstale":"2024-03-28T14:08:00.681505+0000","last_undegraded":"2024-03-28T14:08:00.681505+0000","last_fullsized":"2024-03-28T14:08:00.681505+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_clean_scrub_stamp":"2024-03-28T14:07:58.641775+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:51:23.322586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-28T14:08:01.242313+0000","last_change":"2024-03-28T14:07:58.704650+0000","last_active":"2024-03-28T14:08:01.242313+0000","last_peered":"2024-03-28T14:08:01.242313+0000","last_clean":"2024-03-28T14:08:01.242313+0000","last_became_active":"2024-03-28T14:07:58.704305+0000","last_became_peered":"2024-03-28T14:07:58.704305+0000","last_unstale":"2024-03-28T14:08:01.242313+0000","last_undegraded":"2024-03-28T14:08:01.242313+0000","last_fullsized":"2024-03-28T14:08:01.242313+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:07:57.631994+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:07:57.631994+0000","last_clean_scrub_stamp":"2024-03-28T14:07:57.631994+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T17:35:00.153938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-28T14:08:00.834345+0000","last_change":"2024-03-28T14:07:59.745242+0000","last_active":"2024-03-28T14:08:00.834345+0000","last_peered":"2024-03-28T14:08:00.834345+0000","last_clean":"2024-03-28T14:08:00.834345+0000","last_became_active":"2024-03-28T14:07:59.744734+0000","last_became_peered":"2024-03-28T14:07:59.744734+0000","last_unstale":"2024-03-28T14:08:00.834345+0000","last_undegraded":"2024-03-28T14:08:00.834345+0000","last_fullsized":"2024-03-28T14:08:00.834345+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_clean_scrub_stamp":"2024-03-28T14:07:58.641775+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:49:58.161511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":170,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-28T14:08:00.833744+0000","last_change":"2024-03-28T14:07:49.234448+0000","last_active":"2024-03-28T14:08:00.833744+0000","last_peered":"2024-03-28T14:08:00.833744+0000","last_clean":"2024-03-28T14:08:00.833744+0000","last_became_active":"2024-03-28T14:07:49.233895+0000","last_became_peered":"2024-03-28T14:07:49.233895+0000","last_unstale":"2024-03-28T14:08:00.833744+0000","last_undegraded":"2024-03-28T14:08:00.833744+0000","last_fullsized":"2024-03-28T14:08:00.833744+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:07:48.069157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:07:48.069157+0000","last_clean_scrub_stamp":"2024-03-28T14:07:48.069157+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:11:39.977557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-28T14:08:00.680569+0000","last_change":"2024-03-28T14:07:58.699326+0000","last_active":"2024-03-28T14:08:00.680569+0000","last_peered":"2024-03-28T14:08:00.680569+0000","last_clean":"2024-03-28T14:08:00.680569+0000","last_became_active":"2024-03-28T14:07:58.698258+0000","last_became_peered":"2024-03-28T14:07:58.698258+0000","last_unstale":"2024-03-28T14:08:00.680569+0000","last_undegraded":"2024-03-28T14:08:00.680569+0000","last_fullsized":"2024-03-28T14:08:00.680569+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:07:57.631994+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:07:57.631994+0000","last_clean_scrub_stamp":"2024-03-28T14:07:57.631994+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T18:00:55.131543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-28T14:08:00.681381+0000","last_change":"2024-03-28T14:07:59.744015+0000","last_active":"2024-03-28T14:08:00.681381+0000","last_peered":"2024-03-28T14:08:00.681381+0000","last_clean":"2024-03-28T14:08:00.681381+0000","last_became_active":"2024-03-28T14:07:59.743723+0000","last_became_peered":"2024-03-28T14:07:59.743723+0000","last_unstale":"2024-03-28T14:08:00.681381+0000","last_undegraded":"2024-03-28T14:08:00.681381+0000","last_fullsized":"2024-03-28T14:08:00.681381+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_clean_scrub_stamp":"2024-03-28T14:07:58.641775+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T16:47:04.317583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-28T14:08:00.833592+0000","last_change":"2024-03-28T14:07:58.688278+0000","last_active":"2024-03-28T14:08:00.833592+0000","last_peered":"2024-03-28T14:08:00.833592+0000","last_clean":"2024-03-28T14:08:00.833592+0000","last_became_active":"2024-03-28T14:07:58.687442+0000","last_became_peered":"2024-03-28T14:07:58.687442+0000","last_unstale":"2024-03-28T14:08:00.833592+0000","last_undegraded":"2024-03-28T14:08:00.833592+0000","last_fullsized":"2024-03-28T14:08:00.833592+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:07:57.631994+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:07:57.631994+0000","last_clean_scrub_stamp":"2024-03-28T14:07:57.631994+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T17:35:00.153938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-28T14:08:01.243711+0000","last_change":"2024-03-28T14:07:59.715990+0000","last_active":"2024-03-28T14:08:01.243711+0000","last_peered":"2024-03-28T14:08:01.243711+0000","last_clean":"2024-03-28T14:08:01.243711+0000","last_became_active":"2024-03-28T14:07:59.715627+0000","last_became_peered":"2024-03-28T14:07:59.715627+0000","last_unstale":"2024-03-28T14:08:01.243711+0000","last_undegraded":"2024-03-28T14:08:01.243711+0000","last_fullsized":"2024-03-28T14:08:01.243711+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_clean_scrub_stamp":"2024-03-28T14:07:58.641775+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T18:46:53.473825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-28T14:08:01.242294+0000","last_change":"2024-03-28T14:07:58.663475+0000","last_active":"2024-03-28T14:08:01.242294+0000","last_peered":"2024-03-28T14:08:01.242294+0000","last_clean":"2024-03-28T14:08:01.242294+0000","last_became_active":"2024-03-28T14:07:58.662967+0000","last_became_peered":"2024-03-28T14:07:58.662967+0000","last_unstale":"2024-03-28T14:08:01.242294+0000","last_undegraded":"2024-03-28T14:08:01.242294+0000","last_fullsized":"2024-03-28T14:08:01.242294+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:07:57.631994+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:07:57.631994+0000","last_clean_scrub_stamp":"2024-03-28T14:07:57.631994+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T18:00:55.131543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-28T14:08:00.696122+0000","last_change":"2024-03-28T14:07:59.693645+0000","last_active":"2024-03-28T14:08:00.696122+0000","last_peered":"2024-03-28T14:08:00.696122+0000","last_clean":"2024-03-28T14:08:00.696122+0000","last_became_active":"2024-03-28T14:07:59.693346+0000","last_became_peered":"2024-03-28T14:07:59.693346+0000","last_unstale":"2024-03-28T14:08:00.696122+0000","last_undegraded":"2024-03-28T14:08:00.696122+0000","last_fullsized":"2024-03-28T14:08:00.696122+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_clean_scrub_stamp":"2024-03-28T14:07:58.641775+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:57:11.908191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-28T14:08:00.680856+0000","last_change":"2024-03-28T14:07:58.702868+0000","last_active":"2024-03-28T14:08:00.680856+0000","last_peered":"2024-03-28T14:08:00.680856+0000","last_clean":"2024-03-28T14:08:00.680856+0000","last_became_active":"2024-03-28T14:07:58.700099+0000","last_became_peered":"2024-03-28T14:07:58.700099+0000","last_unstale":"2024-03-28T14:08:00.680856+0000","last_undegraded":"2024-03-28T14:08:00.680856+0000","last_fullsized":"2024-03-28T14:08:00.680856+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:07:57.631994+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:07:57.631994+0000","last_clean_scrub_stamp":"2024-03-28T14:07:57.631994+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:30:19.088624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-28T14:08:01.243699+0000","last_change":"2024-03-28T14:07:59.707739+0000","last_active":"2024-03-28T14:08:01.243699+0000","last_peered":"2024-03-28T14:08:01.243699+0000","last_clean":"2024-03-28T14:08:01.243699+0000","last_became_active":"2024-03-28T14:07:59.705252+0000","last_became_peered":"2024-03-28T14:07:59.705252+0000","last_unstale":"2024-03-28T14:08:01.243699+0000","last_undegraded":"2024-03-28T14:08:01.243699+0000","last_fullsized":"2024-03-28T14:08:01.243699+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_clean_scrub_stamp":"2024-03-28T14:07:58.641775+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:09:39.273902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-28T14:08:00.696496+0000","last_change":"2024-03-28T14:07:58.669940+0000","last_active":"2024-03-28T14:08:00.696496+0000","last_peered":"2024-03-28T14:08:00.696496+0000","last_clean":"2024-03-28T14:08:00.696496+0000","last_became_active":"2024-03-28T14:07:58.669568+0000","last_became_peered":"2024-03-28T14:07:58.669568+0000","last_unstale":"2024-03-28T14:08:00.696496+0000","last_undegraded":"2024-03-28T14:08:00.696496+0000","last_fullsized":"2024-03-28T14:08:00.696496+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:07:57.631994+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:07:57.631994+0000","last_clean_scrub_stamp":"2024-03-28T14:07:57.631994+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:09:38.264121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-28T14:08:00.681619+0000","last_change":"2024-03-28T14:07:59.742899+0000","last_active":"2024-03-28T14:08:00.681619+0000","last_peered":"2024-03-28T14:08:00.681619+0000","last_clean":"2024-03-28T14:08:00.681619+0000","last_became_active":"2024-03-28T14:07:59.740075+0000","last_became_peered":"2024-03-28T14:07:59.740075+0000","last_unstale":"2024-03-28T14:08:00.681619+0000","last_undegraded":"2024-03-28T14:08:00.681619+0000","last_fullsized":"2024-03-28T14:08:00.681619+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_clean_scrub_stamp":"2024-03-28T14:07:58.641775+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:59:25.862480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-28T14:08:00.834925+0000","last_change":"2024-03-28T14:07:58.687874+0000","last_active":"2024-03-28T14:08:00.834925+0000","last_peered":"2024-03-28T14:08:00.834925+0000","last_clean":"2024-03-28T14:08:00.834925+0000","last_became_active":"2024-03-28T14:07:58.686656+0000","last_became_peered":"2024-03-28T14:07:58.686656+0000","last_unstale":"2024-03-28T14:08:00.834925+0000","last_undegraded":"2024-03-28T14:08:00.834925+0000","last_fullsized":"2024-03-28T14:08:00.834925+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:07:57.631994+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:07:57.631994+0000","last_clean_scrub_stamp":"2024-03-28T14:07:57.631994+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T15:40:00.657964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-28T14:08:00.696528+0000","last_change":"2024-03-28T14:07:59.700465+0000","last_active":"2024-03-28T14:08:00.696528+0000","last_peered":"2024-03-28T14:08:00.696528+0000","last_clean":"2024-03-28T14:08:00.696528+0000","last_became_active":"2024-03-28T14:07:59.699705+0000","last_became_peered":"2024-03-28T14:07:59.699705+0000","last_unstale":"2024-03-28T14:08:00.696528+0000","last_undegraded":"2024-03-28T14:08:00.696528+0000","last_fullsized":"2024-03-28T14:08:00.696528+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_clean_scrub_stamp":"2024-03-28T14:07:58.641775+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:25:39.647896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"21'1","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-28T14:08:00.700925+0000","last_change":"2024-03-28T14:07:58.681340+0000","last_active":"2024-03-28T14:08:00.700925+0000","last_peered":"2024-03-28T14:08:00.700925+0000","last_clean":"2024-03-28T14:08:00.700925+0000","last_became_active":"2024-03-28T14:07:58.681006+0000","last_became_peered":"2024-03-28T14:07:58.681006+0000","last_unstale":"2024-03-28T14:08:00.700925+0000","last_undegraded":"2024-03-28T14:08:00.700925+0000","last_fullsized":"2024-03-28T14:08:00.700925+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:07:57.631994+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:07:57.631994+0000","last_clean_scrub_stamp":"2024-03-28T14:07:57.631994+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:04:40.185372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-28T14:08:00.681402+0000","last_change":"2024-03-28T14:07:59.727008+0000","last_active":"2024-03-28T14:08:00.681402+0000","last_peered":"2024-03-28T14:08:00.681402+0000","last_clean":"2024-03-28T14:08:00.681402+0000","last_became_active":"2024-03-28T14:07:59.726702+0000","last_became_peered":"2024-03-28T14:07:59.726702+0000","last_unstale":"2024-03-28T14:08:00.681402+0000","last_undegraded":"2024-03-28T14:08:00.681402+0000","last_fullsized":"2024-03-28T14:08:00.681402+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_clean_scrub_stamp":"2024-03-28T14:07:58.641775+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T17:46:06.253111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"20'2","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-28T14:08:00.703440+0000","last_change":"2024-03-28T14:07:58.685500+0000","last_active":"2024-03-28T14:08:00.703440+0000","last_peered":"2024-03-28T14:08:00.703440+0000","last_clean":"2024-03-28T14:08:00.703440+0000","last_became_active":"2024-03-28T14:07:58.684781+0000","last_became_peered":"2024-03-28T14:07:58.684781+0000","last_unstale":"2024-03-28T14:08:00.703440+0000","last_undegraded":"2024-03-28T14:08:00.703440+0000","last_fullsized":"2024-03-28T14:08:00.703440+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:07:57.631994+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:07:57.631994+0000","last_clean_scrub_stamp":"2024-03-28T14:07:57.631994+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:09:38.264121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-28T14:08:01.244918+0000","last_change":"2024-03-28T14:07:59.707407+0000","last_active":"2024-03-28T14:08:01.244918+0000","last_peered":"2024-03-28T14:08:01.244918+0000","last_clean":"2024-03-28T14:08:01.244918+0000","last_became_active":"2024-03-28T14:07:59.707103+0000","last_became_peered":"2024-03-28T14:07:59.707103+0000","last_unstale":"2024-03-28T14:08:01.244918+0000","last_undegraded":"2024-03-28T14:08:01.244918+0000","last_fullsized":"2024-03-28T14:08:01.244918+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_clean_scrub_stamp":"2024-03-28T14:07:58.641775+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:37:29.402787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-28T14:08:00.833570+0000","last_change":"2024-03-28T14:07:58.682456+0000","last_active":"2024-03-28T14:08:00.833570+0000","last_peered":"2024-03-28T14:08:00.833570+0000","last_clean":"2024-03-28T14:08:00.833570+0000","last_became_active":"2024-03-28T14:07:58.678987+0000","last_became_peered":"2024-03-28T14:07:58.678987+0000","last_unstale":"2024-03-28T14:08:00.833570+0000","last_undegraded":"2024-03-28T14:08:00.833570+0000","last_fullsized":"2024-03-28T14:08:00.833570+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:07:57.631994+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:07:57.631994+0000","last_clean_scrub_stamp":"2024-03-28T14:07:57.631994+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:30:19.088624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-28T14:08:01.243764+0000","last_change":"2024-03-28T14:07:59.697567+0000","last_active":"2024-03-28T14:08:01.243764+0000","last_peered":"2024-03-28T14:08:01.243764+0000","last_clean":"2024-03-28T14:08:01.243764+0000","last_became_active":"2024-03-28T14:07:59.693975+0000","last_became_peered":"2024-03-28T14:07:59.693975+0000","last_unstale":"2024-03-28T14:08:01.243764+0000","last_undegraded":"2024-03-28T14:08:01.243764+0000","last_fullsized":"2024-03-28T14:08:01.243764+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_clean_scrub_stamp":"2024-03-28T14:07:58.641775+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:04:41.195153+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-28T14:08:00.696246+0000","last_change":"2024-03-28T14:07:58.680918+0000","last_active":"2024-03-28T14:08:00.696246+0000","last_peered":"2024-03-28T14:08:00.696246+0000","last_clean":"2024-03-28T14:08:00.696246+0000","last_became_active":"2024-03-28T14:07:58.680548+0000","last_became_peered":"2024-03-28T14:07:58.680548+0000","last_unstale":"2024-03-28T14:08:00.696246+0000","last_undegraded":"2024-03-28T14:08:00.696246+0000","last_fullsized":"2024-03-28T14:08:00.696246+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:07:57.631994+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:07:57.631994+0000","last_clean_scrub_stamp":"2024-03-28T14:07:57.631994+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:30:19.088624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-28T14:08:00.680594+0000","last_change":"2024-03-28T14:07:59.702172+0000","last_active":"2024-03-28T14:08:00.680594+0000","last_peered":"2024-03-28T14:08:00.680594+0000","last_clean":"2024-03-28T14:08:00.680594+0000","last_became_active":"2024-03-28T14:07:59.701709+0000","last_became_peered":"2024-03-28T14:07:59.701709+0000","last_unstale":"2024-03-28T14:08:00.680594+0000","last_undegraded":"2024-03-28T14:08:00.680594+0000","last_fullsized":"2024-03-28T14:08:00.680594+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_clean_scrub_stamp":"2024-03-28T14:07:58.641775+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:57:11.908191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-28T14:08:00.696422+0000","last_change":"2024-03-28T14:07:58.672681+0000","last_active":"2024-03-28T14:08:00.696422+0000","last_peered":"2024-03-28T14:08:00.696422+0000","last_clean":"2024-03-28T14:08:00.696422+0000","last_became_active":"2024-03-28T14:07:58.672094+0000","last_became_peered":"2024-03-28T14:07:58.672094+0000","last_unstale":"2024-03-28T14:08:00.696422+0000","last_undegraded":"2024-03-28T14:08:00.696422+0000","last_fullsized":"2024-03-28T14:08:00.696422+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:07:57.631994+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:07:57.631994+0000","last_clean_scrub_stamp":"2024-03-28T14:07:57.631994+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T17:35:00.153938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-28T14:08:00.681203+0000","last_change":"2024-03-28T14:07:59.705251+0000","last_active":"2024-03-28T14:08:00.681203+0000","last_peered":"2024-03-28T14:08:00.681203+0000","last_clean":"2024-03-28T14:08:00.681203+0000","last_became_active":"2024-03-28T14:07:59.703366+0000","last_became_peered":"2024-03-28T14:07:59.703366+0000","last_unstale":"2024-03-28T14:08:00.681203+0000","last_undegraded":"2024-03-28T14:08:00.681203+0000","last_fullsized":"2024-03-28T14:08:00.681203+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_clean_scrub_stamp":"2024-03-28T14:07:58.641775+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:41:27.409736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"20'2","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-28T14:08:00.834672+0000","last_change":"2024-03-28T14:07:58.688811+0000","last_active":"2024-03-28T14:08:00.834672+0000","last_peered":"2024-03-28T14:08:00.834672+0000","last_clean":"2024-03-28T14:08:00.834672+0000","last_became_active":"2024-03-28T14:07:58.688054+0000","last_became_peered":"2024-03-28T14:07:58.688054+0000","last_unstale":"2024-03-28T14:08:00.834672+0000","last_undegraded":"2024-03-28T14:08:00.834672+0000","last_fullsized":"2024-03-28T14:08:00.834672+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:07:57.631994+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:07:57.631994+0000","last_clean_scrub_stamp":"2024-03-28T14:07:57.631994+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:04:40.185372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-28T14:08:00.702555+0000","last_change":"2024-03-28T14:07:59.709224+0000","last_active":"2024-03-28T14:08:00.702555+0000","last_peered":"2024-03-28T14:08:00.702555+0000","last_clean":"2024-03-28T14:08:00.702555+0000","last_became_active":"2024-03-28T14:07:59.704202+0000","last_became_peered":"2024-03-28T14:07:59.704202+0000","last_unstale":"2024-03-28T14:08:00.702555+0000","last_undegraded":"2024-03-28T14:08:00.702555+0000","last_fullsized":"2024-03-28T14:08:00.702555+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_clean_scrub_stamp":"2024-03-28T14:07:58.641775+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:57:11.908191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-28T14:08:00.681258+0000","last_change":"2024-03-28T14:07:59.739995+0000","last_active":"2024-03-28T14:08:00.681258+0000","last_peered":"2024-03-28T14:08:00.681258+0000","last_clean":"2024-03-28T14:08:00.681258+0000","last_became_active":"2024-03-28T14:07:59.739369+0000","last_became_peered":"2024-03-28T14:07:59.739369+0000","last_unstale":"2024-03-28T14:08:00.681258+0000","last_undegraded":"2024-03-28T14:08:00.681258+0000","last_fullsized":"2024-03-28T14:08:00.681258+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_clean_scrub_stamp":"2024-03-28T14:07:58.641775+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:34:47.572488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"20'2","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-28T14:08:00.834148+0000","last_change":"2024-03-28T14:07:58.705499+0000","last_active":"2024-03-28T14:08:00.834148+0000","last_peered":"2024-03-28T14:08:00.834148+0000","last_clean":"2024-03-28T14:08:00.834148+0000","last_became_active":"2024-03-28T14:07:58.704848+0000","last_became_peered":"2024-03-28T14:07:58.704848+0000","last_unstale":"2024-03-28T14:08:00.834148+0000","last_undegraded":"2024-03-28T14:08:00.834148+0000","last_fullsized":"2024-03-28T14:08:00.834148+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:07:57.631994+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:07:57.631994+0000","last_clean_scrub_stamp":"2024-03-28T14:07:57.631994+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:09:38.264121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-28T14:08:00.680957+0000","last_change":"2024-03-28T14:07:59.740931+0000","last_active":"2024-03-28T14:08:00.680957+0000","last_peered":"2024-03-28T14:08:00.680957+0000","last_clean":"2024-03-28T14:08:00.680957+0000","last_became_active":"2024-03-28T14:07:59.740255+0000","last_became_peered":"2024-03-28T14:07:59.740255+0000","last_unstale":"2024-03-28T14:08:00.680957+0000","last_undegraded":"2024-03-28T14:08:00.680957+0000","last_fullsized":"2024-03-28T14:08:00.680957+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_clean_scrub_stamp":"2024-03-28T14:07:58.641775+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:49:58.161511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"20'2","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-28T14:08:00.834791+0000","last_change":"2024-03-28T14:07:58.693440+0000","last_active":"2024-03-28T14:08:00.834791+0000","last_peered":"2024-03-28T14:08:00.834791+0000","last_clean":"2024-03-28T14:08:00.834791+0000","last_became_active":"2024-03-28T14:07:58.692096+0000","last_became_peered":"2024-03-28T14:07:58.692096+0000","last_unstale":"2024-03-28T14:08:00.834791+0000","last_undegraded":"2024-03-28T14:08:00.834791+0000","last_fullsized":"2024-03-28T14:08:00.834791+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:07:57.631994+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:07:57.631994+0000","last_clean_scrub_stamp":"2024-03-28T14:07:57.631994+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T15:40:58.102613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-28T14:08:01.244001+0000","last_change":"2024-03-28T14:07:59.744398+0000","last_active":"2024-03-28T14:08:01.244001+0000","last_peered":"2024-03-28T14:08:01.244001+0000","last_clean":"2024-03-28T14:08:01.244001+0000","last_became_active":"2024-03-28T14:07:59.744127+0000","last_became_peered":"2024-03-28T14:07:59.744127+0000","last_unstale":"2024-03-28T14:08:01.244001+0000","last_undegraded":"2024-03-28T14:08:01.244001+0000","last_fullsized":"2024-03-28T14:08:01.244001+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_clean_scrub_stamp":"2024-03-28T14:07:58.641775+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T14:20:33.555524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"20'1","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-28T14:08:00.681112+0000","last_change":"2024-03-28T14:07:58.671168+0000","last_active":"2024-03-28T14:08:00.681112+0000","last_peered":"2024-03-28T14:08:00.681112+0000","last_clean":"2024-03-28T14:08:00.681112+0000","last_became_active":"2024-03-28T14:07:58.670691+0000","last_became_peered":"2024-03-28T14:07:58.670691+0000","last_unstale":"2024-03-28T14:08:00.681112+0000","last_undegraded":"2024-03-28T14:08:00.681112+0000","last_fullsized":"2024-03-28T14:08:00.681112+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:07:57.631994+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:07:57.631994+0000","last_clean_scrub_stamp":"2024-03-28T14:07:57.631994+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T17:34:09.030115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-28T14:08:00.680644+0000","last_change":"2024-03-28T14:07:59.727344+0000","last_active":"2024-03-28T14:08:00.680644+0000","last_peered":"2024-03-28T14:08:00.680644+0000","last_clean":"2024-03-28T14:08:00.680644+0000","last_became_active":"2024-03-28T14:07:59.727023+0000","last_became_peered":"2024-03-28T14:07:59.727023+0000","last_unstale":"2024-03-28T14:08:00.680644+0000","last_undegraded":"2024-03-28T14:08:00.680644+0000","last_fullsized":"2024-03-28T14:08:00.680644+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:07:58.641775+0000","last_clean_scrub_stamp":"2024-03-28T14:07:58.641775+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:24:30.197596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"20'4","reported_seq":18,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-28T14:08:01.245810+0000","last_change":"2024-03-28T14:07:58.676958+0000","last_active":"2024-03-28T14:08:01.245810+0000","last_peered":"2024-03-28T14:08:01.245810+0000","last_clean":"2024-03-28T14:08:01.245810+0000","last_became_active":"2024-03-28T14:07:58.676611+0000","last_became_peered":"2024-03-28T14:07:58.676611+0000","last_unstale":"2024-03-28T14:08:01.245810+0000","last_undegraded":"2024-03-28T14:08:01.245810+0000","last_fullsized":"2024-03-28T14:08:01.245810+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:07:57.631994+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:07:57.631994+0000","last_clean_scrub_stamp":"2024-03-28T14:07:57.631994+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:59:04.674426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574855,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27820,"kb_used_data":796,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344020,"statfs":{"total":96636764160,"available":96608276480,"internally_reserved":0,"allocated":815104,"data_stored":666555,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574856,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27248,"kb_used_data":224,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344592,"statfs":{"total":96636764160,"available":96608862208,"internally_reserved":0,"allocated":229376,"data_stored":77217,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574854,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27240,"kb_used_data":216,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344600,"statfs":{"total":96636764160,"available":96608870400,"internally_reserved":0,"allocated":221184,"data_stored":75695,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574855,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27248,"kb_used_data":224,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344592,"statfs":{"total":96636764160,"available":96608862208,"internally_reserved":0,"allocated":229376,"data_stored":75731,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574857,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27248,"kb_used_data":224,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344592,"statfs":{"total":96636764160,"available":96608862208,"internally_reserved":0,"allocated":229376,"data_stored":77217,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574855,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27236,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344604,"statfs":{"total":96636764160,"available":96608874496,"internally_reserved":0,"allocated":217088,"data_stored":75605,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574853,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27244,"kb_used_data":220,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344596,"statfs":{"total":96636764160,"available":96608866304,"internally_reserved":0,"allocated":225280,"data_stored":75765,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574856,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27824,"kb_used_data":804,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344016,"statfs":{"total":96636764160,"available":96608272384,"internally_reserved":0,"allocated":823296,"data_stored":666611,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-28T14:08:06.846 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-03-28T14:08:06.846 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-03-28T14:08:06.846 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2024-03-28T14:08:06.847 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2024-03-28T14:08:07.005 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:07.003+0000 7fe9586dd640 1 Processor -- start 2024-03-28T14:08:07.005 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:07.003+0000 7fe9586dd640 1 -- start start 2024-03-28T14:08:07.008 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:07.003+0000 7fe9586dd640 1 --2- >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fe95002c460 0x7fe9500aeba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:07.008 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:07.003+0000 7fe9586dd640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe95002c840 con 0x7fe9500b8820 2024-03-28T14:08:07.008 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:07.003+0000 7fe9586dd640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe9500b8d00 con 0x7fe95014a920 2024-03-28T14:08:07.008 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:07.003+0000 7fe9586dd640 1 -- --> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] -- mon_getmap magic: 0 -- 0x7fe9500b8fe0 con 0x7fe95002c460 2024-03-28T14:08:07.008 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:07.003+0000 7fe955c78640 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7fe9500b8820 0x7fe95014a210 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:41590/0 (socket says 172.21.15.104:41590) 2024-03-28T14:08:07.008 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:07.003+0000 7fe955c78640 1 -- 172.21.15.104:0/3801131025 learned_addr learned my addr 172.21.15.104:0/3801131025 (peer_addr_for_me v1:172.21.15.104:0/0) 2024-03-28T14:08:07.008 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:07.003+0000 7fe956479640 1 --2- 172.21.15.104:0/3801131025 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fe95002c460 0x7fe9500aeba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:07.008 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:07.003+0000 7fe955477640 1 -- 172.21.15.104:0/3801131025 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 634990705 0 0) 0x7fe95002c840 con 0x7fe9500b8820 2024-03-28T14:08:07.009 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:07.003+0000 7fe955477640 1 -- 172.21.15.104:0/3801131025 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe93c003530 con 0x7fe9500b8820 2024-03-28T14:08:07.009 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:07.003+0000 7fe955477640 1 -- 172.21.15.104:0/3801131025 <== mon.0 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1288281025 0 0) 0x7fe93c003530 con 0x7fe9500b8820 2024-03-28T14:08:07.009 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:07.003+0000 7fe955477640 1 -- 172.21.15.104:0/3801131025 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fe95002c460 msgr2=0x7fe9500aeba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T14:08:07.009 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:07.003+0000 7fe955477640 1 --2- 172.21.15.104:0/3801131025 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fe95002c460 0x7fe9500aeba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:07.009 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:07.003+0000 7fe955477640 1 -- 172.21.15.104:0/3801131025 >> v1:172.21.15.171:6789/0 conn(0x7fe95014a920 legacy=0x7fe950152270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:07.009 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:07.003+0000 7fe955477640 1 -- 172.21.15.104:0/3801131025 --> v1:172.21.15.104:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe9500b7590 con 0x7fe9500b8820 2024-03-28T14:08:07.009 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:07.003+0000 7fe955477640 1 -- 172.21.15.104:0/3801131025 <== mon.0 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3745580849 0 0) 0x7fe940001680 con 0x7fe9500b8820 2024-03-28T14:08:07.009 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:07.003+0000 7fe955477640 1 -- 172.21.15.104:0/3801131025 >> v1:172.21.15.104:6789/0 conn(0x7fe9500b8820 legacy=0x7fe95014a210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:07.009 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:07.003+0000 7fe955477640 1 --2- 172.21.15.104:0/3801131025 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe93c0039f0 0x7fe93c003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:07.009 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:07.003+0000 7fe955477640 1 --2- 172.21.15.104:0/3801131025 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fe95002c460 0x7fe93c004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:07.009 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:07.003+0000 7fe955477640 1 --2- 172.21.15.104:0/3801131025 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fe93c004d50 0x7fe93c0051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:07.009 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:07.003+0000 7fe955477640 1 -- 172.21.15.104:0/3801131025 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7fe9500b8fe0 con 0x7fe93c0039f0 2024-03-28T14:08:07.009 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:07.003+0000 7fe955477640 1 -- 172.21.15.104:0/3801131025 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7fe93c00bb90 con 0x7fe93c004d50 2024-03-28T14:08:07.009 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:07.003+0000 7fe955477640 1 -- 172.21.15.104:0/3801131025 --> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] -- mon_getmap magic: 0 -- 0x7fe93c00bd90 con 0x7fe95002c460 2024-03-28T14:08:07.010 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:07.003+0000 7fe956479640 1 --2- 172.21.15.104:0/3801131025 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe93c0039f0 0x7fe93c003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:07.010 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:07.007+0000 7fe955c78640 1 --2- 172.21.15.104:0/3801131025 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fe95002c460 0x7fe93c004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:07.010 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:07.007+0000 7fe956c7a640 1 --2- 172.21.15.104:0/3801131025 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fe93c004d50 0x7fe93c0051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:07.010 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:07.007+0000 7fe956479640 1 -- 172.21.15.104:0/3801131025 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fe95002c460 msgr2=0x7fe93c004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T14:08:07.010 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:07.007+0000 7fe956479640 1 --2- 172.21.15.104:0/3801131025 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fe95002c460 0x7fe93c004810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:07.010 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:07.007+0000 7fe956479640 1 -- 172.21.15.104:0/3801131025 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fe93c004d50 msgr2=0x7fe93c0051d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T14:08:07.010 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:07.007+0000 7fe956479640 1 --2- 172.21.15.104:0/3801131025 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fe93c004d50 0x7fe93c0051d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:07.010 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:07.007+0000 7fe956479640 1 -- 172.21.15.104:0/3801131025 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe93c003530 con 0x7fe93c0039f0 2024-03-28T14:08:07.010 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:07.007+0000 7fe955c78640 1 --2- 172.21.15.104:0/3801131025 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fe95002c460 0x7fe93c004810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T14:08:07.010 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:07.007+0000 7fe956c7a640 1 --2- 172.21.15.104:0/3801131025 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fe93c004d50 0x7fe93c0051d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T14:08:07.013 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:07.007+0000 7fe956479640 1 --2- 172.21.15.104:0/3801131025 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe93c0039f0 0x7fe93c003e10 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fe948009fa0 tx=0x7fe94801cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=71245e9a50ee2476 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:08:07.013 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:07.007+0000 7fe955477640 1 -- 172.21.15.104:0/3801131025 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9480b2070 con 0x7fe93c0039f0 2024-03-28T14:08:07.014 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:07.007+0000 7fe955477640 1 -- 172.21.15.104:0/3801131025 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe9480a5af0 con 0x7fe93c0039f0 2024-03-28T14:08:07.014 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:07.007+0000 7fe955477640 1 -- 172.21.15.104:0/3801131025 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe948009bf0 con 0x7fe93c0039f0 2024-03-28T14:08:07.014 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:07.007+0000 7fe9586dd640 1 -- 172.21.15.104:0/3801131025 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe93c0039f0 msgr2=0x7fe93c003e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:07.014 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:07.007+0000 7fe9586dd640 1 --2- 172.21.15.104:0/3801131025 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe93c0039f0 0x7fe93c003e10 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fe948009fa0 tx=0x7fe94801cc40 comp rx=0 tx=0).stop 2024-03-28T14:08:07.014 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:07.007+0000 7fe956c7a640 1 -- 172.21.15.104:0/3801131025 reap_dead start 2024-03-28T14:08:07.014 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:07.007+0000 7fe9586dd640 1 -- 172.21.15.104:0/3801131025 shutdown_connections 2024-03-28T14:08:07.014 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:07.007+0000 7fe9586dd640 1 -- 172.21.15.104:0/3801131025 >> 172.21.15.104:0/3801131025 conn(0x7fe9500b9660 msgr2=0x7fe9500b9a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T14:08:07.014 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:07.007+0000 7fe9586dd640 1 -- 172.21.15.104:0/3801131025 shutdown_connections 2024-03-28T14:08:07.014 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:07.007+0000 7fe9586dd640 1 -- 172.21.15.104:0/3801131025 wait complete. 2024-03-28T14:08:07.014 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:07.011+0000 7fe9586dd640 1 Processor -- start 2024-03-28T14:08:07.014 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:07.011+0000 7fe9586dd640 1 -- start start 2024-03-28T14:08:07.014 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:07.011+0000 7fe9586dd640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fe93c0039f0 0x7fe950146050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:07.014 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:07.011+0000 7fe9586dd640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe950149340 0x7fe9501e35b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:07.015 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:07.011+0000 7fe9586dd640 1 --2- >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fe95013f1f0 0x7fe95013f670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:07.015 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:07.011+0000 7fe9586dd640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7fe950153d20 con 0x7fe950149340 2024-03-28T14:08:07.015 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:07.011+0000 7fe9586dd640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7fe950153ba0 con 0x7fe93c0039f0 2024-03-28T14:08:07.015 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:07.011+0000 7fe9586dd640 1 -- --> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] -- mon_getmap magic: 0 -- 0x7fe950153ea0 con 0x7fe95013f1f0 2024-03-28T14:08:07.015 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:07.011+0000 7fe955c78640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe950149340 0x7fe9501e35b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:07.015 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:07.011+0000 7fe955c78640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe950149340 0x7fe9501e35b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:34702/0 (socket says 172.21.15.104:34702) 2024-03-28T14:08:07.015 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:07.011+0000 7fe955c78640 1 -- 172.21.15.104:0/3304984543 learned_addr learned my addr 172.21.15.104:0/3304984543 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-03-28T14:08:07.015 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:07.011+0000 7fe956479640 1 --2- 172.21.15.104:0/3304984543 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fe93c0039f0 0x7fe950146050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:07.015 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:07.011+0000 7fe956c7a640 1 --2- 172.21.15.104:0/3304984543 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fe95013f1f0 0x7fe95013f670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:07.015 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:07.011+0000 7fe955c78640 1 -- 172.21.15.104:0/3304984543 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fe95013f1f0 msgr2=0x7fe95013f670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:07.015 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:07.011+0000 7fe955c78640 1 --2- 172.21.15.104:0/3304984543 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fe95013f1f0 0x7fe95013f670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:07.015 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:07.011+0000 7fe955c78640 1 -- 172.21.15.104:0/3304984543 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fe93c0039f0 msgr2=0x7fe950146050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:07.015 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:07.011+0000 7fe955c78640 1 --2- 172.21.15.104:0/3304984543 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fe93c0039f0 0x7fe950146050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:07.015 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:07.011+0000 7fe955c78640 1 -- 172.21.15.104:0/3304984543 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe9501e3af0 con 0x7fe950149340 2024-03-28T14:08:07.016 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:07.011+0000 7fe955c78640 1 --2- 172.21.15.104:0/3304984543 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe950149340 0x7fe9501e35b0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fe94000e870 tx=0x7fe94000ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:08:07.016 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:07.011+0000 7fe946ffd640 1 -- 172.21.15.104:0/3304984543 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe94000c8d0 con 0x7fe950149340 2024-03-28T14:08:07.016 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:07.011+0000 7fe946ffd640 1 -- 172.21.15.104:0/3304984543 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe940018070 con 0x7fe950149340 2024-03-28T14:08:07.016 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:07.011+0000 7fe9586dd640 1 -- 172.21.15.104:0/3304984543 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe950149c80 con 0x7fe950149340 2024-03-28T14:08:07.016 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:07.011+0000 7fe946ffd640 1 -- 172.21.15.104:0/3304984543 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe94000cc90 con 0x7fe950149340 2024-03-28T14:08:07.016 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:07.011+0000 7fe9586dd640 1 -- 172.21.15.104:0/3304984543 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe950099ae0 con 0x7fe950149340 2024-03-28T14:08:07.016 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:07.011+0000 7fe9586dd640 1 -- 172.21.15.104:0/3304984543 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe9500687c0 con 0x7fe950149340 2024-03-28T14:08:07.017 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:07.011+0000 7fe946ffd640 1 -- 172.21.15.104:0/3304984543 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) ==== 87803+0+0 (secure 0 0 0) 0x7fe940013420 con 0x7fe950149340 2024-03-28T14:08:07.017 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:07.011+0000 7fe946ffd640 1 --2- 172.21.15.104:0/3304984543 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7fe934067ff0 0x7fe93406a4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:07.017 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:07.011+0000 7fe946ffd640 1 -- 172.21.15.104:0/3304984543 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7fe940086110 con 0x7fe950149340 2024-03-28T14:08:07.017 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:07.015+0000 7fe956479640 1 --2- 172.21.15.104:0/3304984543 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7fe934067ff0 0x7fe93406a4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:07.017 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:07.015+0000 7fe956479640 1 --2- 172.21.15.104:0/3304984543 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7fe934067ff0 0x7fe93406a4b0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fe9480af000 tx=0x7fe948004ac0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:08:07.019 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:07.019+0000 7fe946ffd640 1 -- 172.21.15.104:0/3304984543 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179117 (secure 0 0 0) 0x7fe9400568b0 con 0x7fe950149340 2024-03-28T14:08:07.201 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:07.199+0000 7fe9586dd640 1 -- 172.21.15.104:0/3304984543 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) -- 0x7fe9500b9d70 con 0x7fe950149340 2024-03-28T14:08:07.201 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:07.199+0000 7fe946ffd640 1 -- 172.21.15.104:0/3304984543 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) ==== 72+0+46 (secure 0 0 0) 0x7fe9400561b0 con 0x7fe950149340 2024-03-28T14:08:07.201 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-03-28T14:08:07.202 INFO:teuthology.orchestra.run.smithi104.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-03-28T14:08:07.203 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:07.199+0000 7fe9586dd640 1 -- 172.21.15.104:0/3304984543 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7fe934067ff0 msgr2=0x7fe93406a4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:07.204 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:07.199+0000 7fe9586dd640 1 --2- 172.21.15.104:0/3304984543 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7fe934067ff0 0x7fe93406a4b0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fe9480af000 tx=0x7fe948004ac0 comp rx=0 tx=0).stop 2024-03-28T14:08:07.204 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:07.203+0000 7fe9586dd640 1 -- 172.21.15.104:0/3304984543 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe950149340 msgr2=0x7fe9501e35b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:07.204 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:07.203+0000 7fe9586dd640 1 --2- 172.21.15.104:0/3304984543 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe950149340 0x7fe9501e35b0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fe94000e870 tx=0x7fe94000ed30 comp rx=0 tx=0).stop 2024-03-28T14:08:07.204 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:07.203+0000 7fe9586dd640 1 -- 172.21.15.104:0/3304984543 shutdown_connections 2024-03-28T14:08:07.204 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:07.203+0000 7fe9586dd640 1 --2- 172.21.15.104:0/3304984543 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fe95013f1f0 0x7fe95013f670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:07.204 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:07.203+0000 7fe9586dd640 1 --2- 172.21.15.104:0/3304984543 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7fe934067ff0 0x7fe93406a4b0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:07.204 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:07.203+0000 7fe9586dd640 1 --2- 172.21.15.104:0/3304984543 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe950149340 0x7fe9501e35b0 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:07.204 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:07.203+0000 7fe9586dd640 1 --2- 172.21.15.104:0/3304984543 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fe93c0039f0 0x7fe950146050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:07.204 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:07.203+0000 7fe9586dd640 1 -- 172.21.15.104:0/3304984543 >> 172.21.15.104:0/3304984543 conn(0x7fe9500b9660 msgr2=0x7fe9500afe90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T14:08:07.204 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:07.203+0000 7fe9586dd640 1 -- 172.21.15.104:0/3304984543 shutdown_connections 2024-03-28T14:08:07.204 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:07.203+0000 7fe9586dd640 1 -- 172.21.15.104:0/3304984543 wait complete. 2024-03-28T14:08:07.218 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2024-03-28T14:08:07.219 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-28T14:08:07.326 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:07.323+0000 7f36bfa55640 1 Processor -- start 2024-03-28T14:08:07.327 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:07.323+0000 7f36bfa55640 1 -- start start 2024-03-28T14:08:07.329 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:07.327+0000 7f36bfa55640 1 --2- >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f36b80b4fa0 0x7f36b809aa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:07.329 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:07.327+0000 7f36bfa55640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f36b809c450 con 0x7f36b806b830 2024-03-28T14:08:07.329 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:07.327+0000 7f36bfa55640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f36b806bc10 con 0x7f36b813f4c0 2024-03-28T14:08:07.329 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:07.327+0000 7f36bfa55640 1 -- --> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] -- mon_getmap magic: 0 -- 0x7f36b8140000 con 0x7f36b80b4fa0 2024-03-28T14:08:07.329 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:07.327+0000 7f36bd7f1640 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f36b806b830 0x7f36b809c930 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:41592/0 (socket says 172.21.15.104:41592) 2024-03-28T14:08:07.329 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:07.327+0000 7f36bd7f1640 1 -- 172.21.15.104:0/2855985216 learned_addr learned my addr 172.21.15.104:0/2855985216 (peer_addr_for_me v1:172.21.15.104:0/0) 2024-03-28T14:08:07.329 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:07.327+0000 7f36bcff0640 1 --2- 172.21.15.104:0/2855985216 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f36b80b4fa0 0x7f36b809aa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:07.330 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:07.327+0000 7f36affff640 1 -- 172.21.15.104:0/2855985216 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3932363614 0 0) 0x7f36b809c450 con 0x7f36b806b830 2024-03-28T14:08:07.330 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:07.327+0000 7f36affff640 1 -- 172.21.15.104:0/2855985216 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f36a4003530 con 0x7f36b806b830 2024-03-28T14:08:07.330 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:07.327+0000 7f36affff640 1 -- 172.21.15.104:0/2855985216 <== mon.0 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3202029389 0 0) 0x7f36a4003530 con 0x7f36b806b830 2024-03-28T14:08:07.330 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:07.327+0000 7f36affff640 1 -- 172.21.15.104:0/2855985216 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f36b80b4fa0 msgr2=0x7f36b809aa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T14:08:07.330 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:07.327+0000 7f36affff640 1 --2- 172.21.15.104:0/2855985216 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f36b80b4fa0 0x7f36b809aa10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:07.330 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:07.327+0000 7f36affff640 1 -- 172.21.15.104:0/2855985216 >> v1:172.21.15.171:6789/0 conn(0x7f36b813f4c0 legacy=0x7f36b813f8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:07.330 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:07.327+0000 7f36affff640 1 -- 172.21.15.104:0/2855985216 --> v1:172.21.15.104:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f36b8072ea0 con 0x7f36b806b830 2024-03-28T14:08:07.330 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:07.327+0000 7f36affff640 1 -- 172.21.15.104:0/2855985216 <== mon.0 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3745580849 0 0) 0x7f36b4001680 con 0x7f36b806b830 2024-03-28T14:08:07.330 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:07.327+0000 7f36affff640 1 -- 172.21.15.104:0/2855985216 >> v1:172.21.15.104:6789/0 conn(0x7f36b806b830 legacy=0x7f36b809c930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:07.330 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:07.327+0000 7f36affff640 1 --2- 172.21.15.104:0/2855985216 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f36a40039f0 0x7f36a4003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:07.330 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:07.327+0000 7f36affff640 1 --2- 172.21.15.104:0/2855985216 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f36a4004810 0x7f36a4004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:07.330 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:07.327+0000 7f36affff640 1 --2- 172.21.15.104:0/2855985216 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f36b80b4fa0 0x7f36a4005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:07.330 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:07.327+0000 7f36affff640 1 -- 172.21.15.104:0/2855985216 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f36b8140000 con 0x7f36a40039f0 2024-03-28T14:08:07.330 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:07.327+0000 7f36affff640 1 -- 172.21.15.104:0/2855985216 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f36a400bb30 con 0x7f36a4004810 2024-03-28T14:08:07.331 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:07.327+0000 7f36affff640 1 -- 172.21.15.104:0/2855985216 --> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] -- mon_getmap magic: 0 -- 0x7f36a4010140 con 0x7f36b80b4fa0 2024-03-28T14:08:07.331 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:07.327+0000 7f36bd7f1640 1 --2- 172.21.15.104:0/2855985216 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f36a40039f0 0x7f36a4003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:07.331 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:07.327+0000 7f36bcff0640 1 --2- 172.21.15.104:0/2855985216 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f36a4004810 0x7f36a4004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:07.331 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:07.327+0000 7f36bdff2640 1 --2- 172.21.15.104:0/2855985216 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f36b80b4fa0 0x7f36a4005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:07.331 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:07.327+0000 7f36bd7f1640 1 -- 172.21.15.104:0/2855985216 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f36b80b4fa0 msgr2=0x7f36a4005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T14:08:07.331 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:07.327+0000 7f36bd7f1640 1 --2- 172.21.15.104:0/2855985216 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f36b80b4fa0 0x7f36a4005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:07.331 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:07.327+0000 7f36bd7f1640 1 -- 172.21.15.104:0/2855985216 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f36a4004810 msgr2=0x7f36a4004c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T14:08:07.331 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:07.327+0000 7f36bd7f1640 1 --2- 172.21.15.104:0/2855985216 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f36a4004810 0x7f36a4004c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:07.331 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:07.327+0000 7f36bd7f1640 1 -- 172.21.15.104:0/2855985216 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f36a4003530 con 0x7f36a40039f0 2024-03-28T14:08:07.331 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:07.327+0000 7f36bdff2640 1 --2- 172.21.15.104:0/2855985216 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f36b80b4fa0 0x7f36a4005210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T14:08:07.334 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:07.331+0000 7f36bd7f1640 1 --2- 172.21.15.104:0/2855985216 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f36a40039f0 0x7f36a4003e10 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f36b400c240 tx=0x7f36b401cc50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=882b2336ef7faa1e server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:08:07.334 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:07.331+0000 7f36affff640 1 -- 172.21.15.104:0/2855985216 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f36b4007d20 con 0x7f36a40039f0 2024-03-28T14:08:07.334 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:07.331+0000 7f36affff640 1 -- 172.21.15.104:0/2855985216 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f36b40a9070 con 0x7f36a40039f0 2024-03-28T14:08:07.334 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:07.331+0000 7f36affff640 1 -- 172.21.15.104:0/2855985216 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f36b40a5d50 con 0x7f36a40039f0 2024-03-28T14:08:07.334 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:07.331+0000 7f36bfa55640 1 -- 172.21.15.104:0/2855985216 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f36a40039f0 msgr2=0x7f36a4003e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:07.335 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:07.331+0000 7f36bfa55640 1 --2- 172.21.15.104:0/2855985216 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f36a40039f0 0x7f36a4003e10 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f36b400c240 tx=0x7f36b401cc50 comp rx=0 tx=0).stop 2024-03-28T14:08:07.335 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:07.331+0000 7f36bdff2640 1 -- 172.21.15.104:0/2855985216 reap_dead start 2024-03-28T14:08:07.335 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:07.331+0000 7f36bfa55640 1 -- 172.21.15.104:0/2855985216 shutdown_connections 2024-03-28T14:08:07.335 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:07.331+0000 7f36bfa55640 1 -- 172.21.15.104:0/2855985216 >> 172.21.15.104:0/2855985216 conn(0x7f36b80ba390 msgr2=0x7f36b80ba7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T14:08:07.335 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:07.331+0000 7f36bfa55640 1 -- 172.21.15.104:0/2855985216 shutdown_connections 2024-03-28T14:08:07.335 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:07.331+0000 7f36bfa55640 1 -- 172.21.15.104:0/2855985216 wait complete. 2024-03-28T14:08:07.337 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:07.331+0000 7f36bfa55640 1 Processor -- start 2024-03-28T14:08:07.337 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:07.331+0000 7f36bfa55640 1 -- start start 2024-03-28T14:08:07.337 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:07.335+0000 7f36bfa55640 1 --2- >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f36a40039f0 0x7f36b809f540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:07.337 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:07.335+0000 7f36bfa55640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f36b80b3f00 0x7f36b809ce10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:07.337 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:07.335+0000 7f36bfa55640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f36b809c930 0x7f36b81dc080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:07.337 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:07.335+0000 7f36bfa55640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f36b80af7c0 con 0x7f36b809c930 2024-03-28T14:08:07.338 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:07.335+0000 7f36bfa55640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f36b80b6780 con 0x7f36b80b3f00 2024-03-28T14:08:07.338 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:07.335+0000 7f36bfa55640 1 -- --> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] -- mon_getmap magic: 0 -- 0x7f36b80af940 con 0x7f36a40039f0 2024-03-28T14:08:07.338 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:07.335+0000 7f36bdff2640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f36b809c930 0x7f36b81dc080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:07.338 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:07.335+0000 7f36bdff2640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f36b809c930 0x7f36b81dc080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:34728/0 (socket says 172.21.15.104:34728) 2024-03-28T14:08:07.338 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:07.335+0000 7f36bdff2640 1 -- 172.21.15.104:0/114064881 learned_addr learned my addr 172.21.15.104:0/114064881 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-03-28T14:08:07.338 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:07.335+0000 7f36bd7f1640 1 --2- 172.21.15.104:0/114064881 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f36a40039f0 0x7f36b809f540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:07.338 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:07.335+0000 7f36bcff0640 1 --2- 172.21.15.104:0/114064881 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f36b80b3f00 0x7f36b809ce10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:07.338 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:07.335+0000 7f36bdff2640 1 -- 172.21.15.104:0/114064881 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f36a40039f0 msgr2=0x7f36b809f540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:07.338 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:07.335+0000 7f36bdff2640 1 --2- 172.21.15.104:0/114064881 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f36a40039f0 0x7f36b809f540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:07.338 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:07.335+0000 7f36bdff2640 1 -- 172.21.15.104:0/114064881 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f36b80b3f00 msgr2=0x7f36b809ce10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:07.338 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:07.335+0000 7f36bdff2640 1 --2- 172.21.15.104:0/114064881 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f36b80b3f00 0x7f36b809ce10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:07.338 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:07.335+0000 7f36bdff2640 1 -- 172.21.15.104:0/114064881 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f36b809eb50 con 0x7f36b809c930 2024-03-28T14:08:07.338 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:07.335+0000 7f36bdff2640 1 --2- 172.21.15.104:0/114064881 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f36b809c930 0x7f36b81dc080 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f36b8145840 tx=0x7f36b000dc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:08:07.339 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:07.335+0000 7f36adffb640 1 -- 172.21.15.104:0/114064881 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f36b0006d60 con 0x7f36b809c930 2024-03-28T14:08:07.339 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:07.335+0000 7f36adffb640 1 -- 172.21.15.104:0/114064881 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f36b0003980 con 0x7f36b809c930 2024-03-28T14:08:07.339 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:07.335+0000 7f36adffb640 1 -- 172.21.15.104:0/114064881 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f36b0003d70 con 0x7f36b809c930 2024-03-28T14:08:07.339 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:07.335+0000 7f36bfa55640 1 -- 172.21.15.104:0/114064881 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f36b809d4c0 con 0x7f36b809c930 2024-03-28T14:08:07.339 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:07.335+0000 7f36bfa55640 1 -- 172.21.15.104:0/114064881 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f36b80b39e0 con 0x7f36b809c930 2024-03-28T14:08:07.340 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:07.335+0000 7f36bfa55640 1 -- 172.21.15.104:0/114064881 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f36b80b3be0 con 0x7f36b809c930 2024-03-28T14:08:07.340 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:07.335+0000 7f36adffb640 1 -- 172.21.15.104:0/114064881 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) ==== 87803+0+0 (secure 0 0 0) 0x7f36b001b020 con 0x7f36b809c930 2024-03-28T14:08:07.340 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:07.335+0000 7f36adffb640 1 --2- 172.21.15.104:0/114064881 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f369c067fc0 0x7f369c06a480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:07.340 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:07.335+0000 7f36adffb640 1 -- 172.21.15.104:0/114064881 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f36b0086130 con 0x7f36b809c930 2024-03-28T14:08:07.340 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:07.335+0000 7f36bd7f1640 1 --2- 172.21.15.104:0/114064881 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f369c067fc0 0x7f369c06a480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:07.340 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:07.339+0000 7f36bd7f1640 1 --2- 172.21.15.104:0/114064881 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f369c067fc0 0x7f369c06a480 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f36b40ab000 tx=0x7f36b40b1040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:08:07.342 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:07.339+0000 7f36adffb640 1 -- 172.21.15.104:0/114064881 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179117 (secure 0 0 0) 0x7f36b80b3be0 con 0x7f36b809c930 2024-03-28T14:08:07.505 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:07.503+0000 7f36bfa55640 1 -- 172.21.15.104:0/114064881 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f36b80687c0 con 0x7f36b809c930 2024-03-28T14:08:07.506 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:07.503+0000 7f36adffb640 1 -- 172.21.15.104:0/114064881 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) ==== 74+0+16629 (secure 0 0 0) 0x7f36b0056970 con 0x7f36b809c930 2024-03-28T14:08:07.507 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-03-28T14:08:07.507 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":21,"fsid":"428a90f2-3351-435f-8fcf-f7e66c78a13e","created":"2024-03-28T14:07:34.862458+0000","modified":"2024-03-28T14:08:00.671587+0000","last_up_change":"2024-03-28T14:07:46.028122+0000","last_in_change":"2024-03-28T14:07:37.157486+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-28T14:07:47.622988+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-28T14:07:57.147450+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-28T14:07:57.990975+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"1cb70540-4e42-402f-a446-aaec4ca83007","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6800","nonce":3745690918},{"type":"v1","addr":"172.21.15.104:6801","nonce":3745690918}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":3745690918},{"type":"v1","addr":"172.21.15.104:6803","nonce":3745690918}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":3745690918},{"type":"v1","addr":"172.21.15.104:6807","nonce":3745690918}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":3745690918},{"type":"v1","addr":"172.21.15.104:6805","nonce":3745690918}]},"public_addr":"172.21.15.104:6801/3745690918","cluster_addr":"172.21.15.104:6803/3745690918","heartbeat_back_addr":"172.21.15.104:6807/3745690918","heartbeat_front_addr":"172.21.15.104:6805/3745690918","state":["exists","up"]},{"osd":1,"uuid":"3ab7afdd-b980-418c-81f2-004e16de3dca","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":19438674},{"type":"v1","addr":"172.21.15.104:6810","nonce":19438674}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":19438674},{"type":"v1","addr":"172.21.15.104:6814","nonce":19438674}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":19438674},{"type":"v1","addr":"172.21.15.104:6823","nonce":19438674}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":19438674},{"type":"v1","addr":"172.21.15.104:6819","nonce":19438674}]},"public_addr":"172.21.15.104:6810/19438674","cluster_addr":"172.21.15.104:6814/19438674","heartbeat_back_addr":"172.21.15.104:6823/19438674","heartbeat_front_addr":"172.21.15.104:6819/19438674","state":["exists","up"]},{"osd":2,"uuid":"3e2651c8-ecae-4a9e-a053-27240a79eac2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":2352012626},{"type":"v1","addr":"172.21.15.104:6825","nonce":2352012626}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6826","nonce":2352012626},{"type":"v1","addr":"172.21.15.104:6827","nonce":2352012626}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":2352012626},{"type":"v1","addr":"172.21.15.104:6831","nonce":2352012626}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6828","nonce":2352012626},{"type":"v1","addr":"172.21.15.104:6829","nonce":2352012626}]},"public_addr":"172.21.15.104:6825/2352012626","cluster_addr":"172.21.15.104:6827/2352012626","heartbeat_back_addr":"172.21.15.104:6831/2352012626","heartbeat_front_addr":"172.21.15.104:6829/2352012626","state":["exists","up"]},{"osd":3,"uuid":"925c7e0a-38ef-47cd-b288-8bec8456fb17","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6809","nonce":3568347381},{"type":"v1","addr":"172.21.15.104:6811","nonce":3568347381}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6813","nonce":3568347381},{"type":"v1","addr":"172.21.15.104:6815","nonce":3568347381}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":3568347381},{"type":"v1","addr":"172.21.15.104:6822","nonce":3568347381}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6817","nonce":3568347381},{"type":"v1","addr":"172.21.15.104:6818","nonce":3568347381}]},"public_addr":"172.21.15.104:6811/3568347381","cluster_addr":"172.21.15.104:6815/3568347381","heartbeat_back_addr":"172.21.15.104:6822/3568347381","heartbeat_front_addr":"172.21.15.104:6818/3568347381","state":["exists","up"]},{"osd":4,"uuid":"4a8fac86-e595-4a0e-96e3-57edb3cc3f57","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6800","nonce":2107001077},{"type":"v1","addr":"172.21.15.171:6801","nonce":2107001077}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6802","nonce":2107001077},{"type":"v1","addr":"172.21.15.171:6803","nonce":2107001077}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6806","nonce":2107001077},{"type":"v1","addr":"172.21.15.171:6807","nonce":2107001077}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6804","nonce":2107001077},{"type":"v1","addr":"172.21.15.171:6805","nonce":2107001077}]},"public_addr":"172.21.15.171:6801/2107001077","cluster_addr":"172.21.15.171:6803/2107001077","heartbeat_back_addr":"172.21.15.171:6807/2107001077","heartbeat_front_addr":"172.21.15.171:6805/2107001077","state":["exists","up"]},{"osd":5,"uuid":"e48d50b0-beae-40ae-b15a-9a84dbfe88ab","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6808","nonce":2934826925},{"type":"v1","addr":"172.21.15.171:6810","nonce":2934826925}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6812","nonce":2934826925},{"type":"v1","addr":"172.21.15.171:6814","nonce":2934826925}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6819","nonce":2934826925},{"type":"v1","addr":"172.21.15.171:6821","nonce":2934826925}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6816","nonce":2934826925},{"type":"v1","addr":"172.21.15.171:6817","nonce":2934826925}]},"public_addr":"172.21.15.171:6810/2934826925","cluster_addr":"172.21.15.171:6814/2934826925","heartbeat_back_addr":"172.21.15.171:6821/2934826925","heartbeat_front_addr":"172.21.15.171:6817/2934826925","state":["exists","up"]},{"osd":6,"uuid":"5e7c3292-9936-4f35-b267-aed9d4e49323","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6824","nonce":436742919},{"type":"v1","addr":"172.21.15.171:6825","nonce":436742919}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6826","nonce":436742919},{"type":"v1","addr":"172.21.15.171:6827","nonce":436742919}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6830","nonce":436742919},{"type":"v1","addr":"172.21.15.171:6831","nonce":436742919}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6828","nonce":436742919},{"type":"v1","addr":"172.21.15.171:6829","nonce":436742919}]},"public_addr":"172.21.15.171:6825/436742919","cluster_addr":"172.21.15.171:6827/436742919","heartbeat_back_addr":"172.21.15.171:6831/436742919","heartbeat_front_addr":"172.21.15.171:6829/436742919","state":["exists","up"]},{"osd":7,"uuid":"339b8414-d7f0-4fb0-a446-efaf635cb7ff","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6809","nonce":2055631175},{"type":"v1","addr":"172.21.15.171:6811","nonce":2055631175}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6813","nonce":2055631175},{"type":"v1","addr":"172.21.15.171:6815","nonce":2055631175}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6822","nonce":2055631175},{"type":"v1","addr":"172.21.15.171:6823","nonce":2055631175}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6818","nonce":2055631175},{"type":"v1","addr":"172.21.15.171:6820","nonce":2055631175}]},"public_addr":"172.21.15.171:6811/2055631175","cluster_addr":"172.21.15.171:6815/2055631175","heartbeat_back_addr":"172.21.15.171:6823/2055631175","heartbeat_front_addr":"172.21.15.171:6820/2055631175","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T14:07:42.668881+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T14:07:42.708397+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T14:07:42.634191+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T14:07:42.733662+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T14:07:42.651541+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T14:07:42.783181+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T14:07:42.766310+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T14:07:42.670797+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-28T14:08:07.509 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:07.507+0000 7f36bfa55640 1 -- 172.21.15.104:0/114064881 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f369c067fc0 msgr2=0x7f369c06a480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:07.510 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:07.507+0000 7f36bfa55640 1 --2- 172.21.15.104:0/114064881 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f369c067fc0 0x7f369c06a480 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f36b40ab000 tx=0x7f36b40b1040 comp rx=0 tx=0).stop 2024-03-28T14:08:07.510 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:07.507+0000 7f36bfa55640 1 -- 172.21.15.104:0/114064881 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f36b809c930 msgr2=0x7f36b81dc080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:07.510 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:07.507+0000 7f36bfa55640 1 --2- 172.21.15.104:0/114064881 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f36b809c930 0x7f36b81dc080 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f36b8145840 tx=0x7f36b000dc00 comp rx=0 tx=0).stop 2024-03-28T14:08:07.510 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:07.507+0000 7f36bfa55640 1 -- 172.21.15.104:0/114064881 shutdown_connections 2024-03-28T14:08:07.510 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:07.507+0000 7f36bfa55640 1 --2- 172.21.15.104:0/114064881 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f369c067fc0 0x7f369c06a480 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:07.510 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:07.507+0000 7f36bfa55640 1 --2- 172.21.15.104:0/114064881 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f36b809c930 0x7f36b81dc080 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:07.510 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:07.507+0000 7f36bfa55640 1 --2- 172.21.15.104:0/114064881 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f36b80b3f00 0x7f36b809ce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:07.510 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:07.507+0000 7f36bfa55640 1 --2- 172.21.15.104:0/114064881 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f36a40039f0 0x7f36b809f540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:07.510 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:07.507+0000 7f36bfa55640 1 -- 172.21.15.104:0/114064881 >> 172.21.15.104:0/114064881 conn(0x7f36b80ba390 msgr2=0x7f36b80b01a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T14:08:07.511 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:07.507+0000 7f36bfa55640 1 -- 172.21.15.104:0/114064881 shutdown_connections 2024-03-28T14:08:07.511 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:07.507+0000 7f36bfa55640 1 -- 172.21.15.104:0/114064881 wait complete. 2024-03-28T14:08:07.524 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-28T14:07:47.622988+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-28T14:07:57.147450+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-28T14:07:57.990975+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-28T14:08:07.525 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-28T14:08:07.647 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:07.647+0000 7f4be6cc3640 1 Processor -- start 2024-03-28T14:08:07.647 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:07.647+0000 7f4be6cc3640 1 -- start start 2024-03-28T14:08:07.651 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:07.647+0000 7f4be6cc3640 1 --2- >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f4be00b63a0 0x7f4be00af9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:07.651 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:07.647+0000 7f4be6cc3640 1 --2- >> v2:172.21.15.171:3300/0 conn(0x7f4be01445e0 0x7f4be0144a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:07.651 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:07.647+0000 7f4be6cc3640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4be0144f50 con 0x7f4be00b0010 2024-03-28T14:08:07.652 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:07.647+0000 7f4be6cc3640 1 -- --> v2:172.21.15.171:3300/0 -- mon_getmap magic: 0 -- 0x7f4be00af610 con 0x7f4be01445e0 2024-03-28T14:08:07.652 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:07.647+0000 7f4be6cc3640 1 -- --> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] -- mon_getmap magic: 0 -- 0x7f4be00b0490 con 0x7f4be00b63a0 2024-03-28T14:08:07.652 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:07.647+0000 7f4bdffff640 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f4be00b0010 0x7f4be0141670 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:41608/0 (socket says 172.21.15.104:41608) 2024-03-28T14:08:07.652 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:07.647+0000 7f4bdffff640 1 -- 172.21.15.104:0/3764068671 learned_addr learned my addr 172.21.15.104:0/3764068671 (peer_addr_for_me v1:172.21.15.104:0/0) 2024-03-28T14:08:07.652 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:07.647+0000 7f4be4a5f640 1 --2- 172.21.15.104:0/3764068671 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f4be00b63a0 0x7f4be00af9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:07.652 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:07.647+0000 7f4be5260640 1 --2- 172.21.15.104:0/3764068671 >> v2:172.21.15.171:3300/0 conn(0x7f4be01445e0 0x7f4be0144a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:07.652 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:07.647+0000 7f4bdf7fe640 1 -- 172.21.15.104:0/3764068671 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2265446642 0 0) 0x7f4be0144f50 con 0x7f4be00b0010 2024-03-28T14:08:07.652 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:07.647+0000 7f4bdf7fe640 1 -- 172.21.15.104:0/3764068671 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4bc8003530 con 0x7f4be00b0010 2024-03-28T14:08:07.652 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:07.647+0000 7f4bdf7fe640 1 -- 172.21.15.104:0/3764068671 <== mon.0 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 292370924 0 0) 0x7f4bc8003530 con 0x7f4be00b0010 2024-03-28T14:08:07.653 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:07.647+0000 7f4bdf7fe640 1 -- 172.21.15.104:0/3764068671 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f4be00b63a0 msgr2=0x7f4be00af9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T14:08:07.653 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:07.647+0000 7f4bdf7fe640 1 --2- 172.21.15.104:0/3764068671 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f4be00b63a0 0x7f4be00af9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:07.653 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:07.647+0000 7f4bdf7fe640 1 -- 172.21.15.104:0/3764068671 >> v2:172.21.15.171:3300/0 conn(0x7f4be01445e0 msgr2=0x7f4be0144a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T14:08:07.653 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:07.647+0000 7f4bdf7fe640 1 --2- 172.21.15.104:0/3764068671 >> v2:172.21.15.171:3300/0 conn(0x7f4be01445e0 0x7f4be0144a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:07.653 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:07.647+0000 7f4bdf7fe640 1 -- 172.21.15.104:0/3764068671 --> v1:172.21.15.104:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4be00b6780 con 0x7f4be00b0010 2024-03-28T14:08:07.653 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:07.647+0000 7f4bdf7fe640 1 -- 172.21.15.104:0/3764068671 <== mon.0 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3745580849 0 0) 0x7f4bd4001680 con 0x7f4be00b0010 2024-03-28T14:08:07.653 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:07.647+0000 7f4bdf7fe640 1 -- 172.21.15.104:0/3764068671 >> v1:172.21.15.104:6789/0 conn(0x7f4be00b0010 legacy=0x7f4be0141670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:07.653 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:07.647+0000 7f4bdf7fe640 1 --2- 172.21.15.104:0/3764068671 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4bc80039c0 0x7f4bc8003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:07.654 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:07.647+0000 7f4bdf7fe640 1 --2- 172.21.15.104:0/3764068671 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f4bc8004810 0x7f4bc8004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:07.654 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:07.647+0000 7f4bdf7fe640 1 --2- 172.21.15.104:0/3764068671 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f4be00b63a0 0x7f4bc80051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:07.654 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:07.647+0000 7f4bdf7fe640 1 -- 172.21.15.104:0/3764068671 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f4be00b0490 con 0x7f4bc80039c0 2024-03-28T14:08:07.654 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:07.647+0000 7f4bdf7fe640 1 -- 172.21.15.104:0/3764068671 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f4bc800b9d0 con 0x7f4bc8004810 2024-03-28T14:08:07.654 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:07.647+0000 7f4bdf7fe640 1 -- 172.21.15.104:0/3764068671 --> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] -- mon_getmap magic: 0 -- 0x7f4bc800dd60 con 0x7f4be00b63a0 2024-03-28T14:08:07.654 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:07.647+0000 7f4be4a5f640 1 --2- 172.21.15.104:0/3764068671 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4bc80039c0 0x7f4bc8003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:07.654 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:07.647+0000 7f4bdffff640 1 --2- 172.21.15.104:0/3764068671 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f4bc8004810 0x7f4bc8004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:07.654 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:07.647+0000 7f4be5260640 1 --2- 172.21.15.104:0/3764068671 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f4be00b63a0 0x7f4bc80051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:07.654 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:07.647+0000 7f4be4a5f640 1 -- 172.21.15.104:0/3764068671 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f4be00b63a0 msgr2=0x7f4bc80051d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T14:08:07.654 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:07.647+0000 7f4be4a5f640 1 --2- 172.21.15.104:0/3764068671 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f4be00b63a0 0x7f4bc80051d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:07.654 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:07.647+0000 7f4be4a5f640 1 -- 172.21.15.104:0/3764068671 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f4bc8004810 msgr2=0x7f4bc8004c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T14:08:07.655 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:07.647+0000 7f4be4a5f640 1 --2- 172.21.15.104:0/3764068671 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f4bc8004810 0x7f4bc8004c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:07.655 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:07.647+0000 7f4be4a5f640 1 -- 172.21.15.104:0/3764068671 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4bc8003530 con 0x7f4bc80039c0 2024-03-28T14:08:07.655 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:07.647+0000 7f4be5260640 1 --2- 172.21.15.104:0/3764068671 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f4be00b63a0 0x7f4bc80051d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T14:08:07.663 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:07.651+0000 7f4be4a5f640 1 --2- 172.21.15.104:0/3764068671 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4bc80039c0 0x7f4bc8003de0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f4bd000e170 tx=0x7f4bd001cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9631406c1e3648d2 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:08:07.663 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:07.651+0000 7f4bdf7fe640 1 -- 172.21.15.104:0/3764068671 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4bd00b2070 con 0x7f4bc80039c0 2024-03-28T14:08:07.663 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:07.651+0000 7f4bdf7fe640 1 -- 172.21.15.104:0/3764068671 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4bd00a5870 con 0x7f4bc80039c0 2024-03-28T14:08:07.663 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:07.651+0000 7f4bdf7fe640 1 -- 172.21.15.104:0/3764068671 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4bd0009c10 con 0x7f4bc80039c0 2024-03-28T14:08:07.663 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:07.651+0000 7f4be6cc3640 1 -- 172.21.15.104:0/3764068671 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4bc80039c0 msgr2=0x7f4bc8003de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:07.663 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:07.651+0000 7f4be6cc3640 1 --2- 172.21.15.104:0/3764068671 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4bc80039c0 0x7f4bc8003de0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f4bd000e170 tx=0x7f4bd001cc60 comp rx=0 tx=0).stop 2024-03-28T14:08:07.663 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:07.651+0000 7f4be5260640 1 -- 172.21.15.104:0/3764068671 reap_dead start 2024-03-28T14:08:07.663 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:07.651+0000 7f4be6cc3640 1 -- 172.21.15.104:0/3764068671 shutdown_connections 2024-03-28T14:08:07.663 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:07.651+0000 7f4be6cc3640 1 -- 172.21.15.104:0/3764068671 >> 172.21.15.104:0/3764068671 conn(0x7f4be00ba400 msgr2=0x7f4be00ba810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T14:08:07.664 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:07.651+0000 7f4be6cc3640 1 -- 172.21.15.104:0/3764068671 shutdown_connections 2024-03-28T14:08:07.664 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:07.651+0000 7f4be6cc3640 1 -- 172.21.15.104:0/3764068671 wait complete. 2024-03-28T14:08:07.664 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:07.651+0000 7f4be6cc3640 1 Processor -- start 2024-03-28T14:08:07.664 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:07.651+0000 7f4be6cc3640 1 -- start start 2024-03-28T14:08:07.664 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:07.655+0000 7f4be6cc3640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f4bc80039c0 0x7f4be0148570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:07.664 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:07.655+0000 7f4be6cc3640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4be01dbbe0 0x7f4be01dc060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:07.664 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:07.655+0000 7f4be6cc3640 1 --2- >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f4be01dc5a0 0x7f4be01df1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:07.664 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:07.655+0000 7f4be6cc3640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f4be006bf60 con 0x7f4be01dbbe0 2024-03-28T14:08:07.664 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:07.655+0000 7f4be6cc3640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f4be006bde0 con 0x7f4bc80039c0 2024-03-28T14:08:07.664 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:07.655+0000 7f4be6cc3640 1 -- --> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] -- mon_getmap magic: 0 -- 0x7f4be006c0e0 con 0x7f4be01dc5a0 2024-03-28T14:08:07.664 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:07.655+0000 7f4bdffff640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4be01dbbe0 0x7f4be01dc060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:07.664 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:07.655+0000 7f4bdffff640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4be01dbbe0 0x7f4be01dc060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:34746/0 (socket says 172.21.15.104:34746) 2024-03-28T14:08:07.664 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:07.655+0000 7f4bdffff640 1 -- 172.21.15.104:0/1344213682 learned_addr learned my addr 172.21.15.104:0/1344213682 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-03-28T14:08:07.665 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:07.655+0000 7f4be4a5f640 1 --2- 172.21.15.104:0/1344213682 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f4bc80039c0 0x7f4be0148570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:07.665 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:07.655+0000 7f4be5260640 1 --2- 172.21.15.104:0/1344213682 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f4be01dc5a0 0x7f4be01df1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:07.665 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:07.655+0000 7f4bdffff640 1 -- 172.21.15.104:0/1344213682 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f4be01dc5a0 msgr2=0x7f4be01df1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:07.665 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:07.655+0000 7f4bdffff640 1 --2- 172.21.15.104:0/1344213682 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f4be01dc5a0 0x7f4be01df1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:07.665 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:07.655+0000 7f4bdffff640 1 -- 172.21.15.104:0/1344213682 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f4bc80039c0 msgr2=0x7f4be0148570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:07.665 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:07.655+0000 7f4bdffff640 1 --2- 172.21.15.104:0/1344213682 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f4bc80039c0 0x7f4be0148570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:07.665 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:07.655+0000 7f4bdffff640 1 -- 172.21.15.104:0/1344213682 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4be00b3450 con 0x7f4be01dbbe0 2024-03-28T14:08:07.665 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:07.655+0000 7f4be5260640 1 --2- 172.21.15.104:0/1344213682 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f4be01dc5a0 0x7f4be01df1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T14:08:07.665 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:07.655+0000 7f4bdffff640 1 --2- 172.21.15.104:0/1344213682 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4be01dbbe0 0x7f4be01dc060 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f4bd400e7e0 tx=0x7f4bd400eca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:08:07.665 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:07.655+0000 7f4bdd7fa640 1 -- 172.21.15.104:0/1344213682 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4bd400c6c0 con 0x7f4be01dbbe0 2024-03-28T14:08:07.665 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:07.655+0000 7f4bdd7fa640 1 -- 172.21.15.104:0/1344213682 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4bd4018070 con 0x7f4be01dbbe0 2024-03-28T14:08:07.665 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:07.655+0000 7f4be6cc3640 1 -- 172.21.15.104:0/1344213682 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4be00b17d0 con 0x7f4be01dbbe0 2024-03-28T14:08:07.666 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:07.655+0000 7f4bdd7fa640 1 -- 172.21.15.104:0/1344213682 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4bd400cac0 con 0x7f4be01dbbe0 2024-03-28T14:08:07.666 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:07.655+0000 7f4be6cc3640 1 -- 172.21.15.104:0/1344213682 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4be0072660 con 0x7f4be01dbbe0 2024-03-28T14:08:07.666 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:07.655+0000 7f4be6cc3640 1 -- 172.21.15.104:0/1344213682 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4be00687c0 con 0x7f4be01dbbe0 2024-03-28T14:08:07.666 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:07.655+0000 7f4bdd7fa640 1 -- 172.21.15.104:0/1344213682 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) ==== 87803+0+0 (secure 0 0 0) 0x7f4bd400cc60 con 0x7f4be01dbbe0 2024-03-28T14:08:07.666 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:07.655+0000 7f4bdd7fa640 1 --2- 172.21.15.104:0/1344213682 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f4bcc067ff0 0x7f4bcc06a4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:07.666 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:07.655+0000 7f4bdd7fa640 1 -- 172.21.15.104:0/1344213682 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f4bd4085ef0 con 0x7f4be01dbbe0 2024-03-28T14:08:07.666 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:07.655+0000 7f4be4a5f640 1 --2- 172.21.15.104:0/1344213682 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f4bcc067ff0 0x7f4bcc06a4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:07.666 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:07.659+0000 7f4be4a5f640 1 --2- 172.21.15.104:0/1344213682 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f4bcc067ff0 0x7f4bcc06a4b0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f4bd00af000 tx=0x7f4bd0004ac0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:08:07.666 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:07.659+0000 7f4bdd7fa640 1 -- 172.21.15.104:0/1344213682 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179117 (secure 0 0 0) 0x7f4bd4056730 con 0x7f4be01dbbe0 2024-03-28T14:08:07.839 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:07.835+0000 7f4be6cc3640 1 -- 172.21.15.104:0/1344213682 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f4be0144070 con 0x7f4be01dbbe0 2024-03-28T14:08:07.839 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:07.839+0000 7f4bdd7fa640 1 -- 172.21.15.104:0/1344213682 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v21) ==== 93+0+10 (secure 0 0 0) 0x7f4bd4056030 con 0x7f4be01dbbe0 2024-03-28T14:08:07.840 INFO:teuthology.orchestra.run.smithi104.stdout:pg_num: 1 2024-03-28T14:08:07.842 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:07.839+0000 7f4be6cc3640 1 -- 172.21.15.104:0/1344213682 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f4bcc067ff0 msgr2=0x7f4bcc06a4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:07.842 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:07.839+0000 7f4be6cc3640 1 --2- 172.21.15.104:0/1344213682 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f4bcc067ff0 0x7f4bcc06a4b0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f4bd00af000 tx=0x7f4bd0004ac0 comp rx=0 tx=0).stop 2024-03-28T14:08:07.842 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:07.839+0000 7f4be6cc3640 1 -- 172.21.15.104:0/1344213682 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4be01dbbe0 msgr2=0x7f4be01dc060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:07.842 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:07.839+0000 7f4be6cc3640 1 --2- 172.21.15.104:0/1344213682 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4be01dbbe0 0x7f4be01dc060 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f4bd400e7e0 tx=0x7f4bd400eca0 comp rx=0 tx=0).stop 2024-03-28T14:08:07.843 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:07.839+0000 7f4be6cc3640 1 -- 172.21.15.104:0/1344213682 shutdown_connections 2024-03-28T14:08:07.843 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:07.839+0000 7f4be6cc3640 1 --2- 172.21.15.104:0/1344213682 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f4be01dc5a0 0x7f4be01df1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:07.843 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:07.839+0000 7f4be6cc3640 1 --2- 172.21.15.104:0/1344213682 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f4bcc067ff0 0x7f4bcc06a4b0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:07.843 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:07.839+0000 7f4be6cc3640 1 --2- 172.21.15.104:0/1344213682 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4be01dbbe0 0x7f4be01dc060 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:07.843 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:07.839+0000 7f4be6cc3640 1 --2- 172.21.15.104:0/1344213682 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f4bc80039c0 0x7f4be0148570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:07.843 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:07.839+0000 7f4be6cc3640 1 -- 172.21.15.104:0/1344213682 >> 172.21.15.104:0/1344213682 conn(0x7f4be00ba400 msgr2=0x7f4be00af200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T14:08:07.843 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:07.839+0000 7f4be6cc3640 1 -- 172.21.15.104:0/1344213682 shutdown_connections 2024-03-28T14:08:07.843 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:07.839+0000 7f4be6cc3640 1 -- 172.21.15.104:0/1344213682 wait complete. 2024-03-28T14:08:07.857 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-28T14:08:08.007 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:08.003+0000 7f314d8c0640 1 Processor -- start 2024-03-28T14:08:08.007 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:08.003+0000 7f314d8c0640 1 -- start start 2024-03-28T14:08:08.009 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:08.007+0000 7f314d8c0640 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f314806c8f0 0x7f31480b2740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:08.009 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:08.007+0000 7f314d8c0640 1 --2- >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f31480b3730 0x7f3148148c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:08.009 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:08.007+0000 7f314d8c0640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f31480b3b30 con 0x7f31481491b0 2024-03-28T14:08:08.009 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:08.007+0000 7f314d8c0640 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 -- 0x7f31480b0b30 con 0x7f314806c8f0 2024-03-28T14:08:08.009 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:08.007+0000 7f314d8c0640 1 -- --> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] -- mon_getmap magic: 0 -- 0x7f31480b22b0 con 0x7f31480b3730 2024-03-28T14:08:08.009 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:08.007+0000 7f3146ffd640 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f314806c8f0 0x7f31480b2740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:08.009 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:08.007+0000 7f31477fe640 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f31481491b0 0x7f3148147b80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:41610/0 (socket says 172.21.15.104:41610) 2024-03-28T14:08:08.009 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:08.007+0000 7f31477fe640 1 -- 172.21.15.104:0/2743426329 learned_addr learned my addr 172.21.15.104:0/2743426329 (peer_addr_for_me v1:172.21.15.104:0/0) 2024-03-28T14:08:08.009 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:08.007+0000 7f31467fc640 1 --2- 172.21.15.104:0/2743426329 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f31480b3730 0x7f3148148c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:08.009 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:08.007+0000 7f3146ffd640 1 -- 172.21.15.104:0/2743426329 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f31480b3730 msgr2=0x7f3148148c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T14:08:08.010 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:08.007+0000 7f3146ffd640 1 --2- 172.21.15.104:0/2743426329 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f31480b3730 0x7f3148148c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:08.010 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:08.007+0000 7f3146ffd640 1 -- 172.21.15.104:0/2743426329 >> v1:172.21.15.104:6789/0 conn(0x7f31481491b0 legacy=0x7f3148147b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:08.010 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:08.007+0000 7f3146ffd640 1 -- 172.21.15.104:0/2743426329 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f314806ccd0 con 0x7f314806c8f0 2024-03-28T14:08:08.010 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:08.007+0000 7f3145ffb640 1 -- 172.21.15.104:0/2743426329 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2490531027 0 0) 0x7f31480b3b30 con 0x7f31481491b0 2024-03-28T14:08:08.010 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:08.007+0000 7f3146ffd640 1 --2- 172.21.15.104:0/2743426329 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f314806c8f0 0x7f31480b2740 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f31300097c0 tx=0x7f313001c3e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b39fb31c24d3df88 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:08:08.011 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:08.007+0000 7f3145ffb640 1 -- 172.21.15.104:0/2743426329 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31300b1070 con 0x7f314806c8f0 2024-03-28T14:08:08.011 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:08.007+0000 7f3145ffb640 1 -- 172.21.15.104:0/2743426329 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f31300a8050 con 0x7f314806c8f0 2024-03-28T14:08:08.011 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:08.007+0000 7f3145ffb640 1 -- 172.21.15.104:0/2743426329 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31300ac860 con 0x7f314806c8f0 2024-03-28T14:08:08.012 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:08.011+0000 7f314d8c0640 1 -- 172.21.15.104:0/2743426329 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f314806c8f0 msgr2=0x7f31480b2740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:08.012 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:08.011+0000 7f314d8c0640 1 --2- 172.21.15.104:0/2743426329 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f314806c8f0 0x7f31480b2740 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f31300097c0 tx=0x7f313001c3e0 comp rx=0 tx=0).stop 2024-03-28T14:08:08.012 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:08.011+0000 7f314d8c0640 1 -- 172.21.15.104:0/2743426329 shutdown_connections 2024-03-28T14:08:08.012 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:08.011+0000 7f314d8c0640 1 --2- 172.21.15.104:0/2743426329 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f31480b3730 0x7f3148148c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:08.012 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:08.011+0000 7f314d8c0640 1 --2- 172.21.15.104:0/2743426329 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f314806c8f0 0x7f31480b2740 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:08.012 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:08.011+0000 7f314d8c0640 1 -- 172.21.15.104:0/2743426329 >> 172.21.15.104:0/2743426329 conn(0x7f31480b8f00 msgr2=0x7f31480b9310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T14:08:08.012 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:08.011+0000 7f314d8c0640 1 -- 172.21.15.104:0/2743426329 shutdown_connections 2024-03-28T14:08:08.013 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:08.011+0000 7f314d8c0640 1 -- 172.21.15.104:0/2743426329 wait complete. 2024-03-28T14:08:08.013 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:08.011+0000 7f314d8c0640 1 Processor -- start 2024-03-28T14:08:08.013 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:08.011+0000 7f314d8c0640 1 -- start start 2024-03-28T14:08:08.017 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:08.011+0000 7f314d8c0640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f314806c8f0 0x7f31480af920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:08.017 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:08.011+0000 7f314d8c0640 1 --2- >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f31480b3730 0x7f31481e35b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:08.017 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:08.011+0000 7f314d8c0640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f31481491b0 0x7f31481e3af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:08.017 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:08.011+0000 7f314d8c0640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f31480b14e0 con 0x7f314806c8f0 2024-03-28T14:08:08.017 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:08.011+0000 7f314d8c0640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f31480b2f30 con 0x7f31481491b0 2024-03-28T14:08:08.017 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:08.011+0000 7f314d8c0640 1 -- --> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] -- mon_getmap magic: 0 -- 0x7f31480b1660 con 0x7f31480b3730 2024-03-28T14:08:08.018 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:08.011+0000 7f3146ffd640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f314806c8f0 0x7f31480af920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:08.018 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:08.011+0000 7f3146ffd640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f314806c8f0 0x7f31480af920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:34754/0 (socket says 172.21.15.104:34754) 2024-03-28T14:08:08.018 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:08.011+0000 7f3146ffd640 1 -- 172.21.15.104:0/4098048204 learned_addr learned my addr 172.21.15.104:0/4098048204 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-03-28T14:08:08.018 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:08.011+0000 7f3146ffd640 1 -- 172.21.15.104:0/4098048204 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f31480b3730 msgr2=0x7f31481e35b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:08.019 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:08.011+0000 7f3146ffd640 1 --2- 172.21.15.104:0/4098048204 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f31480b3730 0x7f31481e35b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:08.019 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:08.011+0000 7f3146ffd640 1 -- 172.21.15.104:0/4098048204 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f31481491b0 msgr2=0x7f31481e3af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:08.019 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:08.011+0000 7f3146ffd640 1 --2- 172.21.15.104:0/4098048204 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f31481491b0 0x7f31481e3af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:08.019 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:08.011+0000 7f3146ffd640 1 -- 172.21.15.104:0/4098048204 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f314809dab0 con 0x7f314806c8f0 2024-03-28T14:08:08.019 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:08.011+0000 7f3146ffd640 1 --2- 172.21.15.104:0/4098048204 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f314806c8f0 0x7f31480af920 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f313001a010 tx=0x7f313001ac20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:08:08.019 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:08.015+0000 7f313ffff640 1 -- 172.21.15.104:0/4098048204 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31300b1040 con 0x7f314806c8f0 2024-03-28T14:08:08.019 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:08.015+0000 7f313ffff640 1 -- 172.21.15.104:0/4098048204 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f31300bc070 con 0x7f314806c8f0 2024-03-28T14:08:08.019 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:08.015+0000 7f314d8c0640 1 -- 172.21.15.104:0/4098048204 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f314814efc0 con 0x7f314806c8f0 2024-03-28T14:08:08.019 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:08.015+0000 7f313ffff640 1 -- 172.21.15.104:0/4098048204 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31300b7690 con 0x7f314806c8f0 2024-03-28T14:08:08.019 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:08.015+0000 7f314d8c0640 1 -- 172.21.15.104:0/4098048204 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f314813f790 con 0x7f314806c8f0 2024-03-28T14:08:08.019 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:08.015+0000 7f314d8c0640 1 -- 172.21.15.104:0/4098048204 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f31480687c0 con 0x7f314806c8f0 2024-03-28T14:08:08.019 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:08.015+0000 7f313ffff640 1 -- 172.21.15.104:0/4098048204 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) ==== 87803+0+0 (secure 0 0 0) 0x7f31300b7830 con 0x7f314806c8f0 2024-03-28T14:08:08.020 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:08.015+0000 7f313ffff640 1 --2- 172.21.15.104:0/4098048204 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f3128067fc0 0x7f312806a480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:08.020 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:08.015+0000 7f313ffff640 1 -- 172.21.15.104:0/4098048204 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f31301207e0 con 0x7f314806c8f0 2024-03-28T14:08:08.020 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:08.015+0000 7f31467fc640 1 --2- 172.21.15.104:0/4098048204 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f3128067fc0 0x7f312806a480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:08.021 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:08.015+0000 7f31467fc640 1 --2- 172.21.15.104:0/4098048204 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f3128067fc0 0x7f312806a480 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f314813ff30 tx=0x7f3138008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:08:08.021 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:08.019+0000 7f313ffff640 1 -- 172.21.15.104:0/4098048204 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179117 (secure 0 0 0) 0x7f3130016090 con 0x7f314806c8f0 2024-03-28T14:08:08.191 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:08.187+0000 7f314d8c0640 1 -- 172.21.15.104:0/4098048204 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f314806f970 con 0x7f314806c8f0 2024-03-28T14:08:08.192 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:08.191+0000 7f313ffff640 1 -- 172.21.15.104:0/4098048204 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v21) ==== 104+0+11 (secure 0 0 0) 0x7f31300f1020 con 0x7f314806c8f0 2024-03-28T14:08:08.192 INFO:teuthology.orchestra.run.smithi104.stdout:pg_num: 32 2024-03-28T14:08:08.194 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:08.191+0000 7f314d8c0640 1 -- 172.21.15.104:0/4098048204 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f3128067fc0 msgr2=0x7f312806a480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:08.194 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:08.191+0000 7f314d8c0640 1 --2- 172.21.15.104:0/4098048204 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f3128067fc0 0x7f312806a480 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f314813ff30 tx=0x7f3138008040 comp rx=0 tx=0).stop 2024-03-28T14:08:08.194 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:08.191+0000 7f314d8c0640 1 -- 172.21.15.104:0/4098048204 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f314806c8f0 msgr2=0x7f31480af920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:08.194 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:08.191+0000 7f314d8c0640 1 --2- 172.21.15.104:0/4098048204 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f314806c8f0 0x7f31480af920 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f313001a010 tx=0x7f313001ac20 comp rx=0 tx=0).stop 2024-03-28T14:08:08.195 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:08.191+0000 7f314d8c0640 1 -- 172.21.15.104:0/4098048204 shutdown_connections 2024-03-28T14:08:08.195 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:08.191+0000 7f314d8c0640 1 --2- 172.21.15.104:0/4098048204 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f31481491b0 0x7f31481e3af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:08.195 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:08.191+0000 7f314d8c0640 1 --2- 172.21.15.104:0/4098048204 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f31480b3730 0x7f31481e35b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:08.195 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:08.191+0000 7f314d8c0640 1 --2- 172.21.15.104:0/4098048204 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f3128067fc0 0x7f312806a480 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:08.195 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:08.191+0000 7f314d8c0640 1 --2- 172.21.15.104:0/4098048204 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f314806c8f0 0x7f31480af920 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:08.195 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:08.191+0000 7f314d8c0640 1 -- 172.21.15.104:0/4098048204 >> 172.21.15.104:0/4098048204 conn(0x7f31480b8f00 msgr2=0x7f31481400c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T14:08:08.195 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:08.191+0000 7f314d8c0640 1 -- 172.21.15.104:0/4098048204 shutdown_connections 2024-03-28T14:08:08.195 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:08.191+0000 7f314d8c0640 1 -- 172.21.15.104:0/4098048204 wait complete. 2024-03-28T14:08:08.209 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-28T14:08:08.321 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:08.319+0000 7f4365774640 1 Processor -- start 2024-03-28T14:08:08.321 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:08.319+0000 7f4365774640 1 -- start start 2024-03-28T14:08:08.324 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:08.319+0000 7f4365774640 1 --2- >> v2:172.21.15.171:3300/0 conn(0x7f436006c800 0x7f436006d3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:08.324 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:08.319+0000 7f4365774640 1 --2- >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f43600b6ce0 0x7f4360152180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:08.324 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:08.319+0000 7f4365774640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4360147c30 con 0x7f4360072d20 2024-03-28T14:08:08.324 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:08.319+0000 7f4365774640 1 -- --> v2:172.21.15.171:3300/0 -- mon_getmap magic: 0 -- 0x7f4360147f10 con 0x7f436006c800 2024-03-28T14:08:08.325 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:08.319+0000 7f4365774640 1 -- --> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] -- mon_getmap magic: 0 -- 0x7f43601481f0 con 0x7f43600b6ce0 2024-03-28T14:08:08.325 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:08.319+0000 7f435effd640 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f4360072d20 0x7f4360073100 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:41626/0 (socket says 172.21.15.104:41626) 2024-03-28T14:08:08.325 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:08.319+0000 7f435effd640 1 -- 172.21.15.104:0/4283366506 learned_addr learned my addr 172.21.15.104:0/4283366506 (peer_addr_for_me v1:172.21.15.104:0/0) 2024-03-28T14:08:08.325 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:08.319+0000 7f435f7fe640 1 --2- 172.21.15.104:0/4283366506 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f43600b6ce0 0x7f4360152180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:08.325 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:08.319+0000 7f435e7fc640 1 --2- 172.21.15.104:0/4283366506 >> v2:172.21.15.171:3300/0 conn(0x7f436006c800 0x7f436006d3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:08.325 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:08.319+0000 7f435dffb640 1 -- 172.21.15.104:0/4283366506 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 585199414 0 0) 0x7f4360147c30 con 0x7f4360072d20 2024-03-28T14:08:08.325 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:08.319+0000 7f435dffb640 1 -- 172.21.15.104:0/4283366506 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f434c003530 con 0x7f4360072d20 2024-03-28T14:08:08.325 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:08.319+0000 7f435dffb640 1 -- 172.21.15.104:0/4283366506 <== mon.0 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1207947102 0 0) 0x7f434c003530 con 0x7f4360072d20 2024-03-28T14:08:08.325 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:08.319+0000 7f435dffb640 1 -- 172.21.15.104:0/4283366506 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f43600b6ce0 msgr2=0x7f4360152180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T14:08:08.325 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:08.319+0000 7f435dffb640 1 --2- 172.21.15.104:0/4283366506 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f43600b6ce0 0x7f4360152180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:08.325 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:08.319+0000 7f435dffb640 1 -- 172.21.15.104:0/4283366506 >> v2:172.21.15.171:3300/0 conn(0x7f436006c800 msgr2=0x7f436006d3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T14:08:08.326 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:08.319+0000 7f435dffb640 1 --2- 172.21.15.104:0/4283366506 >> v2:172.21.15.171:3300/0 conn(0x7f436006c800 0x7f436006d3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:08.326 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:08.319+0000 7f435dffb640 1 -- 172.21.15.104:0/4283366506 --> v1:172.21.15.104:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f43600b7890 con 0x7f4360072d20 2024-03-28T14:08:08.326 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:08.319+0000 7f435dffb640 1 -- 172.21.15.104:0/4283366506 <== mon.0 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3745580849 0 0) 0x7f4350001680 con 0x7f4360072d20 2024-03-28T14:08:08.326 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:08.319+0000 7f435dffb640 1 -- 172.21.15.104:0/4283366506 >> v1:172.21.15.104:6789/0 conn(0x7f4360072d20 legacy=0x7f4360073100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:08.326 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:08.319+0000 7f435dffb640 1 --2- 172.21.15.104:0/4283366506 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f434c0039c0 0x7f434c003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:08.326 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:08.319+0000 7f435dffb640 1 --2- 172.21.15.104:0/4283366506 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f43600b6ce0 0x7f434c004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:08.326 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:08.319+0000 7f435dffb640 1 --2- 172.21.15.104:0/4283366506 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f434c004d50 0x7f434c0051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:08.326 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:08.319+0000 7f435dffb640 1 -- 172.21.15.104:0/4283366506 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f4360147f10 con 0x7f434c0039c0 2024-03-28T14:08:08.326 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:08.319+0000 7f435dffb640 1 -- 172.21.15.104:0/4283366506 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f43601481f0 con 0x7f434c004d50 2024-03-28T14:08:08.326 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:08.319+0000 7f435dffb640 1 -- 172.21.15.104:0/4283366506 --> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] -- mon_getmap magic: 0 -- 0x7f434c00b9d0 con 0x7f43600b6ce0 2024-03-28T14:08:08.326 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:08.319+0000 7f435effd640 1 --2- 172.21.15.104:0/4283366506 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f434c0039c0 0x7f434c003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:08.326 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:08.319+0000 7f435effd640 1 -- 172.21.15.104:0/4283366506 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f43600b6ce0 msgr2=0x7f434c004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T14:08:08.326 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:08.319+0000 7f435e7fc640 1 --2- 172.21.15.104:0/4283366506 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f43600b6ce0 0x7f434c004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:08.327 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:08.319+0000 7f435f7fe640 1 --2- 172.21.15.104:0/4283366506 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f434c004d50 0x7f434c0051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:08.327 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:08.319+0000 7f435effd640 1 --2- 172.21.15.104:0/4283366506 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f43600b6ce0 0x7f434c004810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:08.327 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:08.319+0000 7f435effd640 1 -- 172.21.15.104:0/4283366506 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f434c004d50 msgr2=0x7f434c0051d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T14:08:08.327 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:08.319+0000 7f435effd640 1 --2- 172.21.15.104:0/4283366506 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f434c004d50 0x7f434c0051d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:08.327 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:08.319+0000 7f435effd640 1 -- 172.21.15.104:0/4283366506 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f434c003530 con 0x7f434c0039c0 2024-03-28T14:08:08.333 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:08.323+0000 7f435effd640 1 --2- 172.21.15.104:0/4283366506 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f434c0039c0 0x7f434c003de0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f435000c240 tx=0x7f435001cca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=84146eb7a2923a91 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:08:08.333 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:08.323+0000 7f435dffb640 1 -- 172.21.15.104:0/4283366506 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4350007d40 con 0x7f434c0039c0 2024-03-28T14:08:08.333 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:08.323+0000 7f435dffb640 1 -- 172.21.15.104:0/4283366506 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f43500a9020 con 0x7f434c0039c0 2024-03-28T14:08:08.333 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:08.323+0000 7f435dffb640 1 -- 172.21.15.104:0/4283366506 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43500a5ba0 con 0x7f434c0039c0 2024-03-28T14:08:08.333 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:08.323+0000 7f4365774640 1 -- 172.21.15.104:0/4283366506 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f434c0039c0 msgr2=0x7f434c003de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:08.333 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:08.323+0000 7f4365774640 1 --2- 172.21.15.104:0/4283366506 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f434c0039c0 0x7f434c003de0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f435000c240 tx=0x7f435001cca0 comp rx=0 tx=0).stop 2024-03-28T14:08:08.333 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:08.323+0000 7f435f7fe640 1 -- 172.21.15.104:0/4283366506 reap_dead start 2024-03-28T14:08:08.333 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:08.323+0000 7f4365774640 1 -- 172.21.15.104:0/4283366506 shutdown_connections 2024-03-28T14:08:08.333 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:08.323+0000 7f4365774640 1 -- 172.21.15.104:0/4283366506 >> 172.21.15.104:0/4283366506 conn(0x7f43600b7b10 msgr2=0x7f43600b7f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T14:08:08.333 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:08.323+0000 7f4365774640 1 -- 172.21.15.104:0/4283366506 shutdown_connections 2024-03-28T14:08:08.334 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:08.323+0000 7f4365774640 1 -- 172.21.15.104:0/4283366506 wait complete. 2024-03-28T14:08:08.334 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:08.323+0000 7f4365774640 1 Processor -- start 2024-03-28T14:08:08.334 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:08.323+0000 7f4365774640 1 -- start start 2024-03-28T14:08:08.334 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:08.327+0000 7f4365774640 1 --2- >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f434c0039c0 0x7f4360141660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:08.334 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:08.327+0000 7f4365774640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f436009b940 0x7f4360141ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:08.334 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:08.327+0000 7f4365774640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f436009c2b0 0x7f43601e39b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:08.334 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:08.327+0000 7f4365774640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f4360152b10 con 0x7f436009b940 2024-03-28T14:08:08.334 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:08.327+0000 7f4365774640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f4360152990 con 0x7f436009c2b0 2024-03-28T14:08:08.334 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:08.327+0000 7f4365774640 1 -- --> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] -- mon_getmap magic: 0 -- 0x7f4360152c90 con 0x7f434c0039c0 2024-03-28T14:08:08.334 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:08.327+0000 7f435e7fc640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f436009b940 0x7f4360141ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:08.335 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:08.327+0000 7f435e7fc640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f436009b940 0x7f4360141ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:34764/0 (socket says 172.21.15.104:34764) 2024-03-28T14:08:08.335 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:08.327+0000 7f435e7fc640 1 -- 172.21.15.104:0/3818826927 learned_addr learned my addr 172.21.15.104:0/3818826927 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-03-28T14:08:08.335 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:08.327+0000 7f435effd640 1 --2- 172.21.15.104:0/3818826927 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f434c0039c0 0x7f4360141660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:08.335 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:08.327+0000 7f435f7fe640 1 --2- 172.21.15.104:0/3818826927 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f436009c2b0 0x7f43601e39b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:08.335 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:08.327+0000 7f435e7fc640 1 -- 172.21.15.104:0/3818826927 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f434c0039c0 msgr2=0x7f4360141660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:08.335 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:08.327+0000 7f435e7fc640 1 --2- 172.21.15.104:0/3818826927 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f434c0039c0 0x7f4360141660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:08.335 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:08.327+0000 7f435e7fc640 1 -- 172.21.15.104:0/3818826927 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f436009c2b0 msgr2=0x7f43601e39b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:08.335 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:08.327+0000 7f435e7fc640 1 --2- 172.21.15.104:0/3818826927 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f436009c2b0 0x7f43601e39b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:08.335 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:08.327+0000 7f435e7fc640 1 -- 172.21.15.104:0/3818826927 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f43601420e0 con 0x7f436009b940 2024-03-28T14:08:08.335 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:08.327+0000 7f435f7fe640 1 --2- 172.21.15.104:0/3818826927 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f436009c2b0 0x7f43601e39b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T14:08:08.336 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:08.327+0000 7f435e7fc640 1 --2- 172.21.15.104:0/3818826927 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f436009b940 0x7f4360141ba0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f4348010890 tx=0x7f4348010d50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:08:08.336 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:08.327+0000 7f43577fe640 1 -- 172.21.15.104:0/3818826927 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f434800f040 con 0x7f436009b940 2024-03-28T14:08:08.336 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:08.327+0000 7f43577fe640 1 -- 172.21.15.104:0/3818826927 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f434800eb00 con 0x7f436009b940 2024-03-28T14:08:08.336 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:08.327+0000 7f4365774640 1 -- 172.21.15.104:0/3818826927 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f436009d080 con 0x7f436009b940 2024-03-28T14:08:08.336 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:08.327+0000 7f43577fe640 1 -- 172.21.15.104:0/3818826927 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f434801c400 con 0x7f436009b940 2024-03-28T14:08:08.336 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:08.327+0000 7f4365774640 1 -- 172.21.15.104:0/3818826927 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f436009ac40 con 0x7f436009b940 2024-03-28T14:08:08.336 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:08.327+0000 7f4365774640 1 -- 172.21.15.104:0/3818826927 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f436009ce80 con 0x7f436009b940 2024-03-28T14:08:08.336 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:08.327+0000 7f43577fe640 1 -- 172.21.15.104:0/3818826927 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) ==== 87803+0+0 (secure 0 0 0) 0x7f434801c5a0 con 0x7f436009b940 2024-03-28T14:08:08.336 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:08.327+0000 7f43577fe640 1 --2- 172.21.15.104:0/3818826927 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f4340067ff0 0x7f434006a4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:08.336 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:08.331+0000 7f43577fe640 1 -- 172.21.15.104:0/3818826927 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f4348086d50 con 0x7f436009b940 2024-03-28T14:08:08.336 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:08.331+0000 7f435effd640 1 --2- 172.21.15.104:0/3818826927 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f4340067ff0 0x7f434006a4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:08.337 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:08.331+0000 7f435effd640 1 --2- 172.21.15.104:0/3818826927 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f4340067ff0 0x7f434006a4b0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f43500aa000 tx=0x7f43500b0040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:08:08.337 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:08.331+0000 7f43577fe640 1 -- 172.21.15.104:0/3818826927 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179117 (secure 0 0 0) 0x7f4348018030 con 0x7f436009b940 2024-03-28T14:08:08.557 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:08.555+0000 7f4365774640 1 -- 172.21.15.104:0/3818826927 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f43600687c0 con 0x7f436009b940 2024-03-28T14:08:08.557 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:08.555+0000 7f43577fe640 1 -- 172.21.15.104:0/3818826927 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v21) ==== 100+0+11 (secure 0 0 0) 0x7f43480575f0 con 0x7f436009b940 2024-03-28T14:08:08.557 INFO:teuthology.orchestra.run.smithi104.stdout:pg_num: 64 2024-03-28T14:08:08.561 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:08.559+0000 7f4365774640 1 -- 172.21.15.104:0/3818826927 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f4340067ff0 msgr2=0x7f434006a4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:08.561 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:08.559+0000 7f4365774640 1 --2- 172.21.15.104:0/3818826927 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f4340067ff0 0x7f434006a4b0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f43500aa000 tx=0x7f43500b0040 comp rx=0 tx=0).stop 2024-03-28T14:08:08.561 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:08.559+0000 7f4365774640 1 -- 172.21.15.104:0/3818826927 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f436009b940 msgr2=0x7f4360141ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:08.561 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:08.559+0000 7f4365774640 1 --2- 172.21.15.104:0/3818826927 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f436009b940 0x7f4360141ba0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f4348010890 tx=0x7f4348010d50 comp rx=0 tx=0).stop 2024-03-28T14:08:08.561 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:08.559+0000 7f4365774640 1 -- 172.21.15.104:0/3818826927 shutdown_connections 2024-03-28T14:08:08.561 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:08.559+0000 7f4365774640 1 --2- 172.21.15.104:0/3818826927 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f436009c2b0 0x7f43601e39b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:08.561 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:08.559+0000 7f4365774640 1 --2- 172.21.15.104:0/3818826927 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f4340067ff0 0x7f434006a4b0 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:08.562 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:08.559+0000 7f4365774640 1 --2- 172.21.15.104:0/3818826927 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f436009b940 0x7f4360141ba0 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:08.562 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:08.559+0000 7f4365774640 1 --2- 172.21.15.104:0/3818826927 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f434c0039c0 0x7f4360141660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:08.562 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:08.559+0000 7f4365774640 1 -- 172.21.15.104:0/3818826927 >> 172.21.15.104:0/3818826927 conn(0x7f43600b7b10 msgr2=0x7f436006f0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T14:08:08.562 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:08.559+0000 7f4365774640 1 -- 172.21.15.104:0/3818826927 shutdown_connections 2024-03-28T14:08:08.562 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:08.559+0000 7f4365774640 1 -- 172.21.15.104:0/3818826927 wait complete. 2024-03-28T14:08:08.578 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T14:08:08.703 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:08.699+0000 7fde0689a640 1 Processor -- start 2024-03-28T14:08:08.703 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:08.699+0000 7fde0689a640 1 -- start start 2024-03-28T14:08:08.704 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:08.703+0000 7fde0689a640 1 --2- >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fde00072bf0 0x7fde000b1e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:08.704 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:08.703+0000 7fde0689a640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fde00072020 con 0x7fde00145ce0 2024-03-28T14:08:08.704 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:08.703+0000 7fde0689a640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fde00072fd0 con 0x7fde00071be0 2024-03-28T14:08:08.704 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:08.703+0000 7fde0689a640 1 -- --> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] -- mon_getmap magic: 0 -- 0x7fde00073250 con 0x7fde00072bf0 2024-03-28T14:08:08.704 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:08.703+0000 7fde04e37640 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7fde00145ce0 0x7fde0014fe90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:41636/0 (socket says 172.21.15.104:41636) 2024-03-28T14:08:08.704 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:08.703+0000 7fde04e37640 1 -- 172.21.15.104:0/1446661510 learned_addr learned my addr 172.21.15.104:0/1446661510 (peer_addr_for_me v1:172.21.15.104:0/0) 2024-03-28T14:08:08.706 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:08.703+0000 7fddfffff640 1 --2- 172.21.15.104:0/1446661510 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fde00072bf0 0x7fde000b1e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:08.706 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:08.703+0000 7fddfeffd640 1 -- 172.21.15.104:0/1446661510 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4100885979 0 0) 0x7fde00072020 con 0x7fde00145ce0 2024-03-28T14:08:08.707 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:08.703+0000 7fddfeffd640 1 -- 172.21.15.104:0/1446661510 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fddec003530 con 0x7fde00145ce0 2024-03-28T14:08:08.707 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:08.703+0000 7fddfeffd640 1 -- 172.21.15.104:0/1446661510 <== mon.0 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4198690459 0 0) 0x7fddec003530 con 0x7fde00145ce0 2024-03-28T14:08:08.707 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:08.703+0000 7fddfeffd640 1 -- 172.21.15.104:0/1446661510 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fde00072bf0 msgr2=0x7fde000b1e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T14:08:08.707 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:08.703+0000 7fddfeffd640 1 --2- 172.21.15.104:0/1446661510 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fde00072bf0 0x7fde000b1e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:08.707 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:08.703+0000 7fddfeffd640 1 -- 172.21.15.104:0/1446661510 >> v1:172.21.15.171:6789/0 conn(0x7fde00071be0 legacy=0x7fde000b12f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:08.707 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:08.703+0000 7fddfeffd640 1 -- 172.21.15.104:0/1446661510 --> v1:172.21.15.104:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fde000734a0 con 0x7fde00145ce0 2024-03-28T14:08:08.707 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:08.703+0000 7fddfeffd640 1 -- 172.21.15.104:0/1446661510 <== mon.0 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3745580849 0 0) 0x7fddf0001a00 con 0x7fde00145ce0 2024-03-28T14:08:08.707 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:08.703+0000 7fddfeffd640 1 -- 172.21.15.104:0/1446661510 >> v1:172.21.15.104:6789/0 conn(0x7fde00145ce0 legacy=0x7fde0014fe90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:08.707 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:08.703+0000 7fddfeffd640 1 --2- 172.21.15.104:0/1446661510 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fddec0039f0 0x7fddec003e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:08.707 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:08.703+0000 7fddfeffd640 1 --2- 172.21.15.104:0/1446661510 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fddec004810 0x7fddec004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:08.707 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:08.703+0000 7fddfeffd640 1 --2- 172.21.15.104:0/1446661510 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fde00072bf0 0x7fddec00b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:08.707 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:08.703+0000 7fddfeffd640 1 -- 172.21.15.104:0/1446661510 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7fde00073250 con 0x7fddec004810 2024-03-28T14:08:08.707 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:08.703+0000 7fddfeffd640 1 -- 172.21.15.104:0/1446661510 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7fddec005530 con 0x7fddec0039f0 2024-03-28T14:08:08.708 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:08.703+0000 7fddfeffd640 1 -- 172.21.15.104:0/1446661510 --> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] -- mon_getmap magic: 0 -- 0x7fddec00bf10 con 0x7fde00072bf0 2024-03-28T14:08:08.708 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:08.703+0000 7fddff7fe640 1 --2- 172.21.15.104:0/1446661510 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fddec004810 0x7fddec004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:08.708 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:08.703+0000 7fddfffff640 1 --2- 172.21.15.104:0/1446661510 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fddec0039f0 0x7fddec003e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:08.708 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:08.703+0000 7fde04e37640 1 --2- 172.21.15.104:0/1446661510 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fde00072bf0 0x7fddec00b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:08.708 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:08.703+0000 7fddff7fe640 1 -- 172.21.15.104:0/1446661510 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fde00072bf0 msgr2=0x7fddec00b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T14:08:08.708 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:08.703+0000 7fddff7fe640 1 --2- 172.21.15.104:0/1446661510 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fde00072bf0 0x7fddec00b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:08.708 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:08.703+0000 7fddff7fe640 1 -- 172.21.15.104:0/1446661510 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fddec0039f0 msgr2=0x7fddec003e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T14:08:08.708 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:08.703+0000 7fddff7fe640 1 --2- 172.21.15.104:0/1446661510 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fddec0039f0 0x7fddec003e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:08.708 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:08.703+0000 7fddff7fe640 1 -- 172.21.15.104:0/1446661510 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fddec003530 con 0x7fddec004810 2024-03-28T14:08:08.709 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:08.707+0000 7fddff7fe640 1 --2- 172.21.15.104:0/1446661510 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fddec004810 0x7fddec004c90 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fdde8001780 tx=0x7fdde801c420 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1036b4f12e67f2f2 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:08:08.709 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:08.707+0000 7fddfeffd640 1 -- 172.21.15.104:0/1446661510 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdde80013a0 con 0x7fddec004810 2024-03-28T14:08:08.709 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:08.707+0000 7fddfeffd640 1 -- 172.21.15.104:0/1446661510 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdde80052f0 con 0x7fddec004810 2024-03-28T14:08:08.709 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:08.707+0000 7fddfeffd640 1 -- 172.21.15.104:0/1446661510 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdde8003840 con 0x7fddec004810 2024-03-28T14:08:08.710 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:08.707+0000 7fde0689a640 1 -- 172.21.15.104:0/1446661510 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fddec004810 msgr2=0x7fddec004c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:08.710 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:08.707+0000 7fde0689a640 1 --2- 172.21.15.104:0/1446661510 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fddec004810 0x7fddec004c90 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fdde8001780 tx=0x7fdde801c420 comp rx=0 tx=0).stop 2024-03-28T14:08:08.710 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:08.707+0000 7fde04e37640 1 -- 172.21.15.104:0/1446661510 reap_dead start 2024-03-28T14:08:08.710 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:08.707+0000 7fde0689a640 1 -- 172.21.15.104:0/1446661510 shutdown_connections 2024-03-28T14:08:08.710 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:08.707+0000 7fde0689a640 1 -- 172.21.15.104:0/1446661510 >> 172.21.15.104:0/1446661510 conn(0x7fde000b7b10 msgr2=0x7fde000b7f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T14:08:08.710 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:08.707+0000 7fde0689a640 1 -- 172.21.15.104:0/1446661510 shutdown_connections 2024-03-28T14:08:08.710 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:08.707+0000 7fde0689a640 1 -- 172.21.15.104:0/1446661510 wait complete. 2024-03-28T14:08:08.713 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:08.707+0000 7fde0689a640 1 Processor -- start 2024-03-28T14:08:08.713 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:08.707+0000 7fde0689a640 1 -- start start 2024-03-28T14:08:08.713 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:08.711+0000 7fde0689a640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fddec004810 0x7fde001504e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:08.713 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:08.711+0000 7fde0689a640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fde0006c9c0 0x7fde0006ce40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:08.713 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:08.711+0000 7fde0689a640 1 --2- >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fde0013b250 0x7fde0013b6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:08.713 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:08.711+0000 7fde0689a640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7fde00151dc0 con 0x7fddec004810 2024-03-28T14:08:08.713 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:08.711+0000 7fde0689a640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7fde001455f0 con 0x7fde0006c9c0 2024-03-28T14:08:08.713 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:08.711+0000 7fde0689a640 1 -- --> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] -- mon_getmap magic: 0 -- 0x7fde00151f40 con 0x7fde0013b250 2024-03-28T14:08:08.713 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:08.711+0000 7fddfffff640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fddec004810 0x7fde001504e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:08.713 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:08.711+0000 7fddfffff640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fddec004810 0x7fde001504e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:34772/0 (socket says 172.21.15.104:34772) 2024-03-28T14:08:08.713 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:08.711+0000 7fddfffff640 1 -- 172.21.15.104:0/2370579963 learned_addr learned my addr 172.21.15.104:0/2370579963 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-03-28T14:08:08.713 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:08.711+0000 7fddff7fe640 1 --2- 172.21.15.104:0/2370579963 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fde0006c9c0 0x7fde0006ce40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:08.713 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:08.711+0000 7fddfffff640 1 -- 172.21.15.104:0/2370579963 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fde0013b250 msgr2=0x7fde0013b6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:08.714 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:08.711+0000 7fddfffff640 1 --2- 172.21.15.104:0/2370579963 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fde0013b250 0x7fde0013b6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:08.714 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:08.711+0000 7fddfffff640 1 -- 172.21.15.104:0/2370579963 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fde0006c9c0 msgr2=0x7fde0006ce40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:08.714 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:08.711+0000 7fddfffff640 1 --2- 172.21.15.104:0/2370579963 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fde0006c9c0 0x7fde0006ce40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:08.714 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:08.711+0000 7fddfffff640 1 -- 172.21.15.104:0/2370579963 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fde000b7fc0 con 0x7fddec004810 2024-03-28T14:08:08.714 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:08.711+0000 7fddfffff640 1 --2- 172.21.15.104:0/2370579963 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fddec004810 0x7fde001504e0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fddf80115e0 tx=0x7fddf8011aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:08:08.714 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:08.711+0000 7fddfcff9640 1 -- 172.21.15.104:0/2370579963 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fddf800f8f0 con 0x7fddec004810 2024-03-28T14:08:08.714 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:08.711+0000 7fddfcff9640 1 -- 172.21.15.104:0/2370579963 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fddf800fa90 con 0x7fddec004810 2024-03-28T14:08:08.714 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:08.711+0000 7fde0689a640 1 -- 172.21.15.104:0/2370579963 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fde0006d380 con 0x7fddec004810 2024-03-28T14:08:08.714 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:08.711+0000 7fddfcff9640 1 -- 172.21.15.104:0/2370579963 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fddf800fe60 con 0x7fddec004810 2024-03-28T14:08:08.714 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:08.711+0000 7fde0689a640 1 -- 172.21.15.104:0/2370579963 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fde0006e3d0 con 0x7fddec004810 2024-03-28T14:08:08.714 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:08.711+0000 7fde0689a640 1 -- 172.21.15.104:0/2370579963 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fde000687c0 con 0x7fddec004810 2024-03-28T14:08:08.715 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:08.711+0000 7fddfcff9640 1 -- 172.21.15.104:0/2370579963 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) ==== 87803+0+0 (secure 0 0 0) 0x7fddf8015460 con 0x7fddec004810 2024-03-28T14:08:08.715 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:08.711+0000 7fddfcff9640 1 --2- 172.21.15.104:0/2370579963 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7fdde0067fc0 0x7fdde006a480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:08.715 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:08.711+0000 7fddfcff9640 1 -- 172.21.15.104:0/2370579963 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7fddf8086e60 con 0x7fddec004810 2024-03-28T14:08:08.715 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:08.711+0000 7fddff7fe640 1 --2- 172.21.15.104:0/2370579963 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7fdde0067fc0 0x7fdde006a480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:08.716 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:08.715+0000 7fddff7fe640 1 --2- 172.21.15.104:0/2370579963 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7fdde0067fc0 0x7fdde006a480 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fdde8001750 tx=0x7fdde80011a0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:08:08.718 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:08.715+0000 7fddfcff9640 1 -- 172.21.15.104:0/2370579963 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179117 (secure 0 0 0) 0x7fde0006d380 con 0x7fddec004810 2024-03-28T14:08:08.914 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:08.911+0000 7fde0689a640 1 -- 172.21.15.104:0/2370579963 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fde00149a00 con 0x7fddec004810 2024-03-28T14:08:08.915 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:08.911+0000 7fddfcff9640 1 -- 172.21.15.104:0/2370579963 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 7 v7) ==== 93+0+3533 (secure 0 0 0) 0x7fddf8019030 con 0x7fddec004810 2024-03-28T14:08:08.915 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-03-28T14:08:08.915 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":7,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4229,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.171:6835/1775867467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6834","nonce":1775867467},{"type":"v1","addr":"172.21.15.171:6835","nonce":1775867467}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":7,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T14:07:59.027458+0000","modified":"2024-03-28T14:08:02.057923+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4245},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4245":{"gid":4245,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.104:6833/3184673325","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":3184673325},{"type":"v1","addr":"172.21.15.104:6833","nonce":3184673325}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4245,"qdb_cluster":[4245]},"id":1}]} 2024-03-28T14:08:08.915 INFO:teuthology.orchestra.run.smithi104.stderr:dumped fsmap epoch 7 2024-03-28T14:08:08.917 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:08.915+0000 7fde0689a640 1 -- 172.21.15.104:0/2370579963 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7fdde0067fc0 msgr2=0x7fdde006a480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:08.917 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:08.915+0000 7fde0689a640 1 --2- 172.21.15.104:0/2370579963 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7fdde0067fc0 0x7fdde006a480 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fdde8001750 tx=0x7fdde80011a0 comp rx=0 tx=0).stop 2024-03-28T14:08:08.917 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:08.915+0000 7fde0689a640 1 -- 172.21.15.104:0/2370579963 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fddec004810 msgr2=0x7fde001504e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:08.918 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:08.915+0000 7fde0689a640 1 --2- 172.21.15.104:0/2370579963 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fddec004810 0x7fde001504e0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fddf80115e0 tx=0x7fddf8011aa0 comp rx=0 tx=0).stop 2024-03-28T14:08:08.918 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:08.915+0000 7fde0689a640 1 -- 172.21.15.104:0/2370579963 shutdown_connections 2024-03-28T14:08:08.918 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:08.915+0000 7fde0689a640 1 --2- 172.21.15.104:0/2370579963 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fde0013b250 0x7fde0013b6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:08.918 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:08.915+0000 7fde0689a640 1 --2- 172.21.15.104:0/2370579963 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fde0006c9c0 0x7fde0006ce40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:08.918 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:08.915+0000 7fde0689a640 1 --2- 172.21.15.104:0/2370579963 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7fdde0067fc0 0x7fdde006a480 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:08.918 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:08.915+0000 7fde0689a640 1 --2- 172.21.15.104:0/2370579963 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fddec004810 0x7fde001504e0 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:08.918 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:08.915+0000 7fde0689a640 1 -- 172.21.15.104:0/2370579963 >> 172.21.15.104:0/2370579963 conn(0x7fde000b7b10 msgr2=0x7fde000ba170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T14:08:08.918 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:08.915+0000 7fde0689a640 1 -- 172.21.15.104:0/2370579963 shutdown_connections 2024-03-28T14:08:08.918 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:08.915+0000 7fde0689a640 1 -- 172.21.15.104:0/2370579963 wait complete. 2024-03-28T14:08:08.934 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-28T14:08:09.039 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:09.035+0000 7ff57d74a640 1 Processor -- start 2024-03-28T14:08:09.039 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:09.035+0000 7ff57d74a640 1 -- start start 2024-03-28T14:08:09.042 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:09.039+0000 7ff57d74a640 1 --2- >> v2:172.21.15.171:3300/0 conn(0x7ff57814afd0 0x7ff5780b3ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:09.042 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:09.039+0000 7ff57d74a640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff57802c840 con 0x7ff578148110 2024-03-28T14:08:09.042 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:09.039+0000 7ff57d74a640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff5780b4850 con 0x7ff57802c460 2024-03-28T14:08:09.042 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:09.039+0000 7ff57d74a640 1 -- --> v2:172.21.15.171:3300/0 -- mon_getmap magic: 0 -- 0x7ff5780b4d70 con 0x7ff57814afd0 2024-03-28T14:08:09.042 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:09.039+0000 7ff5777fe640 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7ff578148110 0x7ff578148560 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:41646/0 (socket says 172.21.15.104:41646) 2024-03-28T14:08:09.042 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:09.039+0000 7ff5777fe640 1 -- 172.21.15.104:0/886798043 learned_addr learned my addr 172.21.15.104:0/886798043 (peer_addr_for_me v1:172.21.15.104:0/0) 2024-03-28T14:08:09.042 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:09.039+0000 7ff56ffff640 1 --2- 172.21.15.104:0/886798043 >> v2:172.21.15.171:3300/0 conn(0x7ff57814afd0 0x7ff5780b3ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:09.043 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:09.039+0000 7ff5767fc640 1 -- 172.21.15.104:0/886798043 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3502435330 0 0) 0x7ff57802c840 con 0x7ff578148110 2024-03-28T14:08:09.043 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:09.039+0000 7ff5767fc640 1 -- 172.21.15.104:0/886798043 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff564003530 con 0x7ff578148110 2024-03-28T14:08:09.043 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:09.039+0000 7ff5767fc640 1 -- 172.21.15.104:0/886798043 <== mon.0 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3477315514 0 0) 0x7ff564003530 con 0x7ff578148110 2024-03-28T14:08:09.043 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:09.039+0000 7ff5767fc640 1 -- 172.21.15.104:0/886798043 >> v2:172.21.15.171:3300/0 conn(0x7ff57814afd0 msgr2=0x7ff5780b3ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T14:08:09.043 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:09.039+0000 7ff5767fc640 1 --2- 172.21.15.104:0/886798043 >> v2:172.21.15.171:3300/0 conn(0x7ff57814afd0 0x7ff5780b3ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:09.043 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:09.039+0000 7ff5767fc640 1 -- 172.21.15.104:0/886798043 >> v1:172.21.15.171:6789/0 conn(0x7ff57802c460 legacy=0x7ff5780b5570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:09.043 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:09.039+0000 7ff5767fc640 1 -- 172.21.15.104:0/886798043 --> v1:172.21.15.104:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff5780b5050 con 0x7ff578148110 2024-03-28T14:08:09.043 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:09.039+0000 7ff5767fc640 1 -- 172.21.15.104:0/886798043 <== mon.0 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3745580849 0 0) 0x7ff570001a00 con 0x7ff578148110 2024-03-28T14:08:09.043 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:09.039+0000 7ff5767fc640 1 -- 172.21.15.104:0/886798043 >> v1:172.21.15.104:6789/0 conn(0x7ff578148110 legacy=0x7ff578148560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:09.043 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:09.039+0000 7ff5767fc640 1 --2- 172.21.15.104:0/886798043 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff5640039c0 0x7ff564003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:09.044 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:09.039+0000 7ff5767fc640 1 --2- 172.21.15.104:0/886798043 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7ff564004810 0x7ff564004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:09.044 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:09.039+0000 7ff5767fc640 1 --2- 172.21.15.104:0/886798043 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7ff5640051d0 0x7ff56400b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:09.044 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:09.039+0000 7ff5767fc640 1 -- 172.21.15.104:0/886798043 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7ff5780b4d70 con 0x7ff5640039c0 2024-03-28T14:08:09.044 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:09.039+0000 7ff5767fc640 1 -- 172.21.15.104:0/886798043 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7ff56400c170 con 0x7ff5640051d0 2024-03-28T14:08:09.044 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:09.039+0000 7ff5767fc640 1 -- 172.21.15.104:0/886798043 --> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] -- mon_getmap magic: 0 -- 0x7ff56400c370 con 0x7ff564004810 2024-03-28T14:08:09.044 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:09.039+0000 7ff576ffd640 1 --2- 172.21.15.104:0/886798043 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff5640039c0 0x7ff564003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:09.044 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:09.039+0000 7ff5777fe640 1 --2- 172.21.15.104:0/886798043 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7ff5640051d0 0x7ff56400b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:09.044 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:09.039+0000 7ff576ffd640 1 -- 172.21.15.104:0/886798043 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7ff564004810 msgr2=0x7ff564004c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T14:08:09.044 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:09.039+0000 7ff576ffd640 1 --2- 172.21.15.104:0/886798043 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7ff564004810 0x7ff564004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:09.044 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:09.039+0000 7ff576ffd640 1 -- 172.21.15.104:0/886798043 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7ff5640051d0 msgr2=0x7ff56400b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T14:08:09.044 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:09.039+0000 7ff576ffd640 1 --2- 172.21.15.104:0/886798043 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7ff5640051d0 0x7ff56400b9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:09.044 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:09.039+0000 7ff576ffd640 1 -- 172.21.15.104:0/886798043 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff564003530 con 0x7ff5640039c0 2024-03-28T14:08:09.044 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:09.039+0000 7ff5777fe640 1 -- 172.21.15.104:0/886798043 reap_dead start 2024-03-28T14:08:09.048 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:09.039+0000 7ff576ffd640 1 --2- 172.21.15.104:0/886798043 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff5640039c0 0x7ff564003de0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7ff568001780 tx=0x7ff56801c400 comp rx=0 tx=0).ready entity=mon.0 client_cookie=87867226f9494ad5 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:08:09.048 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:09.043+0000 7ff5767fc640 1 -- 172.21.15.104:0/886798043 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5680013a0 con 0x7ff5640039c0 2024-03-28T14:08:09.048 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:09.043+0000 7ff5767fc640 1 -- 172.21.15.104:0/886798043 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff5680052f0 con 0x7ff5640039c0 2024-03-28T14:08:09.048 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:09.043+0000 7ff5767fc640 1 -- 172.21.15.104:0/886798043 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff568003840 con 0x7ff5640039c0 2024-03-28T14:08:09.048 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:09.043+0000 7ff57d74a640 1 -- 172.21.15.104:0/886798043 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff5640039c0 msgr2=0x7ff564003de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:09.048 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:09.043+0000 7ff57d74a640 1 --2- 172.21.15.104:0/886798043 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff5640039c0 0x7ff564003de0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7ff568001780 tx=0x7ff56801c400 comp rx=0 tx=0).stop 2024-03-28T14:08:09.049 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:09.043+0000 7ff57d74a640 1 -- 172.21.15.104:0/886798043 shutdown_connections 2024-03-28T14:08:09.049 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:09.043+0000 7ff57d74a640 1 --2- 172.21.15.104:0/886798043 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff5640039c0 0x7ff564003de0 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:09.049 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:09.043+0000 7ff57d74a640 1 -- 172.21.15.104:0/886798043 >> 172.21.15.104:0/886798043 conn(0x7ff5780b95f0 msgr2=0x7ff5780b99d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T14:08:09.049 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:09.043+0000 7ff57d74a640 1 -- 172.21.15.104:0/886798043 shutdown_connections 2024-03-28T14:08:09.049 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:09.043+0000 7ff57d74a640 1 -- 172.21.15.104:0/886798043 wait complete. 2024-03-28T14:08:09.049 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:09.043+0000 7ff57d74a640 1 Processor -- start 2024-03-28T14:08:09.049 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:09.043+0000 7ff57d74a640 1 -- start start 2024-03-28T14:08:09.049 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:09.043+0000 7ff57d74a640 1 --2- >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7ff5640039c0 0x7ff57809bc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:09.049 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:09.043+0000 7ff57d74a640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff5780b0900 0x7ff5780b1d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:09.050 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:09.043+0000 7ff57d74a640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7ff5780b2c60 0x7ff578099280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:09.050 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:09.043+0000 7ff57d74a640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7ff57806d180 con 0x7ff5780b0900 2024-03-28T14:08:09.050 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:09.043+0000 7ff57d74a640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7ff57806d000 con 0x7ff5780b2c60 2024-03-28T14:08:09.050 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:09.043+0000 7ff57d74a640 1 -- --> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] -- mon_getmap magic: 0 -- 0x7ff57806d300 con 0x7ff5640039c0 2024-03-28T14:08:09.050 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:09.043+0000 7ff56ffff640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff5780b0900 0x7ff5780b1d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:09.050 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:09.043+0000 7ff56ffff640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff5780b0900 0x7ff5780b1d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:34786/0 (socket says 172.21.15.104:34786) 2024-03-28T14:08:09.050 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:09.043+0000 7ff56ffff640 1 -- 172.21.15.104:0/2534182630 learned_addr learned my addr 172.21.15.104:0/2534182630 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-03-28T14:08:09.051 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:09.043+0000 7ff56ffff640 1 -- 172.21.15.104:0/2534182630 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7ff5640039c0 msgr2=0x7ff57809bc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:09.051 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:09.043+0000 7ff56ffff640 1 --2- 172.21.15.104:0/2534182630 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7ff5640039c0 0x7ff57809bc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:09.051 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:09.043+0000 7ff56ffff640 1 -- 172.21.15.104:0/2534182630 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7ff5780b2c60 msgr2=0x7ff578099280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:09.051 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:09.043+0000 7ff56ffff640 1 --2- 172.21.15.104:0/2534182630 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7ff5780b2c60 0x7ff578099280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:09.051 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:09.043+0000 7ff56ffff640 1 -- 172.21.15.104:0/2534182630 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff5780b2270 con 0x7ff5780b0900 2024-03-28T14:08:09.051 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:09.043+0000 7ff56ffff640 1 --2- 172.21.15.104:0/2534182630 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff5780b0900 0x7ff5780b1d30 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7ff56000e870 tx=0x7ff56000ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:08:09.051 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:09.043+0000 7ff56f7fe640 1 -- 172.21.15.104:0/2534182630 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff560017070 con 0x7ff5780b0900 2024-03-28T14:08:09.051 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:09.043+0000 7ff56f7fe640 1 -- 172.21.15.104:0/2534182630 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff56000c940 con 0x7ff5780b0900 2024-03-28T14:08:09.051 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:09.043+0000 7ff57d74a640 1 -- 172.21.15.104:0/2534182630 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff5780b0630 con 0x7ff5780b0900 2024-03-28T14:08:09.051 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:09.043+0000 7ff56f7fe640 1 -- 172.21.15.104:0/2534182630 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff56000cd30 con 0x7ff5780b0900 2024-03-28T14:08:09.052 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:09.043+0000 7ff57d74a640 1 -- 172.21.15.104:0/2534182630 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff57809a840 con 0x7ff5780b0900 2024-03-28T14:08:09.052 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:09.043+0000 7ff57d74a640 1 -- 172.21.15.104:0/2534182630 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff5780687c0 con 0x7ff5780b0900 2024-03-28T14:08:09.052 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:09.047+0000 7ff56f7fe640 1 -- 172.21.15.104:0/2534182630 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) ==== 87803+0+0 (secure 0 0 0) 0x7ff56001a070 con 0x7ff5780b0900 2024-03-28T14:08:09.052 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:09.047+0000 7ff56f7fe640 1 --2- 172.21.15.104:0/2534182630 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7ff558067fc0 0x7ff55806a480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:09.053 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:09.047+0000 7ff56f7fe640 1 -- 172.21.15.104:0/2534182630 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7ff5600863a0 con 0x7ff5780b0900 2024-03-28T14:08:09.053 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:09.047+0000 7ff576ffd640 1 --2- 172.21.15.104:0/2534182630 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7ff558067fc0 0x7ff55806a480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:09.053 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:09.047+0000 7ff576ffd640 1 --2- 172.21.15.104:0/2534182630 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7ff558067fc0 0x7ff55806a480 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7ff568001750 tx=0x7ff568001130 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:08:09.058 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:09.055+0000 7ff56f7fe640 1 -- 172.21.15.104:0/2534182630 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179117 (secure 0 0 0) 0x7ff5780b2270 con 0x7ff5780b0900 2024-03-28T14:08:09.217 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:09.215+0000 7ff57d74a640 1 -- 172.21.15.104:0/2534182630 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7ff578149190 con 0x7ff5780b0900 2024-03-28T14:08:09.219 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:09.219+0000 7ff56f7fe640 1 -- 172.21.15.104:0/2534182630 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) ==== 74+0+16629 (secure 0 0 0) 0x7ff560056b40 con 0x7ff5780b0900 2024-03-28T14:08:09.219 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-03-28T14:08:09.220 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":21,"fsid":"428a90f2-3351-435f-8fcf-f7e66c78a13e","created":"2024-03-28T14:07:34.862458+0000","modified":"2024-03-28T14:08:00.671587+0000","last_up_change":"2024-03-28T14:07:46.028122+0000","last_in_change":"2024-03-28T14:07:37.157486+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-28T14:07:47.622988+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-28T14:07:57.147450+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-28T14:07:57.990975+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"1cb70540-4e42-402f-a446-aaec4ca83007","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6800","nonce":3745690918},{"type":"v1","addr":"172.21.15.104:6801","nonce":3745690918}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":3745690918},{"type":"v1","addr":"172.21.15.104:6803","nonce":3745690918}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":3745690918},{"type":"v1","addr":"172.21.15.104:6807","nonce":3745690918}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":3745690918},{"type":"v1","addr":"172.21.15.104:6805","nonce":3745690918}]},"public_addr":"172.21.15.104:6801/3745690918","cluster_addr":"172.21.15.104:6803/3745690918","heartbeat_back_addr":"172.21.15.104:6807/3745690918","heartbeat_front_addr":"172.21.15.104:6805/3745690918","state":["exists","up"]},{"osd":1,"uuid":"3ab7afdd-b980-418c-81f2-004e16de3dca","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":19438674},{"type":"v1","addr":"172.21.15.104:6810","nonce":19438674}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":19438674},{"type":"v1","addr":"172.21.15.104:6814","nonce":19438674}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":19438674},{"type":"v1","addr":"172.21.15.104:6823","nonce":19438674}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":19438674},{"type":"v1","addr":"172.21.15.104:6819","nonce":19438674}]},"public_addr":"172.21.15.104:6810/19438674","cluster_addr":"172.21.15.104:6814/19438674","heartbeat_back_addr":"172.21.15.104:6823/19438674","heartbeat_front_addr":"172.21.15.104:6819/19438674","state":["exists","up"]},{"osd":2,"uuid":"3e2651c8-ecae-4a9e-a053-27240a79eac2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":2352012626},{"type":"v1","addr":"172.21.15.104:6825","nonce":2352012626}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6826","nonce":2352012626},{"type":"v1","addr":"172.21.15.104:6827","nonce":2352012626}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":2352012626},{"type":"v1","addr":"172.21.15.104:6831","nonce":2352012626}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6828","nonce":2352012626},{"type":"v1","addr":"172.21.15.104:6829","nonce":2352012626}]},"public_addr":"172.21.15.104:6825/2352012626","cluster_addr":"172.21.15.104:6827/2352012626","heartbeat_back_addr":"172.21.15.104:6831/2352012626","heartbeat_front_addr":"172.21.15.104:6829/2352012626","state":["exists","up"]},{"osd":3,"uuid":"925c7e0a-38ef-47cd-b288-8bec8456fb17","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6809","nonce":3568347381},{"type":"v1","addr":"172.21.15.104:6811","nonce":3568347381}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6813","nonce":3568347381},{"type":"v1","addr":"172.21.15.104:6815","nonce":3568347381}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":3568347381},{"type":"v1","addr":"172.21.15.104:6822","nonce":3568347381}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6817","nonce":3568347381},{"type":"v1","addr":"172.21.15.104:6818","nonce":3568347381}]},"public_addr":"172.21.15.104:6811/3568347381","cluster_addr":"172.21.15.104:6815/3568347381","heartbeat_back_addr":"172.21.15.104:6822/3568347381","heartbeat_front_addr":"172.21.15.104:6818/3568347381","state":["exists","up"]},{"osd":4,"uuid":"4a8fac86-e595-4a0e-96e3-57edb3cc3f57","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6800","nonce":2107001077},{"type":"v1","addr":"172.21.15.171:6801","nonce":2107001077}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6802","nonce":2107001077},{"type":"v1","addr":"172.21.15.171:6803","nonce":2107001077}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6806","nonce":2107001077},{"type":"v1","addr":"172.21.15.171:6807","nonce":2107001077}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6804","nonce":2107001077},{"type":"v1","addr":"172.21.15.171:6805","nonce":2107001077}]},"public_addr":"172.21.15.171:6801/2107001077","cluster_addr":"172.21.15.171:6803/2107001077","heartbeat_back_addr":"172.21.15.171:6807/2107001077","heartbeat_front_addr":"172.21.15.171:6805/2107001077","state":["exists","up"]},{"osd":5,"uuid":"e48d50b0-beae-40ae-b15a-9a84dbfe88ab","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6808","nonce":2934826925},{"type":"v1","addr":"172.21.15.171:6810","nonce":2934826925}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6812","nonce":2934826925},{"type":"v1","addr":"172.21.15.171:6814","nonce":2934826925}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6819","nonce":2934826925},{"type":"v1","addr":"172.21.15.171:6821","nonce":2934826925}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6816","nonce":2934826925},{"type":"v1","addr":"172.21.15.171:6817","nonce":2934826925}]},"public_addr":"172.21.15.171:6810/2934826925","cluster_addr":"172.21.15.171:6814/2934826925","heartbeat_back_addr":"172.21.15.171:6821/2934826925","heartbeat_front_addr":"172.21.15.171:6817/2934826925","state":["exists","up"]},{"osd":6,"uuid":"5e7c3292-9936-4f35-b267-aed9d4e49323","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6824","nonce":436742919},{"type":"v1","addr":"172.21.15.171:6825","nonce":436742919}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6826","nonce":436742919},{"type":"v1","addr":"172.21.15.171:6827","nonce":436742919}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6830","nonce":436742919},{"type":"v1","addr":"172.21.15.171:6831","nonce":436742919}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6828","nonce":436742919},{"type":"v1","addr":"172.21.15.171:6829","nonce":436742919}]},"public_addr":"172.21.15.171:6825/436742919","cluster_addr":"172.21.15.171:6827/436742919","heartbeat_back_addr":"172.21.15.171:6831/436742919","heartbeat_front_addr":"172.21.15.171:6829/436742919","state":["exists","up"]},{"osd":7,"uuid":"339b8414-d7f0-4fb0-a446-efaf635cb7ff","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6809","nonce":2055631175},{"type":"v1","addr":"172.21.15.171:6811","nonce":2055631175}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6813","nonce":2055631175},{"type":"v1","addr":"172.21.15.171:6815","nonce":2055631175}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6822","nonce":2055631175},{"type":"v1","addr":"172.21.15.171:6823","nonce":2055631175}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6818","nonce":2055631175},{"type":"v1","addr":"172.21.15.171:6820","nonce":2055631175}]},"public_addr":"172.21.15.171:6811/2055631175","cluster_addr":"172.21.15.171:6815/2055631175","heartbeat_back_addr":"172.21.15.171:6823/2055631175","heartbeat_front_addr":"172.21.15.171:6820/2055631175","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T14:07:42.668881+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T14:07:42.708397+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T14:07:42.634191+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T14:07:42.733662+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T14:07:42.651541+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T14:07:42.783181+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T14:07:42.766310+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T14:07:42.670797+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-28T14:08:09.221 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:09.219+0000 7ff57d74a640 1 -- 172.21.15.104:0/2534182630 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7ff558067fc0 msgr2=0x7ff55806a480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:09.221 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:09.219+0000 7ff57d74a640 1 --2- 172.21.15.104:0/2534182630 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7ff558067fc0 0x7ff55806a480 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7ff568001750 tx=0x7ff568001130 comp rx=0 tx=0).stop 2024-03-28T14:08:09.221 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:09.219+0000 7ff57d74a640 1 -- 172.21.15.104:0/2534182630 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff5780b0900 msgr2=0x7ff5780b1d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:09.222 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:09.219+0000 7ff57d74a640 1 --2- 172.21.15.104:0/2534182630 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff5780b0900 0x7ff5780b1d30 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7ff56000e870 tx=0x7ff56000ed30 comp rx=0 tx=0).stop 2024-03-28T14:08:09.222 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:09.219+0000 7ff57d74a640 1 -- 172.21.15.104:0/2534182630 shutdown_connections 2024-03-28T14:08:09.222 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:09.219+0000 7ff57d74a640 1 --2- 172.21.15.104:0/2534182630 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7ff5780b2c60 0x7ff578099280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:09.222 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:09.219+0000 7ff57d74a640 1 --2- 172.21.15.104:0/2534182630 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7ff558067fc0 0x7ff55806a480 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:09.222 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:09.219+0000 7ff57d74a640 1 --2- 172.21.15.104:0/2534182630 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff5780b0900 0x7ff5780b1d30 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:09.222 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:09.219+0000 7ff57d74a640 1 --2- 172.21.15.104:0/2534182630 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7ff5640039c0 0x7ff57809bc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:09.222 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:09.219+0000 7ff57d74a640 1 -- 172.21.15.104:0/2534182630 >> 172.21.15.104:0/2534182630 conn(0x7ff5780b95f0 msgr2=0x7ff578149990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T14:08:09.222 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:09.219+0000 7ff57d74a640 1 -- 172.21.15.104:0/2534182630 shutdown_connections 2024-03-28T14:08:09.222 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:09.219+0000 7ff57d74a640 1 -- 172.21.15.104:0/2534182630 wait complete. 2024-03-28T14:08:09.237 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-28T14:07:47.622988+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-28T14:07:57.147450+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-28T14:07:57.990975+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-28T14:08:09.238 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-28T14:08:09.348 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:09.347+0000 7f04938f1640 1 Processor -- start 2024-03-28T14:08:09.348 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:09.347+0000 7f04938f1640 1 -- start start 2024-03-28T14:08:09.350 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:09.347+0000 7f04938f1640 1 --2- >> v2:172.21.15.171:3300/0 conn(0x7f048c14b260 0x7f048c14b6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:09.350 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:09.347+0000 7f04938f1640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f048c0b1230 con 0x7f048c150090 2024-03-28T14:08:09.351 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:09.347+0000 7f04938f1640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f048c06bcc0 con 0x7f048c06b8e0 2024-03-28T14:08:09.351 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:09.347+0000 7f04938f1640 1 -- --> v2:172.21.15.171:3300/0 -- mon_getmap magic: 0 -- 0x7f048c14bc30 con 0x7f048c14b260 2024-03-28T14:08:09.351 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:09.347+0000 7f0491e8e640 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f048c150090 0x7f048c152480 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:41650/0 (socket says 172.21.15.104:41650) 2024-03-28T14:08:09.351 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:09.347+0000 7f0491e8e640 1 -- 172.21.15.104:0/970322489 learned_addr learned my addr 172.21.15.104:0/970322489 (peer_addr_for_me v1:172.21.15.104:0/0) 2024-03-28T14:08:09.351 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:09.347+0000 7f0490e8c640 1 --2- 172.21.15.104:0/970322489 >> v2:172.21.15.171:3300/0 conn(0x7f048c14b260 0x7f048c14b6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:09.351 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:09.347+0000 7f048bfff640 1 -- 172.21.15.104:0/970322489 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2636951751 0 0) 0x7f048c0b1230 con 0x7f048c150090 2024-03-28T14:08:09.351 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:09.347+0000 7f048bfff640 1 -- 172.21.15.104:0/970322489 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0474003530 con 0x7f048c150090 2024-03-28T14:08:09.351 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:09.347+0000 7f048bfff640 1 -- 172.21.15.104:0/970322489 <== mon.0 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 398958456 0 0) 0x7f0474003530 con 0x7f048c150090 2024-03-28T14:08:09.351 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:09.347+0000 7f048bfff640 1 -- 172.21.15.104:0/970322489 >> v2:172.21.15.171:3300/0 conn(0x7f048c14b260 msgr2=0x7f048c14b6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T14:08:09.351 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:09.347+0000 7f048bfff640 1 --2- 172.21.15.104:0/970322489 >> v2:172.21.15.171:3300/0 conn(0x7f048c14b260 0x7f048c14b6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:09.351 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:09.347+0000 7f048bfff640 1 -- 172.21.15.104:0/970322489 >> v1:172.21.15.171:6789/0 conn(0x7f048c06b8e0 legacy=0x7f048c0b0b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:09.351 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:09.347+0000 7f048bfff640 1 -- 172.21.15.104:0/970322489 --> v1:172.21.15.104:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f048c0b5000 con 0x7f048c150090 2024-03-28T14:08:09.351 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:09.347+0000 7f048bfff640 1 -- 172.21.15.104:0/970322489 <== mon.0 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3745580849 0 0) 0x7f047c001a00 con 0x7f048c150090 2024-03-28T14:08:09.352 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:09.347+0000 7f048bfff640 1 -- 172.21.15.104:0/970322489 >> v1:172.21.15.104:6789/0 conn(0x7f048c150090 legacy=0x7f048c152480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:09.352 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:09.347+0000 7f048bfff640 1 --2- 172.21.15.104:0/970322489 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f04740039c0 0x7f0474003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:09.352 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:09.347+0000 7f048bfff640 1 --2- 172.21.15.104:0/970322489 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f0474004810 0x7f0474004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:09.352 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:09.347+0000 7f048bfff640 1 --2- 172.21.15.104:0/970322489 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f04740052f0 0x7f047400b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:09.352 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:09.347+0000 7f048bfff640 1 -- 172.21.15.104:0/970322489 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f048c14bc30 con 0x7f04740039c0 2024-03-28T14:08:09.352 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:09.347+0000 7f048bfff640 1 -- 172.21.15.104:0/970322489 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f047400c1c0 con 0x7f04740052f0 2024-03-28T14:08:09.352 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:09.347+0000 7f048bfff640 1 -- 172.21.15.104:0/970322489 --> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] -- mon_getmap magic: 0 -- 0x7f047400c3c0 con 0x7f0474004810 2024-03-28T14:08:09.352 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:09.347+0000 7f049168d640 1 --2- 172.21.15.104:0/970322489 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f04740039c0 0x7f0474003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:09.352 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:09.347+0000 7f0490e8c640 1 --2- 172.21.15.104:0/970322489 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f0474004810 0x7f0474004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:09.352 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:09.347+0000 7f0491e8e640 1 --2- 172.21.15.104:0/970322489 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f04740052f0 0x7f047400b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:09.352 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:09.347+0000 7f049168d640 1 -- 172.21.15.104:0/970322489 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f0474004810 msgr2=0x7f0474004c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T14:08:09.352 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:09.347+0000 7f049168d640 1 --2- 172.21.15.104:0/970322489 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f0474004810 0x7f0474004c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:09.352 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:09.347+0000 7f049168d640 1 -- 172.21.15.104:0/970322489 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f04740052f0 msgr2=0x7f047400b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T14:08:09.352 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:09.347+0000 7f049168d640 1 --2- 172.21.15.104:0/970322489 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f04740052f0 0x7f047400b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:09.353 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:09.347+0000 7f049168d640 1 -- 172.21.15.104:0/970322489 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0474003530 con 0x7f04740039c0 2024-03-28T14:08:09.353 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:09.347+0000 7f0491e8e640 1 --2- 172.21.15.104:0/970322489 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f04740052f0 0x7f047400b9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T14:08:09.353 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:09.347+0000 7f0491e8e640 1 -- 172.21.15.104:0/970322489 reap_dead start 2024-03-28T14:08:09.354 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:09.351+0000 7f049168d640 1 --2- 172.21.15.104:0/970322489 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f04740039c0 0x7f0474003de0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f0484001750 tx=0x7f048401c3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dfef073f8bd14eb1 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:08:09.354 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:09.351+0000 7f048bfff640 1 -- 172.21.15.104:0/970322489 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04840013a0 con 0x7f04740039c0 2024-03-28T14:08:09.354 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:09.351+0000 7f048bfff640 1 -- 172.21.15.104:0/970322489 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f04840052f0 con 0x7f04740039c0 2024-03-28T14:08:09.354 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:09.351+0000 7f048bfff640 1 -- 172.21.15.104:0/970322489 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0484003840 con 0x7f04740039c0 2024-03-28T14:08:09.354 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:09.351+0000 7f04938f1640 1 -- 172.21.15.104:0/970322489 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f04740039c0 msgr2=0x7f0474003de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:09.354 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:09.351+0000 7f04938f1640 1 --2- 172.21.15.104:0/970322489 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f04740039c0 0x7f0474003de0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f0484001750 tx=0x7f048401c3c0 comp rx=0 tx=0).stop 2024-03-28T14:08:09.354 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:09.351+0000 7f04938f1640 1 -- 172.21.15.104:0/970322489 shutdown_connections 2024-03-28T14:08:09.354 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:09.351+0000 7f04938f1640 1 --2- 172.21.15.104:0/970322489 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f04740039c0 0x7f0474003de0 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:09.354 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:09.351+0000 7f04938f1640 1 -- 172.21.15.104:0/970322489 >> 172.21.15.104:0/970322489 conn(0x7f048c0ba200 msgr2=0x7f048c0ba610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T14:08:09.354 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:09.351+0000 7f04938f1640 1 -- 172.21.15.104:0/970322489 shutdown_connections 2024-03-28T14:08:09.355 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:09.351+0000 7f04938f1640 1 -- 172.21.15.104:0/970322489 wait complete. 2024-03-28T14:08:09.357 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:09.351+0000 7f04938f1640 1 Processor -- start 2024-03-28T14:08:09.357 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:09.351+0000 7f04938f1640 1 -- start start 2024-03-28T14:08:09.357 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:09.355+0000 7f04938f1640 1 --2- >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f04740039c0 0x7f048c072430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:09.357 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:09.355+0000 7f04938f1640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f048c09aa10 0x7f048c1e1110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:09.357 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:09.355+0000 7f04938f1640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f048c149de0 0x7f048c144200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:09.357 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:09.355+0000 7f04938f1640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f048c0af3f0 con 0x7f048c149de0 2024-03-28T14:08:09.357 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:09.355+0000 7f04938f1640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f048c14ba80 con 0x7f048c09aa10 2024-03-28T14:08:09.357 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:09.355+0000 7f04938f1640 1 -- --> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] -- mon_getmap magic: 0 -- 0x7f048c0af050 con 0x7f04740039c0 2024-03-28T14:08:09.357 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:09.355+0000 7f0491e8e640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f048c149de0 0x7f048c144200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:09.357 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:09.355+0000 7f0491e8e640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f048c149de0 0x7f048c144200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:34792/0 (socket says 172.21.15.104:34792) 2024-03-28T14:08:09.358 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:09.355+0000 7f0491e8e640 1 -- 172.21.15.104:0/1432619890 learned_addr learned my addr 172.21.15.104:0/1432619890 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-03-28T14:08:09.358 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:09.355+0000 7f049168d640 1 --2- 172.21.15.104:0/1432619890 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f04740039c0 0x7f048c072430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:09.358 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:09.355+0000 7f0490e8c640 1 --2- 172.21.15.104:0/1432619890 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f048c09aa10 0x7f048c1e1110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:09.358 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:09.355+0000 7f0491e8e640 1 -- 172.21.15.104:0/1432619890 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f04740039c0 msgr2=0x7f048c072430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:09.358 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:09.355+0000 7f0491e8e640 1 --2- 172.21.15.104:0/1432619890 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f04740039c0 0x7f048c072430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:09.358 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:09.355+0000 7f0491e8e640 1 -- 172.21.15.104:0/1432619890 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f048c09aa10 msgr2=0x7f048c1e1110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:09.358 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:09.355+0000 7f0491e8e640 1 --2- 172.21.15.104:0/1432619890 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f048c09aa10 0x7f048c1e1110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:09.358 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:09.355+0000 7f0491e8e640 1 -- 172.21.15.104:0/1432619890 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f048c072970 con 0x7f048c149de0 2024-03-28T14:08:09.358 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:09.355+0000 7f0490e8c640 1 --2- 172.21.15.104:0/1432619890 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f048c09aa10 0x7f048c1e1110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T14:08:09.358 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:09.355+0000 7f0491e8e640 1 --2- 172.21.15.104:0/1432619890 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f048c149de0 0x7f048c144200 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f047c000f90 tx=0x7f047c00b4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:08:09.358 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:09.355+0000 7f0489ffb640 1 -- 172.21.15.104:0/1432619890 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f047c004d70 con 0x7f048c149de0 2024-03-28T14:08:09.358 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:09.355+0000 7f0489ffb640 1 -- 172.21.15.104:0/1432619890 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f047c00baa0 con 0x7f048c149de0 2024-03-28T14:08:09.358 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:09.355+0000 7f04938f1640 1 -- 172.21.15.104:0/1432619890 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f048c09bb60 con 0x7f048c149de0 2024-03-28T14:08:09.358 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:09.355+0000 7f0489ffb640 1 -- 172.21.15.104:0/1432619890 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f047c00be90 con 0x7f048c149de0 2024-03-28T14:08:09.359 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:09.355+0000 7f04938f1640 1 -- 172.21.15.104:0/1432619890 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f048c1e0910 con 0x7f048c149de0 2024-03-28T14:08:09.359 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:09.355+0000 7f04938f1640 1 -- 172.21.15.104:0/1432619890 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f048c0687c0 con 0x7f048c149de0 2024-03-28T14:08:09.359 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:09.355+0000 7f0489ffb640 1 -- 172.21.15.104:0/1432619890 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) ==== 87803+0+0 (secure 0 0 0) 0x7f047c01b060 con 0x7f048c149de0 2024-03-28T14:08:09.360 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:09.355+0000 7f0489ffb640 1 --2- 172.21.15.104:0/1432619890 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f0478068160 0x7f047806a620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:09.360 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:09.355+0000 7f0489ffb640 1 -- 172.21.15.104:0/1432619890 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f047c0863b0 con 0x7f048c149de0 2024-03-28T14:08:09.360 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:09.355+0000 7f049168d640 1 --2- 172.21.15.104:0/1432619890 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f0478068160 0x7f047806a620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:09.360 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:09.359+0000 7f049168d640 1 --2- 172.21.15.104:0/1432619890 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f0478068160 0x7f047806a620 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f04740057f0 tx=0x7f04840011a0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:08:09.362 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:09.359+0000 7f0489ffb640 1 -- 172.21.15.104:0/1432619890 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179117 (secure 0 0 0) 0x7f048c072970 con 0x7f048c149de0 2024-03-28T14:08:09.510 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:09.507+0000 7f04938f1640 1 -- 172.21.15.104:0/1432619890 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f048c1465d0 con 0x7f048c149de0 2024-03-28T14:08:09.510 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:09.507+0000 7f0489ffb640 1 -- 172.21.15.104:0/1432619890 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v21) ==== 93+0+10 (secure 0 0 0) 0x7f047c056bf0 con 0x7f048c149de0 2024-03-28T14:08:09.511 INFO:teuthology.orchestra.run.smithi104.stdout:pg_num: 1 2024-03-28T14:08:09.512 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:09.511+0000 7f04938f1640 1 -- 172.21.15.104:0/1432619890 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f0478068160 msgr2=0x7f047806a620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:09.512 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:09.511+0000 7f04938f1640 1 --2- 172.21.15.104:0/1432619890 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f0478068160 0x7f047806a620 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f04740057f0 tx=0x7f04840011a0 comp rx=0 tx=0).stop 2024-03-28T14:08:09.512 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:09.511+0000 7f04938f1640 1 -- 172.21.15.104:0/1432619890 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f048c149de0 msgr2=0x7f048c144200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:09.512 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:09.511+0000 7f04938f1640 1 --2- 172.21.15.104:0/1432619890 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f048c149de0 0x7f048c144200 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f047c000f90 tx=0x7f047c00b4a0 comp rx=0 tx=0).stop 2024-03-28T14:08:09.512 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:09.511+0000 7f04938f1640 1 -- 172.21.15.104:0/1432619890 shutdown_connections 2024-03-28T14:08:09.512 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:09.511+0000 7f04938f1640 1 --2- 172.21.15.104:0/1432619890 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f0478068160 0x7f047806a620 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:09.512 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:09.511+0000 7f04938f1640 1 --2- 172.21.15.104:0/1432619890 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f048c149de0 0x7f048c144200 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:09.512 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:09.511+0000 7f04938f1640 1 --2- 172.21.15.104:0/1432619890 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f048c09aa10 0x7f048c1e1110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:09.512 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:09.511+0000 7f04938f1640 1 --2- 172.21.15.104:0/1432619890 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f04740039c0 0x7f048c072430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:09.512 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:09.511+0000 7f04938f1640 1 -- 172.21.15.104:0/1432619890 >> 172.21.15.104:0/1432619890 conn(0x7f048c0ba200 msgr2=0x7f048c0b6a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T14:08:09.513 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:09.511+0000 7f04938f1640 1 -- 172.21.15.104:0/1432619890 shutdown_connections 2024-03-28T14:08:09.513 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:09.511+0000 7f04938f1640 1 -- 172.21.15.104:0/1432619890 wait complete. 2024-03-28T14:08:09.527 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-28T14:08:09.640 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:09.639+0000 7fd4fcd12640 1 Processor -- start 2024-03-28T14:08:09.640 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:09.639+0000 7fd4fcd12640 1 -- start start 2024-03-28T14:08:09.643 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:09.639+0000 7fd4fcd12640 1 --2- >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fd4f802c460 0x7fd4f80b3ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:09.643 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:09.639+0000 7fd4fcd12640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd4f802c840 con 0x7fd4f80b4530 2024-03-28T14:08:09.643 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:09.639+0000 7fd4fcd12640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd4f80b4f20 con 0x7fd4f813fc50 2024-03-28T14:08:09.643 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:09.639+0000 7fd4fcd12640 1 -- --> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] -- mon_getmap magic: 0 -- 0x7fd4f80b5200 con 0x7fd4f802c460 2024-03-28T14:08:09.643 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:09.639+0000 7fd4f5d9b640 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7fd4f80b4530 0x7fd4f813f540 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:41656/0 (socket says 172.21.15.104:41656) 2024-03-28T14:08:09.643 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:09.639+0000 7fd4f5d9b640 1 -- 172.21.15.104:0/1647032676 learned_addr learned my addr 172.21.15.104:0/1647032676 (peer_addr_for_me v1:172.21.15.104:0/0) 2024-03-28T14:08:09.643 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:09.639+0000 7fd4f559a640 1 -- 172.21.15.104:0/1647032676 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1285074 0 0) 0x7fd4f802c840 con 0x7fd4f80b4530 2024-03-28T14:08:09.644 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:09.639+0000 7fd4f659c640 1 --2- 172.21.15.104:0/1647032676 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fd4f802c460 0x7fd4f80b3ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:09.644 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:09.639+0000 7fd4f559a640 1 -- 172.21.15.104:0/1647032676 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd4e4003530 con 0x7fd4f80b4530 2024-03-28T14:08:09.644 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:09.639+0000 7fd4f559a640 1 -- 172.21.15.104:0/1647032676 <== mon.0 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4121414109 0 0) 0x7fd4e4003530 con 0x7fd4f80b4530 2024-03-28T14:08:09.644 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:09.639+0000 7fd4f559a640 1 -- 172.21.15.104:0/1647032676 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fd4f802c460 msgr2=0x7fd4f80b3ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T14:08:09.644 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:09.639+0000 7fd4f559a640 1 --2- 172.21.15.104:0/1647032676 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fd4f802c460 0x7fd4f80b3ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:09.644 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:09.639+0000 7fd4f559a640 1 -- 172.21.15.104:0/1647032676 >> v1:172.21.15.171:6789/0 conn(0x7fd4f813fc50 legacy=0x7fd4f81456c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:09.644 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:09.639+0000 7fd4f559a640 1 -- 172.21.15.104:0/1647032676 --> v1:172.21.15.104:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd4f80b54e0 con 0x7fd4f80b4530 2024-03-28T14:08:09.644 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:09.639+0000 7fd4f559a640 1 -- 172.21.15.104:0/1647032676 <== mon.0 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3745580849 0 0) 0x7fd4e0001680 con 0x7fd4f80b4530 2024-03-28T14:08:09.644 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:09.639+0000 7fd4f559a640 1 -- 172.21.15.104:0/1647032676 >> v1:172.21.15.104:6789/0 conn(0x7fd4f80b4530 legacy=0x7fd4f813f540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:09.644 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:09.639+0000 7fd4f559a640 1 --2- 172.21.15.104:0/1647032676 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fd4e40039f0 0x7fd4e4003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:09.644 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:09.639+0000 7fd4f559a640 1 --2- 172.21.15.104:0/1647032676 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd4e4004810 0x7fd4e4004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:09.644 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:09.639+0000 7fd4f559a640 1 --2- 172.21.15.104:0/1647032676 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fd4f802c460 0x7fd4e400b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:09.645 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:09.639+0000 7fd4f559a640 1 -- 172.21.15.104:0/1647032676 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7fd4f80b5200 con 0x7fd4e4004810 2024-03-28T14:08:09.645 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:09.639+0000 7fd4f559a640 1 -- 172.21.15.104:0/1647032676 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7fd4e4005530 con 0x7fd4e40039f0 2024-03-28T14:08:09.645 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:09.639+0000 7fd4f559a640 1 -- 172.21.15.104:0/1647032676 --> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] -- mon_getmap magic: 0 -- 0x7fd4e400bf10 con 0x7fd4f802c460 2024-03-28T14:08:09.645 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:09.639+0000 7fd4f5d9b640 1 --2- 172.21.15.104:0/1647032676 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd4e4004810 0x7fd4e4004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:09.645 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:09.639+0000 7fd4f659c640 1 --2- 172.21.15.104:0/1647032676 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fd4e40039f0 0x7fd4e4003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:09.645 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:09.639+0000 7fd4f5d9b640 1 -- 172.21.15.104:0/1647032676 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fd4f802c460 msgr2=0x7fd4e400b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T14:08:09.645 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:09.639+0000 7fd4f6d9d640 1 --2- 172.21.15.104:0/1647032676 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fd4f802c460 0x7fd4e400b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:09.645 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:09.639+0000 7fd4f5d9b640 1 --2- 172.21.15.104:0/1647032676 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fd4f802c460 0x7fd4e400b9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:09.645 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:09.639+0000 7fd4f5d9b640 1 -- 172.21.15.104:0/1647032676 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fd4e40039f0 msgr2=0x7fd4e4003e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T14:08:09.645 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:09.639+0000 7fd4f5d9b640 1 --2- 172.21.15.104:0/1647032676 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fd4e40039f0 0x7fd4e4003e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:09.645 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:09.639+0000 7fd4f5d9b640 1 -- 172.21.15.104:0/1647032676 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd4e4003530 con 0x7fd4e4004810 2024-03-28T14:08:09.645 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:09.639+0000 7fd4f659c640 1 --2- 172.21.15.104:0/1647032676 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fd4e40039f0 0x7fd4e4003e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T14:08:09.650 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:09.643+0000 7fd4f5d9b640 1 --2- 172.21.15.104:0/1647032676 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd4e4004810 0x7fd4e4004c90 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fd4e000c190 tx=0x7fd4e001cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c041df7c486a638a server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:08:09.650 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:09.643+0000 7fd4f559a640 1 -- 172.21.15.104:0/1647032676 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd4e0007d00 con 0x7fd4e4004810 2024-03-28T14:08:09.650 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:09.643+0000 7fd4f559a640 1 -- 172.21.15.104:0/1647032676 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd4e00a9020 con 0x7fd4e4004810 2024-03-28T14:08:09.650 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:09.643+0000 7fd4f559a640 1 -- 172.21.15.104:0/1647032676 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd4e0005ae0 con 0x7fd4e4004810 2024-03-28T14:08:09.650 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:09.643+0000 7fd4fcd12640 1 -- 172.21.15.104:0/1647032676 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd4e4004810 msgr2=0x7fd4e4004c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:09.650 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:09.643+0000 7fd4fcd12640 1 --2- 172.21.15.104:0/1647032676 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd4e4004810 0x7fd4e4004c90 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fd4e000c190 tx=0x7fd4e001cc40 comp rx=0 tx=0).stop 2024-03-28T14:08:09.650 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:09.643+0000 7fd4f6d9d640 1 -- 172.21.15.104:0/1647032676 reap_dead start 2024-03-28T14:08:09.650 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:09.643+0000 7fd4fcd12640 1 -- 172.21.15.104:0/1647032676 shutdown_connections 2024-03-28T14:08:09.651 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:09.643+0000 7fd4fcd12640 1 -- 172.21.15.104:0/1647032676 >> 172.21.15.104:0/1647032676 conn(0x7fd4f80b95f0 msgr2=0x7fd4f80b99d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T14:08:09.651 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:09.643+0000 7fd4fcd12640 1 -- 172.21.15.104:0/1647032676 shutdown_connections 2024-03-28T14:08:09.651 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:09.643+0000 7fd4fcd12640 1 -- 172.21.15.104:0/1647032676 wait complete. 2024-03-28T14:08:09.651 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:09.643+0000 7fd4fcd12640 1 Processor -- start 2024-03-28T14:08:09.651 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:09.643+0000 7fd4fcd12640 1 -- start start 2024-03-28T14:08:09.651 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:09.643+0000 7fd4fcd12640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fd4e4004810 0x7fd4f80b0f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:09.651 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:09.643+0000 7fd4fcd12640 1 --2- >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fd4f80b1b90 0x7fd4f81e1a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:09.651 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:09.643+0000 7fd4fcd12640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd4f8098390 0x7fd4f8098810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:09.651 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:09.643+0000 7fd4fcd12640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7fd4f8151b30 con 0x7fd4f8098390 2024-03-28T14:08:09.651 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:09.643+0000 7fd4fcd12640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7fd4f81519b0 con 0x7fd4e4004810 2024-03-28T14:08:09.651 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:09.643+0000 7fd4fcd12640 1 -- --> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] -- mon_getmap magic: 0 -- 0x7fd4f8151cb0 con 0x7fd4f80b1b90 2024-03-28T14:08:09.651 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:09.643+0000 7fd4f6d9d640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd4f8098390 0x7fd4f8098810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:09.652 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:09.643+0000 7fd4f6d9d640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd4f8098390 0x7fd4f8098810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:34822/0 (socket says 172.21.15.104:34822) 2024-03-28T14:08:09.652 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:09.643+0000 7fd4f6d9d640 1 -- 172.21.15.104:0/1080332947 learned_addr learned my addr 172.21.15.104:0/1080332947 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-03-28T14:08:09.652 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:09.643+0000 7fd4f5d9b640 1 --2- 172.21.15.104:0/1080332947 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fd4f80b1b90 0x7fd4f81e1a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:09.652 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:09.643+0000 7fd4f659c640 1 --2- 172.21.15.104:0/1080332947 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fd4e4004810 0x7fd4f80b0f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:09.652 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:09.643+0000 7fd4f6d9d640 1 -- 172.21.15.104:0/1080332947 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fd4f80b1b90 msgr2=0x7fd4f81e1a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:09.652 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:09.643+0000 7fd4f6d9d640 1 --2- 172.21.15.104:0/1080332947 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fd4f80b1b90 0x7fd4f81e1a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:09.652 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:09.643+0000 7fd4f6d9d640 1 -- 172.21.15.104:0/1080332947 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fd4e4004810 msgr2=0x7fd4f80b0f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:09.652 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:09.643+0000 7fd4f6d9d640 1 --2- 172.21.15.104:0/1080332947 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fd4e4004810 0x7fd4f80b0f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:09.652 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:09.643+0000 7fd4f6d9d640 1 -- 172.21.15.104:0/1080332947 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd4f80b1ff0 con 0x7fd4f8098390 2024-03-28T14:08:09.652 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:09.643+0000 7fd4f6d9d640 1 --2- 172.21.15.104:0/1080332947 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd4f8098390 0x7fd4f8098810 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fd4f0000f90 tx=0x7fd4f000dc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:08:09.652 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:09.643+0000 7fd4eeffd640 1 -- 172.21.15.104:0/1080332947 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd4f0018070 con 0x7fd4f8098390 2024-03-28T14:08:09.653 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:09.643+0000 7fd4eeffd640 1 -- 172.21.15.104:0/1080332947 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd4f0003b80 con 0x7fd4f8098390 2024-03-28T14:08:09.653 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:09.643+0000 7fd4fcd12640 1 -- 172.21.15.104:0/1080332947 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd4f80b2270 con 0x7fd4f8098390 2024-03-28T14:08:09.653 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:09.643+0000 7fd4eeffd640 1 -- 172.21.15.104:0/1080332947 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd4f0005070 con 0x7fd4f8098390 2024-03-28T14:08:09.653 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:09.643+0000 7fd4fcd12640 1 -- 172.21.15.104:0/1080332947 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd4f81e24d0 con 0x7fd4f8098390 2024-03-28T14:08:09.653 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:09.647+0000 7fd4fcd12640 1 -- 172.21.15.104:0/1080332947 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd4f81e26d0 con 0x7fd4f8098390 2024-03-28T14:08:09.653 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:09.647+0000 7fd4eeffd640 1 -- 172.21.15.104:0/1080332947 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) ==== 87803+0+0 (secure 0 0 0) 0x7fd4f001b070 con 0x7fd4f8098390 2024-03-28T14:08:09.653 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:09.647+0000 7fd4eeffd640 1 --2- 172.21.15.104:0/1080332947 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7fd4d8067fc0 0x7fd4d806a480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:09.653 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:09.647+0000 7fd4eeffd640 1 -- 172.21.15.104:0/1080332947 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7fd4f0085ba0 con 0x7fd4f8098390 2024-03-28T14:08:09.653 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:09.647+0000 7fd4f659c640 1 --2- 172.21.15.104:0/1080332947 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7fd4d8067fc0 0x7fd4d806a480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:09.653 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:09.647+0000 7fd4f659c640 1 --2- 172.21.15.104:0/1080332947 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7fd4d8067fc0 0x7fd4d806a480 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fd4e800c890 tx=0x7fd4e8008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:08:09.658 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:09.655+0000 7fd4eeffd640 1 -- 172.21.15.104:0/1080332947 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179117 (secure 0 0 0) 0x7fd4f81e26d0 con 0x7fd4f8098390 2024-03-28T14:08:09.811 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:09.807+0000 7fd4fcd12640 1 -- 172.21.15.104:0/1080332947 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fd4f81e1f90 con 0x7fd4f8098390 2024-03-28T14:08:09.812 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:09.811+0000 7fd4eeffd640 1 -- 172.21.15.104:0/1080332947 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v21) ==== 104+0+11 (secure 0 0 0) 0x7fd4f80b1ff0 con 0x7fd4f8098390 2024-03-28T14:08:09.812 INFO:teuthology.orchestra.run.smithi104.stdout:pg_num: 32 2024-03-28T14:08:09.814 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:09.811+0000 7fd4fcd12640 1 -- 172.21.15.104:0/1080332947 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7fd4d8067fc0 msgr2=0x7fd4d806a480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:09.814 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:09.811+0000 7fd4fcd12640 1 --2- 172.21.15.104:0/1080332947 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7fd4d8067fc0 0x7fd4d806a480 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fd4e800c890 tx=0x7fd4e8008040 comp rx=0 tx=0).stop 2024-03-28T14:08:09.814 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:09.811+0000 7fd4fcd12640 1 -- 172.21.15.104:0/1080332947 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd4f8098390 msgr2=0x7fd4f8098810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:09.814 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:09.811+0000 7fd4fcd12640 1 --2- 172.21.15.104:0/1080332947 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd4f8098390 0x7fd4f8098810 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fd4f0000f90 tx=0x7fd4f000dc00 comp rx=0 tx=0).stop 2024-03-28T14:08:09.815 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:09.811+0000 7fd4fcd12640 1 -- 172.21.15.104:0/1080332947 shutdown_connections 2024-03-28T14:08:09.815 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:09.811+0000 7fd4fcd12640 1 --2- 172.21.15.104:0/1080332947 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7fd4d8067fc0 0x7fd4d806a480 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:09.815 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:09.811+0000 7fd4fcd12640 1 --2- 172.21.15.104:0/1080332947 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd4f8098390 0x7fd4f8098810 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:09.815 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:09.811+0000 7fd4fcd12640 1 --2- 172.21.15.104:0/1080332947 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fd4f80b1b90 0x7fd4f81e1a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:09.815 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:09.811+0000 7fd4fcd12640 1 --2- 172.21.15.104:0/1080332947 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fd4e4004810 0x7fd4f80b0f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:09.815 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:09.811+0000 7fd4fcd12640 1 -- 172.21.15.104:0/1080332947 >> 172.21.15.104:0/1080332947 conn(0x7fd4f80b95f0 msgr2=0x7fd4f813e1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T14:08:09.815 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:09.811+0000 7fd4fcd12640 1 -- 172.21.15.104:0/1080332947 shutdown_connections 2024-03-28T14:08:09.815 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:09.811+0000 7fd4fcd12640 1 -- 172.21.15.104:0/1080332947 wait complete. 2024-03-28T14:08:09.829 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-28T14:08:09.935 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:09.931+0000 7f6c56abd640 1 Processor -- start 2024-03-28T14:08:09.935 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:09.931+0000 7f6c56abd640 1 -- start start 2024-03-28T14:08:09.936 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:09.935+0000 7f6c56abd640 1 --2- >> v2:172.21.15.171:3300/0 conn(0x7f6c5006e4d0 0x7f6c50147150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:09.936 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:09.935+0000 7f6c56abd640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6c5013e360 con 0x7f6c500b03a0 2024-03-28T14:08:09.936 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:09.935+0000 7f6c56abd640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6c500b5cf0 con 0x7f6c5013df80 2024-03-28T14:08:09.936 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:09.935+0000 7f6c56abd640 1 -- --> v2:172.21.15.171:3300/0 -- mon_getmap magic: 0 -- 0x7f6c500ae980 con 0x7f6c5006e4d0 2024-03-28T14:08:09.936 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:09.935+0000 7f6c5505a640 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f6c500b03a0 0x7f6c5014e4e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:41660/0 (socket says 172.21.15.104:41660) 2024-03-28T14:08:09.936 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:09.935+0000 7f6c5505a640 1 -- 172.21.15.104:0/2662198713 learned_addr learned my addr 172.21.15.104:0/2662198713 (peer_addr_for_me v1:172.21.15.104:0/0) 2024-03-28T14:08:09.938 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:09.935+0000 7f6c4ffff640 1 --2- 172.21.15.104:0/2662198713 >> v2:172.21.15.171:3300/0 conn(0x7f6c5006e4d0 0x7f6c50147150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:09.938 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:09.935+0000 7f6c4f7fe640 1 -- 172.21.15.104:0/2662198713 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1358141376 0 0) 0x7f6c5013e360 con 0x7f6c500b03a0 2024-03-28T14:08:09.938 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:09.935+0000 7f6c4f7fe640 1 -- 172.21.15.104:0/2662198713 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6c40003530 con 0x7f6c500b03a0 2024-03-28T14:08:09.939 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:09.935+0000 7f6c4f7fe640 1 -- 172.21.15.104:0/2662198713 <== mon.0 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1475445882 0 0) 0x7f6c40003530 con 0x7f6c500b03a0 2024-03-28T14:08:09.939 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:09.935+0000 7f6c4f7fe640 1 -- 172.21.15.104:0/2662198713 >> v2:172.21.15.171:3300/0 conn(0x7f6c5006e4d0 msgr2=0x7f6c50147150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T14:08:09.939 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:09.935+0000 7f6c4f7fe640 1 --2- 172.21.15.104:0/2662198713 >> v2:172.21.15.171:3300/0 conn(0x7f6c5006e4d0 0x7f6c50147150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:09.939 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:09.935+0000 7f6c4f7fe640 1 -- 172.21.15.104:0/2662198713 >> v1:172.21.15.171:6789/0 conn(0x7f6c5013df80 legacy=0x7f6c5006d000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:09.939 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:09.935+0000 7f6c4f7fe640 1 -- 172.21.15.104:0/2662198713 --> v1:172.21.15.104:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6c500b5830 con 0x7f6c500b03a0 2024-03-28T14:08:09.939 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:09.935+0000 7f6c4f7fe640 1 -- 172.21.15.104:0/2662198713 <== mon.0 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3745580849 0 0) 0x7f6c48001a00 con 0x7f6c500b03a0 2024-03-28T14:08:09.939 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:09.935+0000 7f6c4f7fe640 1 -- 172.21.15.104:0/2662198713 >> v1:172.21.15.104:6789/0 conn(0x7f6c500b03a0 legacy=0x7f6c5014e4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:09.939 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:09.935+0000 7f6c4f7fe640 1 --2- 172.21.15.104:0/2662198713 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f6c400039c0 0x7f6c40003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:09.939 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:09.935+0000 7f6c4f7fe640 1 --2- 172.21.15.104:0/2662198713 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f6c40004810 0x7f6c40004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:09.939 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:09.935+0000 7f6c4f7fe640 1 --2- 172.21.15.104:0/2662198713 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6c400051d0 0x7f6c4000b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:09.939 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:09.935+0000 7f6c4f7fe640 1 -- 172.21.15.104:0/2662198713 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f6c500ae980 con 0x7f6c400051d0 2024-03-28T14:08:09.939 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:09.935+0000 7f6c4f7fe640 1 -- 172.21.15.104:0/2662198713 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f6c4000c070 con 0x7f6c40004810 2024-03-28T14:08:09.939 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:09.935+0000 7f6c4f7fe640 1 -- 172.21.15.104:0/2662198713 --> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] -- mon_getmap magic: 0 -- 0x7f6c4000c270 con 0x7f6c400039c0 2024-03-28T14:08:09.939 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:09.935+0000 7f6c5505a640 1 --2- 172.21.15.104:0/2662198713 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6c400051d0 0x7f6c4000b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:09.940 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:09.935+0000 7f6c4ffff640 1 --2- 172.21.15.104:0/2662198713 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f6c40004810 0x7f6c40004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:09.940 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:09.935+0000 7f6c54859640 1 --2- 172.21.15.104:0/2662198713 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f6c400039c0 0x7f6c40003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:09.940 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:09.935+0000 7f6c5505a640 1 -- 172.21.15.104:0/2662198713 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f6c400039c0 msgr2=0x7f6c40003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T14:08:09.940 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:09.935+0000 7f6c5505a640 1 --2- 172.21.15.104:0/2662198713 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f6c400039c0 0x7f6c40003de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:09.940 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:09.935+0000 7f6c5505a640 1 -- 172.21.15.104:0/2662198713 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f6c40004810 msgr2=0x7f6c40004c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T14:08:09.940 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:09.935+0000 7f6c5505a640 1 --2- 172.21.15.104:0/2662198713 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f6c40004810 0x7f6c40004c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:09.940 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:09.935+0000 7f6c5505a640 1 -- 172.21.15.104:0/2662198713 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6c40003530 con 0x7f6c400051d0 2024-03-28T14:08:09.940 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:09.935+0000 7f6c54859640 1 --2- 172.21.15.104:0/2662198713 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f6c400039c0 0x7f6c40003de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T14:08:09.943 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:09.935+0000 7f6c5505a640 1 -- 172.21.15.104:0/2662198713 reap_dead start 2024-03-28T14:08:09.943 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:09.939+0000 7f6c5505a640 1 --2- 172.21.15.104:0/2662198713 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6c400051d0 0x7f6c4000b9d0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f6c4800a850 tx=0x7f6c4801d3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3388ab38d52061a3 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:08:09.943 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:09.939+0000 7f6c4f7fe640 1 -- 172.21.15.104:0/2662198713 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c4800a4a0 con 0x7f6c400051d0 2024-03-28T14:08:09.943 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:09.939+0000 7f6c4f7fe640 1 -- 172.21.15.104:0/2662198713 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6c480ad650 con 0x7f6c400051d0 2024-03-28T14:08:09.943 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:09.939+0000 7f6c4f7fe640 1 -- 172.21.15.104:0/2662198713 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c480ada40 con 0x7f6c400051d0 2024-03-28T14:08:09.943 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:09.939+0000 7f6c56abd640 1 -- 172.21.15.104:0/2662198713 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6c400051d0 msgr2=0x7f6c4000b9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:09.944 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:09.939+0000 7f6c56abd640 1 --2- 172.21.15.104:0/2662198713 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6c400051d0 0x7f6c4000b9d0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f6c4800a850 tx=0x7f6c4801d3a0 comp rx=0 tx=0).stop 2024-03-28T14:08:09.944 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:09.939+0000 7f6c56abd640 1 -- 172.21.15.104:0/2662198713 shutdown_connections 2024-03-28T14:08:09.944 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:09.939+0000 7f6c56abd640 1 --2- 172.21.15.104:0/2662198713 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6c400051d0 0x7f6c4000b9d0 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:09.944 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:09.939+0000 7f6c56abd640 1 -- 172.21.15.104:0/2662198713 >> 172.21.15.104:0/2662198713 conn(0x7f6c500b95f0 msgr2=0x7f6c500b99d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T14:08:09.944 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:09.939+0000 7f6c56abd640 1 -- 172.21.15.104:0/2662198713 shutdown_connections 2024-03-28T14:08:09.944 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:09.939+0000 7f6c56abd640 1 -- 172.21.15.104:0/2662198713 wait complete. 2024-03-28T14:08:09.944 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:09.939+0000 7f6c56abd640 1 Processor -- start 2024-03-28T14:08:09.944 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:09.939+0000 7f6c56abd640 1 -- start start 2024-03-28T14:08:09.944 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:09.939+0000 7f6c56abd640 1 --2- >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f6c400051d0 0x7f6c5009d610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:09.944 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:09.939+0000 7f6c56abd640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6c50073500 0x7f6c50141ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:09.944 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:09.939+0000 7f6c56abd640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f6c50073960 0x7f6c50142430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:09.944 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:09.939+0000 7f6c56abd640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f6c500b5aa0 con 0x7f6c50073500 2024-03-28T14:08:09.944 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:09.939+0000 7f6c56abd640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f6c500aff10 con 0x7f6c50073960 2024-03-28T14:08:09.945 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:09.939+0000 7f6c56abd640 1 -- --> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] -- mon_getmap magic: 0 -- 0x7f6c5006dba0 con 0x7f6c400051d0 2024-03-28T14:08:09.945 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:09.939+0000 7f6c4ffff640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6c50073500 0x7f6c50141ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:09.945 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:09.939+0000 7f6c4ffff640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6c50073500 0x7f6c50141ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:34836/0 (socket says 172.21.15.104:34836) 2024-03-28T14:08:09.945 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:09.939+0000 7f6c4ffff640 1 -- 172.21.15.104:0/2611992803 learned_addr learned my addr 172.21.15.104:0/2611992803 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-03-28T14:08:09.945 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:09.939+0000 7f6c54859640 1 --2- 172.21.15.104:0/2611992803 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f6c400051d0 0x7f6c5009d610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:09.945 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:09.939+0000 7f6c5505a640 1 --2- 172.21.15.104:0/2611992803 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f6c50073960 0x7f6c50142430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:09.945 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:09.939+0000 7f6c4ffff640 1 -- 172.21.15.104:0/2611992803 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f6c400051d0 msgr2=0x7f6c5009d610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:09.945 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:09.939+0000 7f6c4ffff640 1 --2- 172.21.15.104:0/2611992803 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f6c400051d0 0x7f6c5009d610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:09.945 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:09.939+0000 7f6c4ffff640 1 -- 172.21.15.104:0/2611992803 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f6c50073960 msgr2=0x7f6c50142430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:09.945 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:09.939+0000 7f6c4ffff640 1 --2- 172.21.15.104:0/2611992803 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f6c50073960 0x7f6c50142430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:09.945 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:09.939+0000 7f6c4ffff640 1 -- 172.21.15.104:0/2611992803 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6c50144470 con 0x7f6c50073500 2024-03-28T14:08:09.945 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:09.939+0000 7f6c4ffff640 1 --2- 172.21.15.104:0/2611992803 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6c50073500 0x7f6c50141ef0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f6c3c00b3b0 tx=0x7f6c3c003ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:08:09.945 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:09.939+0000 7f6c4d7fa640 1 -- 172.21.15.104:0/2611992803 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c3c01c070 con 0x7f6c50073500 2024-03-28T14:08:09.946 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:09.939+0000 7f6c4d7fa640 1 -- 172.21.15.104:0/2611992803 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6c3c00db90 con 0x7f6c50073500 2024-03-28T14:08:09.946 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:09.939+0000 7f6c4d7fa640 1 -- 172.21.15.104:0/2611992803 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c3c00a070 con 0x7f6c50073500 2024-03-28T14:08:09.946 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:09.939+0000 7f6c56abd640 1 -- 172.21.15.104:0/2611992803 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6c501447b0 con 0x7f6c50073500 2024-03-28T14:08:09.946 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:09.939+0000 7f6c56abd640 1 -- 172.21.15.104:0/2611992803 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6c50141a50 con 0x7f6c50073500 2024-03-28T14:08:09.946 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:09.943+0000 7f6c56abd640 1 -- 172.21.15.104:0/2611992803 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6c500687c0 con 0x7f6c50073500 2024-03-28T14:08:09.947 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:09.943+0000 7f6c4d7fa640 1 -- 172.21.15.104:0/2611992803 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) ==== 87803+0+0 (secure 0 0 0) 0x7f6c3c01f070 con 0x7f6c50073500 2024-03-28T14:08:09.947 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:09.943+0000 7f6c4d7fa640 1 --2- 172.21.15.104:0/2611992803 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f6c34067fc0 0x7f6c3406a480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:09.947 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:09.943+0000 7f6c4d7fa640 1 -- 172.21.15.104:0/2611992803 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f6c3c086840 con 0x7f6c50073500 2024-03-28T14:08:09.947 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:09.943+0000 7f6c54859640 1 --2- 172.21.15.104:0/2611992803 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f6c34067fc0 0x7f6c3406a480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:09.947 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:09.943+0000 7f6c54859640 1 --2- 172.21.15.104:0/2611992803 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f6c34067fc0 0x7f6c3406a480 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f6c44000f70 tx=0x7f6c44009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:08:09.949 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:09.947+0000 7f6c4d7fa640 1 -- 172.21.15.104:0/2611992803 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179117 (secure 0 0 0) 0x7f6c501447b0 con 0x7f6c50073500 2024-03-28T14:08:10.097 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:10.095+0000 7f6c56abd640 1 -- 172.21.15.104:0/2611992803 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f6c500b8de0 con 0x7f6c50073500 2024-03-28T14:08:10.098 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:10.095+0000 7f6c4d7fa640 1 -- 172.21.15.104:0/2611992803 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v21) ==== 100+0+11 (secure 0 0 0) 0x7f6c3c056ff0 con 0x7f6c50073500 2024-03-28T14:08:10.098 INFO:teuthology.orchestra.run.smithi104.stdout:pg_num: 64 2024-03-28T14:08:10.100 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:10.099+0000 7f6c56abd640 1 -- 172.21.15.104:0/2611992803 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f6c34067fc0 msgr2=0x7f6c3406a480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:10.100 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:10.099+0000 7f6c56abd640 1 --2- 172.21.15.104:0/2611992803 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f6c34067fc0 0x7f6c3406a480 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f6c44000f70 tx=0x7f6c44009040 comp rx=0 tx=0).stop 2024-03-28T14:08:10.100 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:10.099+0000 7f6c56abd640 1 -- 172.21.15.104:0/2611992803 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6c50073500 msgr2=0x7f6c50141ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:10.100 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:10.099+0000 7f6c56abd640 1 --2- 172.21.15.104:0/2611992803 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6c50073500 0x7f6c50141ef0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f6c3c00b3b0 tx=0x7f6c3c003ab0 comp rx=0 tx=0).stop 2024-03-28T14:08:10.100 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:10.099+0000 7f6c56abd640 1 -- 172.21.15.104:0/2611992803 shutdown_connections 2024-03-28T14:08:10.100 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:10.099+0000 7f6c56abd640 1 --2- 172.21.15.104:0/2611992803 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f6c50073960 0x7f6c50142430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:10.100 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:10.099+0000 7f6c56abd640 1 --2- 172.21.15.104:0/2611992803 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f6c34067fc0 0x7f6c3406a480 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:10.101 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:10.099+0000 7f6c56abd640 1 --2- 172.21.15.104:0/2611992803 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6c50073500 0x7f6c50141ef0 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:10.101 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:10.099+0000 7f6c56abd640 1 --2- 172.21.15.104:0/2611992803 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f6c400051d0 0x7f6c5009d610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:10.101 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:10.099+0000 7f6c56abd640 1 -- 172.21.15.104:0/2611992803 >> 172.21.15.104:0/2611992803 conn(0x7f6c500b95f0 msgr2=0x7f6c5006d000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T14:08:10.101 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:10.099+0000 7f6c56abd640 1 -- 172.21.15.104:0/2611992803 shutdown_connections 2024-03-28T14:08:10.101 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:10.099+0000 7f6c56abd640 1 -- 172.21.15.104:0/2611992803 wait complete. 2024-03-28T14:08:10.115 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T14:08:10.239 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:10.235+0000 7f9496e4a640 1 Processor -- start 2024-03-28T14:08:10.239 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:10.235+0000 7f9496e4a640 1 -- start start 2024-03-28T14:08:10.241 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:10.239+0000 7f9496e4a640 1 --2- >> v2:172.21.15.171:3300/0 conn(0x7f9490072f80 0x7f949006d3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:10.241 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:10.239+0000 7f9496e4a640 1 --2- >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f949006d8e0 0x7f94900b6a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:10.241 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:10.239+0000 7f9496e4a640 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f94900b6f60 0x7f949014ae50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:10.241 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:10.239+0000 7f9496e4a640 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 -- 0x7f9490148380 con 0x7f94900b6f60 2024-03-28T14:08:10.241 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:10.239+0000 7f9496e4a640 1 -- --> v2:172.21.15.171:3300/0 -- mon_getmap magic: 0 -- 0x7f9490148810 con 0x7f9490072f80 2024-03-28T14:08:10.241 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:10.239+0000 7f9496e4a640 1 -- --> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] -- mon_getmap magic: 0 -- 0x7f9490147d30 con 0x7f949006d8e0 2024-03-28T14:08:10.241 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:10.239+0000 7f94953e7640 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f94900b6f60 0x7f949014ae50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:10.241 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:10.239+0000 7f94953e7640 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f94900b6f60 0x7f949014ae50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:34842/0 (socket says 172.21.15.104:34842) 2024-03-28T14:08:10.242 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:10.239+0000 7f94953e7640 1 -- 172.21.15.104:0/3549648334 learned_addr learned my addr 172.21.15.104:0/3549648334 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-03-28T14:08:10.242 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:10.239+0000 7f948ffff640 1 --2- 172.21.15.104:0/3549648334 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f949006d8e0 0x7f94900b6a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:10.242 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:10.239+0000 7f9494be6640 1 --2- 172.21.15.104:0/3549648334 >> v2:172.21.15.171:3300/0 conn(0x7f9490072f80 0x7f949006d3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:10.242 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:10.239+0000 7f94953e7640 1 -- 172.21.15.104:0/3549648334 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f949006d8e0 msgr2=0x7f94900b6a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T14:08:10.242 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:10.239+0000 7f94953e7640 1 --2- 172.21.15.104:0/3549648334 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f949006d8e0 0x7f94900b6a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:10.242 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:10.239+0000 7f94953e7640 1 -- 172.21.15.104:0/3549648334 >> v2:172.21.15.171:3300/0 conn(0x7f9490072f80 msgr2=0x7f949006d3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T14:08:10.242 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:10.239+0000 7f94953e7640 1 --2- 172.21.15.104:0/3549648334 >> v2:172.21.15.171:3300/0 conn(0x7f9490072f80 0x7f949006d3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:10.242 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:10.239+0000 7f94953e7640 1 -- 172.21.15.104:0/3549648334 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9490148010 con 0x7f94900b6f60 2024-03-28T14:08:10.244 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:10.239+0000 7f94953e7640 1 --2- 172.21.15.104:0/3549648334 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f94900b6f60 0x7f949014ae50 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f9490147f40 tx=0x7f948001c400 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c66314d68c51257d server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:08:10.244 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:10.239+0000 7f948f7fe640 1 -- 172.21.15.104:0/3549648334 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94800b0070 con 0x7f94900b6f60 2024-03-28T14:08:10.244 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:10.239+0000 7f948f7fe640 1 -- 172.21.15.104:0/3549648334 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f94800ab600 con 0x7f94900b6f60 2024-03-28T14:08:10.244 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:10.239+0000 7f948f7fe640 1 -- 172.21.15.104:0/3549648334 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94800ab9d0 con 0x7f94900b6f60 2024-03-28T14:08:10.244 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:10.243+0000 7f9496e4a640 1 -- 172.21.15.104:0/3549648334 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f94900b6f60 msgr2=0x7f949014ae50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:10.245 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:10.243+0000 7f9496e4a640 1 --2- 172.21.15.104:0/3549648334 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f94900b6f60 0x7f949014ae50 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f9490147f40 tx=0x7f948001c400 comp rx=0 tx=0).stop 2024-03-28T14:08:10.245 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:10.243+0000 7f9496e4a640 1 -- 172.21.15.104:0/3549648334 shutdown_connections 2024-03-28T14:08:10.245 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:10.243+0000 7f9496e4a640 1 --2- 172.21.15.104:0/3549648334 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f949006d8e0 0x7f94900b6a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:10.245 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:10.243+0000 7f9496e4a640 1 --2- 172.21.15.104:0/3549648334 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f94900b6f60 0x7f949014ae50 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:10.245 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:10.243+0000 7f9496e4a640 1 --2- 172.21.15.104:0/3549648334 >> v2:172.21.15.171:3300/0 conn(0x7f9490072f80 0x7f949006d3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:10.245 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:10.243+0000 7f9496e4a640 1 -- 172.21.15.104:0/3549648334 >> 172.21.15.104:0/3549648334 conn(0x7f94900b7b10 msgr2=0x7f94900b7f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T14:08:10.245 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:10.243+0000 7f9496e4a640 1 -- 172.21.15.104:0/3549648334 shutdown_connections 2024-03-28T14:08:10.245 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:10.243+0000 7f9496e4a640 1 -- 172.21.15.104:0/3549648334 wait complete. 2024-03-28T14:08:10.248 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:10.243+0000 7f9496e4a640 1 Processor -- start 2024-03-28T14:08:10.248 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:10.243+0000 7f9496e4a640 1 -- start start 2024-03-28T14:08:10.248 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:10.243+0000 7f9496e4a640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f949006d8e0 0x7f949013a5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:10.248 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:10.243+0000 7f9496e4a640 1 --2- >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f9490072f80 0x7f94901e3d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:10.249 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:10.243+0000 7f9496e4a640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f94900b6f60 0x7f9490139320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:10.249 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:10.243+0000 7f9496e4a640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f9490154190 con 0x7f94900b6f60 2024-03-28T14:08:10.249 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:10.243+0000 7f9496e4a640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f9490147af0 con 0x7f949006d8e0 2024-03-28T14:08:10.249 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:10.243+0000 7f9496e4a640 1 -- --> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] -- mon_getmap magic: 0 -- 0x7f9490154310 con 0x7f9490072f80 2024-03-28T14:08:10.249 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:10.243+0000 7f94953e7640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f94900b6f60 0x7f9490139320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:10.249 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:10.243+0000 7f94953e7640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f94900b6f60 0x7f9490139320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:34856/0 (socket says 172.21.15.104:34856) 2024-03-28T14:08:10.249 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:10.243+0000 7f94953e7640 1 -- 172.21.15.104:0/3821392133 learned_addr learned my addr 172.21.15.104:0/3821392133 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-03-28T14:08:10.249 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:10.243+0000 7f9494be6640 1 --2- 172.21.15.104:0/3821392133 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f949006d8e0 0x7f949013a5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:10.249 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:10.243+0000 7f948ffff640 1 --2- 172.21.15.104:0/3821392133 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f9490072f80 0x7f94901e3d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:10.249 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:10.243+0000 7f94953e7640 1 -- 172.21.15.104:0/3821392133 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f9490072f80 msgr2=0x7f94901e3d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:10.249 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:10.243+0000 7f94953e7640 1 --2- 172.21.15.104:0/3821392133 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f9490072f80 0x7f94901e3d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:10.250 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:10.243+0000 7f94953e7640 1 -- 172.21.15.104:0/3821392133 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f949006d8e0 msgr2=0x7f949013a5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:10.250 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:10.243+0000 7f94953e7640 1 --2- 172.21.15.104:0/3821392133 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f949006d8e0 0x7f949013a5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:10.250 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:10.243+0000 7f94953e7640 1 -- 172.21.15.104:0/3821392133 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9490149700 con 0x7f94900b6f60 2024-03-28T14:08:10.250 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:10.243+0000 7f94953e7640 1 --2- 172.21.15.104:0/3821392133 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f94900b6f60 0x7f9490139320 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f94800ae040 tx=0x7f94800191a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:08:10.250 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:10.243+0000 7f948d7fa640 1 -- 172.21.15.104:0/3821392133 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94800b8020 con 0x7f94900b6f60 2024-03-28T14:08:10.250 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:10.243+0000 7f948d7fa640 1 -- 172.21.15.104:0/3821392133 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f94800b4610 con 0x7f94900b6f60 2024-03-28T14:08:10.250 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:10.243+0000 7f948d7fa640 1 -- 172.21.15.104:0/3821392133 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94800b0040 con 0x7f94900b6f60 2024-03-28T14:08:10.250 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:10.243+0000 7f9496e4a640 1 -- 172.21.15.104:0/3821392133 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f949013aaf0 con 0x7f94900b6f60 2024-03-28T14:08:10.251 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:10.243+0000 7f9496e4a640 1 -- 172.21.15.104:0/3821392133 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f949013d330 con 0x7f94900b6f60 2024-03-28T14:08:10.251 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:10.247+0000 7f9496e4a640 1 -- 172.21.15.104:0/3821392133 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f94900687c0 con 0x7f94900b6f60 2024-03-28T14:08:10.251 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:10.247+0000 7f948d7fa640 1 -- 172.21.15.104:0/3821392133 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) ==== 87803+0+0 (secure 0 0 0) 0x7f94800bb020 con 0x7f94900b6f60 2024-03-28T14:08:10.252 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:10.247+0000 7f948d7fa640 1 --2- 172.21.15.104:0/3821392133 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f9470067ff0 0x7f947006a4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:10.252 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:10.247+0000 7f948d7fa640 1 -- 172.21.15.104:0/3821392133 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f948011feb0 con 0x7f94900b6f60 2024-03-28T14:08:10.252 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:10.247+0000 7f9494be6640 1 --2- 172.21.15.104:0/3821392133 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f9470067ff0 0x7f947006a4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:10.252 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:10.247+0000 7f9494be6640 1 --2- 172.21.15.104:0/3821392133 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f9470067ff0 0x7f947006a4b0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f9488003fd0 tx=0x7f94880039a0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:08:10.253 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:10.251+0000 7f948d7fa640 1 -- 172.21.15.104:0/3821392133 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179117 (secure 0 0 0) 0x7f94800c2070 con 0x7f94900b6f60 2024-03-28T14:08:10.491 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:10.487+0000 7f9496e4a640 1 -- 172.21.15.104:0/3821392133 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f94900b9150 con 0x7f94900b6f60 2024-03-28T14:08:10.492 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:10.491+0000 7f948d7fa640 1 -- 172.21.15.104:0/3821392133 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 7 v7) ==== 93+0+3533 (secure 0 0 0) 0x7f9480016110 con 0x7f94900b6f60 2024-03-28T14:08:10.492 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-03-28T14:08:10.493 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":7,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4229,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.171:6835/1775867467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6834","nonce":1775867467},{"type":"v1","addr":"172.21.15.171:6835","nonce":1775867467}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":7,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T14:07:59.027458+0000","modified":"2024-03-28T14:08:02.057923+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4245},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4245":{"gid":4245,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.104:6833/3184673325","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":3184673325},{"type":"v1","addr":"172.21.15.104:6833","nonce":3184673325}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4245,"qdb_cluster":[4245]},"id":1}]} 2024-03-28T14:08:10.493 INFO:teuthology.orchestra.run.smithi104.stderr:dumped fsmap epoch 7 2024-03-28T14:08:10.494 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:10.491+0000 7f9496e4a640 1 -- 172.21.15.104:0/3821392133 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f9470067ff0 msgr2=0x7f947006a4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:10.494 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:10.491+0000 7f9496e4a640 1 --2- 172.21.15.104:0/3821392133 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f9470067ff0 0x7f947006a4b0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f9488003fd0 tx=0x7f94880039a0 comp rx=0 tx=0).stop 2024-03-28T14:08:10.494 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:10.491+0000 7f9496e4a640 1 -- 172.21.15.104:0/3821392133 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f94900b6f60 msgr2=0x7f9490139320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:10.494 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:10.491+0000 7f9496e4a640 1 --2- 172.21.15.104:0/3821392133 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f94900b6f60 0x7f9490139320 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f94800ae040 tx=0x7f94800191a0 comp rx=0 tx=0).stop 2024-03-28T14:08:10.494 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:10.491+0000 7f9496e4a640 1 -- 172.21.15.104:0/3821392133 shutdown_connections 2024-03-28T14:08:10.494 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:10.491+0000 7f9496e4a640 1 --2- 172.21.15.104:0/3821392133 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f9470067ff0 0x7f947006a4b0 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:10.494 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:10.491+0000 7f9496e4a640 1 --2- 172.21.15.104:0/3821392133 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f94900b6f60 0x7f9490139320 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:10.494 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:10.491+0000 7f9496e4a640 1 --2- 172.21.15.104:0/3821392133 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f9490072f80 0x7f94901e3d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:10.495 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:10.491+0000 7f9496e4a640 1 --2- 172.21.15.104:0/3821392133 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f949006d8e0 0x7f949013a5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:10.495 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:10.491+0000 7f9496e4a640 1 -- 172.21.15.104:0/3821392133 >> 172.21.15.104:0/3821392133 conn(0x7f94900b7b10 msgr2=0x7f949006f0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T14:08:10.495 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:10.491+0000 7f9496e4a640 1 -- 172.21.15.104:0/3821392133 shutdown_connections 2024-03-28T14:08:10.495 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:10.491+0000 7f9496e4a640 1 -- 172.21.15.104:0/3821392133 wait complete. 2024-03-28T14:08:10.520 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-28T14:08:10.647 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:10.643+0000 7f43df600640 1 Processor -- start 2024-03-28T14:08:10.647 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:10.643+0000 7f43df600640 1 -- start start 2024-03-28T14:08:10.649 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:10.647+0000 7f43df600640 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f43d80bb8c0 0x7f43d80b5950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:10.649 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:10.647+0000 7f43df600640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f43d80bbca0 con 0x7f43d80b6790 2024-03-28T14:08:10.650 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:10.647+0000 7f43df600640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f43d80b6de0 con 0x7f43d80b1710 2024-03-28T14:08:10.650 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:10.647+0000 7f43df600640 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 -- 0x7f43d80b7430 con 0x7f43d80bb8c0 2024-03-28T14:08:10.650 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:10.647+0000 7f43dcb9b640 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f43d80b6790 0x7f43d813f470 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:41674/0 (socket says 172.21.15.104:41674) 2024-03-28T14:08:10.650 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:10.647+0000 7f43dcb9b640 1 -- 172.21.15.104:0/3525969444 learned_addr learned my addr 172.21.15.104:0/3525969444 (peer_addr_for_me v1:172.21.15.104:0/0) 2024-03-28T14:08:10.650 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:10.647+0000 7f43dd39c640 1 --2- 172.21.15.104:0/3525969444 >> v2:172.21.15.104:3300/0 conn(0x7f43d80bb8c0 0x7f43d80b5950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:10.651 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:10.647+0000 7f43d7fff640 1 -- 172.21.15.104:0/3525969444 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3428250143 0 0) 0x7f43d80bbca0 con 0x7f43d80b6790 2024-03-28T14:08:10.651 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:10.647+0000 7f43d7fff640 1 -- 172.21.15.104:0/3525969444 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f43c0003530 con 0x7f43d80b6790 2024-03-28T14:08:10.651 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:10.647+0000 7f43dd39c640 1 -- 172.21.15.104:0/3525969444 >> v1:172.21.15.171:6789/0 conn(0x7f43d80b1710 legacy=0x7f43d80b1b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:10.651 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:10.647+0000 7f43dd39c640 1 -- 172.21.15.104:0/3525969444 >> v1:172.21.15.104:6789/0 conn(0x7f43d80b6790 legacy=0x7f43d813f470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:10.651 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:10.647+0000 7f43dd39c640 1 -- 172.21.15.104:0/3525969444 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f43d80b70c0 con 0x7f43d80bb8c0 2024-03-28T14:08:10.653 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:10.647+0000 7f43dd39c640 1 --2- 172.21.15.104:0/3525969444 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f43d80bb8c0 0x7f43d80b5950 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f43d00097c0 tx=0x7f43d001c380 comp rx=0 tx=0).ready entity=mon.0 client_cookie=afe3fb28016ed32c server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:08:10.653 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:10.651+0000 7f43d7fff640 1 -- 172.21.15.104:0/3525969444 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43d00b0070 con 0x7f43d80bb8c0 2024-03-28T14:08:10.653 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:10.651+0000 7f43d7fff640 1 -- 172.21.15.104:0/3525969444 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f43d00ab600 con 0x7f43d80bb8c0 2024-03-28T14:08:10.654 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:10.651+0000 7f43d7fff640 1 -- 172.21.15.104:0/3525969444 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43d00ab9b0 con 0x7f43d80bb8c0 2024-03-28T14:08:10.654 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:10.651+0000 7f43df600640 1 -- 172.21.15.104:0/3525969444 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f43d80bb8c0 msgr2=0x7f43d80b5950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:10.654 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:10.651+0000 7f43df600640 1 --2- 172.21.15.104:0/3525969444 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f43d80bb8c0 0x7f43d80b5950 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f43d00097c0 tx=0x7f43d001c380 comp rx=0 tx=0).stop 2024-03-28T14:08:10.654 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:10.651+0000 7f43df600640 1 -- 172.21.15.104:0/3525969444 shutdown_connections 2024-03-28T14:08:10.655 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:10.651+0000 7f43df600640 1 --2- 172.21.15.104:0/3525969444 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f43d80bb8c0 0x7f43d80b5950 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:10.655 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:10.651+0000 7f43df600640 1 -- 172.21.15.104:0/3525969444 >> 172.21.15.104:0/3525969444 conn(0x7f43d802c800 msgr2=0x7f43d80baf50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T14:08:10.655 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:10.651+0000 7f43df600640 1 -- 172.21.15.104:0/3525969444 shutdown_connections 2024-03-28T14:08:10.655 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:10.651+0000 7f43df600640 1 -- 172.21.15.104:0/3525969444 wait complete. 2024-03-28T14:08:10.655 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:10.651+0000 7f43df600640 1 Processor -- start 2024-03-28T14:08:10.655 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:10.651+0000 7f43df600640 1 -- start start 2024-03-28T14:08:10.660 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:10.651+0000 7f43df600640 1 --2- >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f43d80b1710 0x7f43d8142340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:10.661 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:10.651+0000 7f43df600640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f43d80b6790 0x7f43d806d540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:10.661 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:10.651+0000 7f43df600640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f43d80bb8c0 0x7f43d813c980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:10.661 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:10.651+0000 7f43df600640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f43d8153a20 con 0x7f43d80b6790 2024-03-28T14:08:10.661 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:10.651+0000 7f43df600640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f43d81538a0 con 0x7f43d80bb8c0 2024-03-28T14:08:10.661 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:10.651+0000 7f43df600640 1 -- --> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] -- mon_getmap magic: 0 -- 0x7f43d8153ba0 con 0x7f43d80b1710 2024-03-28T14:08:10.662 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:10.651+0000 7f43dcb9b640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f43d80b6790 0x7f43d806d540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:10.662 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:10.651+0000 7f43dcb9b640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f43d80b6790 0x7f43d806d540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:34874/0 (socket says 172.21.15.104:34874) 2024-03-28T14:08:10.662 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:10.651+0000 7f43dcb9b640 1 -- 172.21.15.104:0/3781806505 learned_addr learned my addr 172.21.15.104:0/3781806505 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-03-28T14:08:10.662 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:10.651+0000 7f43dd39c640 1 --2- 172.21.15.104:0/3781806505 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f43d80b1710 0x7f43d8142340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:10.662 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:10.651+0000 7f43ddb9d640 1 --2- 172.21.15.104:0/3781806505 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f43d80bb8c0 0x7f43d813c980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:10.663 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:10.651+0000 7f43dcb9b640 1 -- 172.21.15.104:0/3781806505 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f43d80b1710 msgr2=0x7f43d8142340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:10.663 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:10.651+0000 7f43dcb9b640 1 --2- 172.21.15.104:0/3781806505 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f43d80b1710 0x7f43d8142340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:10.663 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:10.651+0000 7f43dcb9b640 1 -- 172.21.15.104:0/3781806505 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f43d80bb8c0 msgr2=0x7f43d813c980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:10.663 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:10.651+0000 7f43dcb9b640 1 --2- 172.21.15.104:0/3781806505 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f43d80bb8c0 0x7f43d813c980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:10.663 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:10.651+0000 7f43dcb9b640 1 -- 172.21.15.104:0/3781806505 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f43d8135140 con 0x7f43d80b6790 2024-03-28T14:08:10.664 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:10.651+0000 7f43ddb9d640 1 --2- 172.21.15.104:0/3781806505 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f43d80bb8c0 0x7f43d813c980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T14:08:10.664 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:10.651+0000 7f43dcb9b640 1 --2- 172.21.15.104:0/3781806505 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f43d80b6790 0x7f43d806d540 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f43cc00c4d0 tx=0x7f43cc00c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:08:10.664 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:10.651+0000 7f43d5ffb640 1 -- 172.21.15.104:0/3781806505 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43cc017070 con 0x7f43d80b6790 2024-03-28T14:08:10.664 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:10.651+0000 7f43d5ffb640 1 -- 172.21.15.104:0/3781806505 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f43cc005cf0 con 0x7f43d80b6790 2024-03-28T14:08:10.664 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:10.651+0000 7f43df600640 1 -- 172.21.15.104:0/3781806505 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f43d813e730 con 0x7f43d80b6790 2024-03-28T14:08:10.664 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:10.651+0000 7f43d5ffb640 1 -- 172.21.15.104:0/3781806505 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43cc007270 con 0x7f43d80b6790 2024-03-28T14:08:10.665 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:10.651+0000 7f43df600640 1 -- 172.21.15.104:0/3781806505 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f43d8142c20 con 0x7f43d80b6790 2024-03-28T14:08:10.665 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:10.655+0000 7f43df600640 1 -- 172.21.15.104:0/3781806505 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f43d813e540 con 0x7f43d80b6790 2024-03-28T14:08:10.665 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:10.655+0000 7f43d5ffb640 1 -- 172.21.15.104:0/3781806505 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) ==== 87803+0+0 (secure 0 0 0) 0x7f43cc01a070 con 0x7f43d80b6790 2024-03-28T14:08:10.665 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:10.655+0000 7f43d5ffb640 1 --2- 172.21.15.104:0/3781806505 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f43c4067fc0 0x7f43c406a480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:10.666 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:10.655+0000 7f43d5ffb640 1 -- 172.21.15.104:0/3781806505 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f43cc0857b0 con 0x7f43d80b6790 2024-03-28T14:08:10.666 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:10.655+0000 7f43dd39c640 1 --2- 172.21.15.104:0/3781806505 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f43c4067fc0 0x7f43c406a480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:10.666 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:10.655+0000 7f43dd39c640 1 --2- 172.21.15.104:0/3781806505 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f43c4067fc0 0x7f43c406a480 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f43d00ad000 tx=0x7f43d00a6000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:08:10.667 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:10.659+0000 7f43d5ffb640 1 -- 172.21.15.104:0/3781806505 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179117 (secure 0 0 0) 0x7f43cc0559a0 con 0x7f43d80b6790 2024-03-28T14:08:10.824 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:10.823+0000 7f43df600640 1 -- 172.21.15.104:0/3781806505 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f43d80687c0 con 0x7f43d80b6790 2024-03-28T14:08:10.825 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:10.823+0000 7f43d5ffb640 1 -- 172.21.15.104:0/3781806505 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) ==== 74+0+16629 (secure 0 0 0) 0x7f43cc059ca0 con 0x7f43d80b6790 2024-03-28T14:08:10.826 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-03-28T14:08:10.826 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":21,"fsid":"428a90f2-3351-435f-8fcf-f7e66c78a13e","created":"2024-03-28T14:07:34.862458+0000","modified":"2024-03-28T14:08:00.671587+0000","last_up_change":"2024-03-28T14:07:46.028122+0000","last_in_change":"2024-03-28T14:07:37.157486+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-28T14:07:47.622988+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-28T14:07:57.147450+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-28T14:07:57.990975+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"1cb70540-4e42-402f-a446-aaec4ca83007","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6800","nonce":3745690918},{"type":"v1","addr":"172.21.15.104:6801","nonce":3745690918}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":3745690918},{"type":"v1","addr":"172.21.15.104:6803","nonce":3745690918}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":3745690918},{"type":"v1","addr":"172.21.15.104:6807","nonce":3745690918}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":3745690918},{"type":"v1","addr":"172.21.15.104:6805","nonce":3745690918}]},"public_addr":"172.21.15.104:6801/3745690918","cluster_addr":"172.21.15.104:6803/3745690918","heartbeat_back_addr":"172.21.15.104:6807/3745690918","heartbeat_front_addr":"172.21.15.104:6805/3745690918","state":["exists","up"]},{"osd":1,"uuid":"3ab7afdd-b980-418c-81f2-004e16de3dca","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":19438674},{"type":"v1","addr":"172.21.15.104:6810","nonce":19438674}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":19438674},{"type":"v1","addr":"172.21.15.104:6814","nonce":19438674}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":19438674},{"type":"v1","addr":"172.21.15.104:6823","nonce":19438674}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":19438674},{"type":"v1","addr":"172.21.15.104:6819","nonce":19438674}]},"public_addr":"172.21.15.104:6810/19438674","cluster_addr":"172.21.15.104:6814/19438674","heartbeat_back_addr":"172.21.15.104:6823/19438674","heartbeat_front_addr":"172.21.15.104:6819/19438674","state":["exists","up"]},{"osd":2,"uuid":"3e2651c8-ecae-4a9e-a053-27240a79eac2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":2352012626},{"type":"v1","addr":"172.21.15.104:6825","nonce":2352012626}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6826","nonce":2352012626},{"type":"v1","addr":"172.21.15.104:6827","nonce":2352012626}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":2352012626},{"type":"v1","addr":"172.21.15.104:6831","nonce":2352012626}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6828","nonce":2352012626},{"type":"v1","addr":"172.21.15.104:6829","nonce":2352012626}]},"public_addr":"172.21.15.104:6825/2352012626","cluster_addr":"172.21.15.104:6827/2352012626","heartbeat_back_addr":"172.21.15.104:6831/2352012626","heartbeat_front_addr":"172.21.15.104:6829/2352012626","state":["exists","up"]},{"osd":3,"uuid":"925c7e0a-38ef-47cd-b288-8bec8456fb17","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6809","nonce":3568347381},{"type":"v1","addr":"172.21.15.104:6811","nonce":3568347381}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6813","nonce":3568347381},{"type":"v1","addr":"172.21.15.104:6815","nonce":3568347381}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":3568347381},{"type":"v1","addr":"172.21.15.104:6822","nonce":3568347381}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6817","nonce":3568347381},{"type":"v1","addr":"172.21.15.104:6818","nonce":3568347381}]},"public_addr":"172.21.15.104:6811/3568347381","cluster_addr":"172.21.15.104:6815/3568347381","heartbeat_back_addr":"172.21.15.104:6822/3568347381","heartbeat_front_addr":"172.21.15.104:6818/3568347381","state":["exists","up"]},{"osd":4,"uuid":"4a8fac86-e595-4a0e-96e3-57edb3cc3f57","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6800","nonce":2107001077},{"type":"v1","addr":"172.21.15.171:6801","nonce":2107001077}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6802","nonce":2107001077},{"type":"v1","addr":"172.21.15.171:6803","nonce":2107001077}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6806","nonce":2107001077},{"type":"v1","addr":"172.21.15.171:6807","nonce":2107001077}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6804","nonce":2107001077},{"type":"v1","addr":"172.21.15.171:6805","nonce":2107001077}]},"public_addr":"172.21.15.171:6801/2107001077","cluster_addr":"172.21.15.171:6803/2107001077","heartbeat_back_addr":"172.21.15.171:6807/2107001077","heartbeat_front_addr":"172.21.15.171:6805/2107001077","state":["exists","up"]},{"osd":5,"uuid":"e48d50b0-beae-40ae-b15a-9a84dbfe88ab","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6808","nonce":2934826925},{"type":"v1","addr":"172.21.15.171:6810","nonce":2934826925}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6812","nonce":2934826925},{"type":"v1","addr":"172.21.15.171:6814","nonce":2934826925}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6819","nonce":2934826925},{"type":"v1","addr":"172.21.15.171:6821","nonce":2934826925}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6816","nonce":2934826925},{"type":"v1","addr":"172.21.15.171:6817","nonce":2934826925}]},"public_addr":"172.21.15.171:6810/2934826925","cluster_addr":"172.21.15.171:6814/2934826925","heartbeat_back_addr":"172.21.15.171:6821/2934826925","heartbeat_front_addr":"172.21.15.171:6817/2934826925","state":["exists","up"]},{"osd":6,"uuid":"5e7c3292-9936-4f35-b267-aed9d4e49323","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6824","nonce":436742919},{"type":"v1","addr":"172.21.15.171:6825","nonce":436742919}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6826","nonce":436742919},{"type":"v1","addr":"172.21.15.171:6827","nonce":436742919}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6830","nonce":436742919},{"type":"v1","addr":"172.21.15.171:6831","nonce":436742919}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6828","nonce":436742919},{"type":"v1","addr":"172.21.15.171:6829","nonce":436742919}]},"public_addr":"172.21.15.171:6825/436742919","cluster_addr":"172.21.15.171:6827/436742919","heartbeat_back_addr":"172.21.15.171:6831/436742919","heartbeat_front_addr":"172.21.15.171:6829/436742919","state":["exists","up"]},{"osd":7,"uuid":"339b8414-d7f0-4fb0-a446-efaf635cb7ff","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6809","nonce":2055631175},{"type":"v1","addr":"172.21.15.171:6811","nonce":2055631175}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6813","nonce":2055631175},{"type":"v1","addr":"172.21.15.171:6815","nonce":2055631175}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6822","nonce":2055631175},{"type":"v1","addr":"172.21.15.171:6823","nonce":2055631175}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6818","nonce":2055631175},{"type":"v1","addr":"172.21.15.171:6820","nonce":2055631175}]},"public_addr":"172.21.15.171:6811/2055631175","cluster_addr":"172.21.15.171:6815/2055631175","heartbeat_back_addr":"172.21.15.171:6823/2055631175","heartbeat_front_addr":"172.21.15.171:6820/2055631175","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T14:07:42.668881+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T14:07:42.708397+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T14:07:42.634191+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T14:07:42.733662+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T14:07:42.651541+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T14:07:42.783181+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T14:07:42.766310+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T14:07:42.670797+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-28T14:08:10.828 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:10.827+0000 7f43df600640 1 -- 172.21.15.104:0/3781806505 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f43c4067fc0 msgr2=0x7f43c406a480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:10.828 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:10.827+0000 7f43df600640 1 --2- 172.21.15.104:0/3781806505 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f43c4067fc0 0x7f43c406a480 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f43d00ad000 tx=0x7f43d00a6000 comp rx=0 tx=0).stop 2024-03-28T14:08:10.828 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:10.827+0000 7f43df600640 1 -- 172.21.15.104:0/3781806505 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f43d80b6790 msgr2=0x7f43d806d540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:10.828 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:10.827+0000 7f43df600640 1 --2- 172.21.15.104:0/3781806505 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f43d80b6790 0x7f43d806d540 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f43cc00c4d0 tx=0x7f43cc00c990 comp rx=0 tx=0).stop 2024-03-28T14:08:10.828 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:10.827+0000 7f43df600640 1 -- 172.21.15.104:0/3781806505 shutdown_connections 2024-03-28T14:08:10.828 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:10.827+0000 7f43df600640 1 --2- 172.21.15.104:0/3781806505 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f43d80bb8c0 0x7f43d813c980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:10.828 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:10.827+0000 7f43df600640 1 --2- 172.21.15.104:0/3781806505 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f43c4067fc0 0x7f43c406a480 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:10.828 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:10.827+0000 7f43df600640 1 --2- 172.21.15.104:0/3781806505 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f43d80b6790 0x7f43d806d540 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:10.828 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:10.827+0000 7f43df600640 1 --2- 172.21.15.104:0/3781806505 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f43d80b1710 0x7f43d8142340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:10.828 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:10.827+0000 7f43df600640 1 -- 172.21.15.104:0/3781806505 >> 172.21.15.104:0/3781806505 conn(0x7f43d802c800 msgr2=0x7f43d8072360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T14:08:10.829 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:10.827+0000 7f43df600640 1 -- 172.21.15.104:0/3781806505 shutdown_connections 2024-03-28T14:08:10.829 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:10.827+0000 7f43df600640 1 -- 172.21.15.104:0/3781806505 wait complete. 2024-03-28T14:08:10.845 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T14:08:10.952 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:10.951+0000 7f67d77e2640 1 Processor -- start 2024-03-28T14:08:10.952 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:10.951+0000 7f67d77e2640 1 -- start start 2024-03-28T14:08:10.954 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:10.951+0000 7f67d77e2640 1 --2- >> v2:172.21.15.171:3300/0 conn(0x7f67d00b6260 0x7f67d009cad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:10.954 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:10.951+0000 7f67d77e2640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f67d006e200 con 0x7f67d00733f0 2024-03-28T14:08:10.954 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:10.951+0000 7f67d77e2640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f67d00af2e0 con 0x7f67d006de20 2024-03-28T14:08:10.954 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:10.951+0000 7f67d77e2640 1 -- --> v2:172.21.15.171:3300/0 -- mon_getmap magic: 0 -- 0x7f67d00af820 con 0x7f67d00b6260 2024-03-28T14:08:10.954 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:10.951+0000 7f67d4d7d640 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f67d00733f0 0x7f67d014cfc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:41688/0 (socket says 172.21.15.104:41688) 2024-03-28T14:08:10.954 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:10.951+0000 7f67d4d7d640 1 -- 172.21.15.104:0/195803014 learned_addr learned my addr 172.21.15.104:0/195803014 (peer_addr_for_me v1:172.21.15.104:0/0) 2024-03-28T14:08:10.954 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:10.951+0000 7f67d5d7f640 1 --2- 172.21.15.104:0/195803014 >> v2:172.21.15.171:3300/0 conn(0x7f67d00b6260 0x7f67d009cad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:10.954 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:10.951+0000 7f67c7fff640 1 -- 172.21.15.104:0/195803014 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4165464725 0 0) 0x7f67d006e200 con 0x7f67d00733f0 2024-03-28T14:08:10.954 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:10.951+0000 7f67c7fff640 1 -- 172.21.15.104:0/195803014 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f67bc003530 con 0x7f67d00733f0 2024-03-28T14:08:10.954 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:10.951+0000 7f67c7fff640 1 -- 172.21.15.104:0/195803014 <== mon.0 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 769876639 0 0) 0x7f67bc003530 con 0x7f67d00733f0 2024-03-28T14:08:10.955 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:10.951+0000 7f67c7fff640 1 -- 172.21.15.104:0/195803014 >> v2:172.21.15.171:3300/0 conn(0x7f67d00b6260 msgr2=0x7f67d009cad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T14:08:10.955 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:10.951+0000 7f67c7fff640 1 --2- 172.21.15.104:0/195803014 >> v2:172.21.15.171:3300/0 conn(0x7f67d00b6260 0x7f67d009cad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:10.955 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:10.951+0000 7f67c7fff640 1 -- 172.21.15.104:0/195803014 >> v1:172.21.15.171:6789/0 conn(0x7f67d006de20 legacy=0x7f67d009c3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:10.955 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:10.951+0000 7f67c7fff640 1 -- 172.21.15.104:0/195803014 --> v1:172.21.15.104:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f67d00aefb0 con 0x7f67d00733f0 2024-03-28T14:08:10.955 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:10.951+0000 7f67c7fff640 1 -- 172.21.15.104:0/195803014 <== mon.0 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3745580849 0 0) 0x7f67c0001680 con 0x7f67d00733f0 2024-03-28T14:08:10.955 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:10.951+0000 7f67c7fff640 1 -- 172.21.15.104:0/195803014 >> v1:172.21.15.104:6789/0 conn(0x7f67d00733f0 legacy=0x7f67d014cfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:10.955 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:10.951+0000 7f67c7fff640 1 --2- 172.21.15.104:0/195803014 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f67bc0039c0 0x7f67bc003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:10.955 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:10.951+0000 7f67c7fff640 1 --2- 172.21.15.104:0/195803014 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f67bc004810 0x7f67bc004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:10.955 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:10.951+0000 7f67c7fff640 1 --2- 172.21.15.104:0/195803014 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f67bc0051d0 0x7f67bc00dbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:10.955 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:10.951+0000 7f67c7fff640 1 -- 172.21.15.104:0/195803014 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f67bc00e350 con 0x7f67bc004810 2024-03-28T14:08:10.955 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:10.951+0000 7f67c7fff640 1 -- 172.21.15.104:0/195803014 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f67bc0099f0 con 0x7f67bc0051d0 2024-03-28T14:08:10.955 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:10.951+0000 7f67c7fff640 1 -- 172.21.15.104:0/195803014 --> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] -- mon_getmap magic: 0 -- 0x7f67bc009bf0 con 0x7f67bc0039c0 2024-03-28T14:08:10.955 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:10.951+0000 7f67d4d7d640 1 --2- 172.21.15.104:0/195803014 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f67bc004810 0x7f67bc004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:10.955 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:10.951+0000 7f67d557e640 1 --2- 172.21.15.104:0/195803014 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f67bc0039c0 0x7f67bc003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:10.956 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:10.951+0000 7f67d5d7f640 1 --2- 172.21.15.104:0/195803014 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f67bc0051d0 0x7f67bc00dbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:10.956 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:10.951+0000 7f67d4d7d640 1 -- 172.21.15.104:0/195803014 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f67bc0039c0 msgr2=0x7f67bc003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T14:08:10.956 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:10.951+0000 7f67d4d7d640 1 --2- 172.21.15.104:0/195803014 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f67bc0039c0 0x7f67bc003de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:10.956 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:10.951+0000 7f67d4d7d640 1 -- 172.21.15.104:0/195803014 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f67bc0051d0 msgr2=0x7f67bc00dbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T14:08:10.956 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:10.951+0000 7f67d4d7d640 1 --2- 172.21.15.104:0/195803014 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f67bc0051d0 0x7f67bc00dbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:10.956 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:10.951+0000 7f67d4d7d640 1 -- 172.21.15.104:0/195803014 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f67bc003530 con 0x7f67bc004810 2024-03-28T14:08:10.956 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:10.951+0000 7f67d5d7f640 1 -- 172.21.15.104:0/195803014 reap_dead start 2024-03-28T14:08:10.957 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:10.955+0000 7f67d4d7d640 1 --2- 172.21.15.104:0/195803014 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f67bc004810 0x7f67bc004c90 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f67c000c210 tx=0x7f67c001ca80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d9fedc4bee6a0211 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:08:10.957 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:10.955+0000 7f67c7fff640 1 -- 172.21.15.104:0/195803014 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67c0007d40 con 0x7f67bc004810 2024-03-28T14:08:10.957 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:10.955+0000 7f67c7fff640 1 -- 172.21.15.104:0/195803014 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f67c00a9070 con 0x7f67bc004810 2024-03-28T14:08:10.957 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:10.955+0000 7f67c7fff640 1 -- 172.21.15.104:0/195803014 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67c0005d80 con 0x7f67bc004810 2024-03-28T14:08:10.957 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:10.955+0000 7f67d77e2640 1 -- 172.21.15.104:0/195803014 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f67bc004810 msgr2=0x7f67bc004c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:10.957 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:10.955+0000 7f67d77e2640 1 --2- 172.21.15.104:0/195803014 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f67bc004810 0x7f67bc004c90 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f67c000c210 tx=0x7f67c001ca80 comp rx=0 tx=0).stop 2024-03-28T14:08:10.958 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:10.955+0000 7f67d77e2640 1 -- 172.21.15.104:0/195803014 shutdown_connections 2024-03-28T14:08:10.958 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:10.955+0000 7f67d77e2640 1 --2- 172.21.15.104:0/195803014 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f67bc004810 0x7f67bc004c90 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:10.958 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:10.955+0000 7f67d77e2640 1 -- 172.21.15.104:0/195803014 >> 172.21.15.104:0/195803014 conn(0x7f67d00ba390 msgr2=0x7f67d00ba7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T14:08:10.958 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:10.955+0000 7f67d77e2640 1 -- 172.21.15.104:0/195803014 shutdown_connections 2024-03-28T14:08:10.958 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:10.955+0000 7f67d77e2640 1 -- 172.21.15.104:0/195803014 wait complete. 2024-03-28T14:08:10.960 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:10.955+0000 7f67d77e2640 1 Processor -- start 2024-03-28T14:08:10.960 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:10.955+0000 7f67d77e2640 1 -- start start 2024-03-28T14:08:10.960 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:10.955+0000 7f67d77e2640 1 --2- >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f67bc004810 0x7f67d009f860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:10.960 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:10.955+0000 7f67d77e2640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f67d00b23d0 0x7f67d00b2c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:10.960 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:10.955+0000 7f67d77e2640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f67d00b3bf0 0x7f67d01df960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:10.960 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:10.955+0000 7f67d77e2640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f67d00afe60 con 0x7f67d00b3bf0 2024-03-28T14:08:10.960 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:10.955+0000 7f67d77e2640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f67d009cde0 con 0x7f67d00b23d0 2024-03-28T14:08:10.961 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:10.955+0000 7f67d77e2640 1 -- --> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] -- mon_getmap magic: 0 -- 0x7f67d0147ca0 con 0x7f67bc004810 2024-03-28T14:08:10.961 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:10.955+0000 7f67d5d7f640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f67d00b3bf0 0x7f67d01df960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:10.961 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:10.955+0000 7f67d5d7f640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f67d00b3bf0 0x7f67d01df960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:34902/0 (socket says 172.21.15.104:34902) 2024-03-28T14:08:10.961 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:10.955+0000 7f67d5d7f640 1 -- 172.21.15.104:0/3703600298 learned_addr learned my addr 172.21.15.104:0/3703600298 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-03-28T14:08:10.961 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:10.955+0000 7f67d557e640 1 --2- 172.21.15.104:0/3703600298 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f67bc004810 0x7f67d009f860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:10.961 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:10.959+0000 7f67d4d7d640 1 --2- 172.21.15.104:0/3703600298 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f67d00b23d0 0x7f67d00b2c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:10.961 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:10.959+0000 7f67d5d7f640 1 -- 172.21.15.104:0/3703600298 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f67bc004810 msgr2=0x7f67d009f860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:10.961 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:10.959+0000 7f67d5d7f640 1 --2- 172.21.15.104:0/3703600298 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f67bc004810 0x7f67d009f860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:10.961 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:10.959+0000 7f67d5d7f640 1 -- 172.21.15.104:0/3703600298 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f67d00b23d0 msgr2=0x7f67d00b2c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:10.961 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:10.959+0000 7f67d5d7f640 1 --2- 172.21.15.104:0/3703600298 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f67d00b23d0 0x7f67d00b2c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:10.961 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:10.959+0000 7f67d5d7f640 1 -- 172.21.15.104:0/3703600298 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f67d00b16d0 con 0x7f67d00b3bf0 2024-03-28T14:08:10.961 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:10.959+0000 7f67d4d7d640 1 --2- 172.21.15.104:0/3703600298 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f67d00b23d0 0x7f67d00b2c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T14:08:10.961 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:10.959+0000 7f67d5d7f640 1 --2- 172.21.15.104:0/3703600298 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f67d00b3bf0 0x7f67d01df960 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f67d009aac0 tx=0x7f67c80115a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:08:10.961 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:10.959+0000 7f67c5ffb640 1 -- 172.21.15.104:0/3703600298 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67c8011db0 con 0x7f67d00b3bf0 2024-03-28T14:08:10.962 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:10.959+0000 7f67c5ffb640 1 -- 172.21.15.104:0/3703600298 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f67c801b070 con 0x7f67d00b3bf0 2024-03-28T14:08:10.962 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:10.959+0000 7f67d77e2640 1 -- 172.21.15.104:0/3703600298 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f67d00b3480 con 0x7f67d00b3bf0 2024-03-28T14:08:10.962 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:10.959+0000 7f67c5ffb640 1 -- 172.21.15.104:0/3703600298 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67c80039a0 con 0x7f67d00b3bf0 2024-03-28T14:08:10.962 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:10.959+0000 7f67d77e2640 1 -- 172.21.15.104:0/3703600298 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f67d00b3190 con 0x7f67d00b3bf0 2024-03-28T14:08:10.963 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:10.959+0000 7f67d77e2640 1 -- 172.21.15.104:0/3703600298 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f67d00687c0 con 0x7f67d00b3bf0 2024-03-28T14:08:10.963 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:10.959+0000 7f67c5ffb640 1 -- 172.21.15.104:0/3703600298 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) ==== 87803+0+0 (secure 0 0 0) 0x7f67c8003b40 con 0x7f67d00b3bf0 2024-03-28T14:08:10.963 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:10.959+0000 7f67c5ffb640 1 --2- 172.21.15.104:0/3703600298 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f67b4067fc0 0x7f67b406a480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:10.963 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:10.959+0000 7f67c5ffb640 1 -- 172.21.15.104:0/3703600298 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f67c8086ce0 con 0x7f67d00b3bf0 2024-03-28T14:08:10.963 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:10.959+0000 7f67d557e640 1 --2- 172.21.15.104:0/3703600298 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f67b4067fc0 0x7f67b406a480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:10.963 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:10.963+0000 7f67d557e640 1 --2- 172.21.15.104:0/3703600298 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f67b4067fc0 0x7f67b406a480 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f67cc001660 tx=0x7f67cc009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:08:10.967 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:10.963+0000 7f67c5ffb640 1 -- 172.21.15.104:0/3703600298 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179117 (secure 0 0 0) 0x7f67c8057520 con 0x7f67d00b3bf0 2024-03-28T14:08:11.160 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:11.159+0000 7f67d77e2640 1 -- 172.21.15.104:0/3703600298 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f67d0142cc0 con 0x7f67d00b3bf0 2024-03-28T14:08:11.160 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:11.159+0000 7f67c5ffb640 1 -- 172.21.15.104:0/3703600298 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 7 v7) ==== 93+0+3533 (secure 0 0 0) 0x7f67c8056e20 con 0x7f67d00b3bf0 2024-03-28T14:08:11.161 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-03-28T14:08:11.161 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":7,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4229,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.171:6835/1775867467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6834","nonce":1775867467},{"type":"v1","addr":"172.21.15.171:6835","nonce":1775867467}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":7,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T14:07:59.027458+0000","modified":"2024-03-28T14:08:02.057923+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4245},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4245":{"gid":4245,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.104:6833/3184673325","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":3184673325},{"type":"v1","addr":"172.21.15.104:6833","nonce":3184673325}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4245,"qdb_cluster":[4245]},"id":1}]} 2024-03-28T14:08:11.161 INFO:teuthology.orchestra.run.smithi104.stderr:dumped fsmap epoch 7 2024-03-28T14:08:11.163 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:11.159+0000 7f67d77e2640 1 -- 172.21.15.104:0/3703600298 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f67b4067fc0 msgr2=0x7f67b406a480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:11.163 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:11.159+0000 7f67d77e2640 1 --2- 172.21.15.104:0/3703600298 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f67b4067fc0 0x7f67b406a480 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f67cc001660 tx=0x7f67cc009040 comp rx=0 tx=0).stop 2024-03-28T14:08:11.163 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:11.159+0000 7f67d77e2640 1 -- 172.21.15.104:0/3703600298 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f67d00b3bf0 msgr2=0x7f67d01df960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:11.163 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:11.159+0000 7f67d77e2640 1 --2- 172.21.15.104:0/3703600298 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f67d00b3bf0 0x7f67d01df960 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f67d009aac0 tx=0x7f67c80115a0 comp rx=0 tx=0).stop 2024-03-28T14:08:11.163 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:11.159+0000 7f67d77e2640 1 -- 172.21.15.104:0/3703600298 shutdown_connections 2024-03-28T14:08:11.163 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:11.159+0000 7f67d77e2640 1 --2- 172.21.15.104:0/3703600298 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f67b4067fc0 0x7f67b406a480 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:11.163 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:11.159+0000 7f67d77e2640 1 --2- 172.21.15.104:0/3703600298 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f67d00b3bf0 0x7f67d01df960 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:11.164 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:11.159+0000 7f67d77e2640 1 --2- 172.21.15.104:0/3703600298 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f67d00b23d0 0x7f67d00b2c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:11.164 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:11.159+0000 7f67d77e2640 1 --2- 172.21.15.104:0/3703600298 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f67bc004810 0x7f67d009f860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:11.164 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:11.159+0000 7f67d77e2640 1 -- 172.21.15.104:0/3703600298 >> 172.21.15.104:0/3703600298 conn(0x7f67d00ba390 msgr2=0x7f67d006b5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T14:08:11.164 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:11.159+0000 7f67d77e2640 1 -- 172.21.15.104:0/3703600298 shutdown_connections 2024-03-28T14:08:11.164 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:11.159+0000 7f67d77e2640 1 -- 172.21.15.104:0/3703600298 wait complete. 2024-03-28T14:08:11.380 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 7, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-28T14:07:59.027458+0000', 'modified': '2024-03-28T14:08:02.057923+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 4245}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4245': {'gid': 4245, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.104:6833/3184673325', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.104:6832', 'nonce': 3184673325}, {'type': 'v1', 'addr': '172.21.15.104:6833', 'nonce': 3184673325}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4245, 'qdb_cluster': [4245]} 2024-03-28T14:08:11.380 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-03-28T14:08:11.380 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2024-03-28T14:08:11.514 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:11.511+0000 7fc449ed8640 1 Processor -- start 2024-03-28T14:08:11.514 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:11.511+0000 7fc449ed8640 1 -- start start 2024-03-28T14:08:11.517 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:11.511+0000 7fc449ed8640 1 --2- >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fc44406e230 0x7fc44413fb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:11.517 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:11.511+0000 7fc449ed8640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc44402c840 con 0x7fc44409e9a0 2024-03-28T14:08:11.517 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:11.511+0000 7fc449ed8640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc4440b5ed0 con 0x7fc44402c460 2024-03-28T14:08:11.517 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:11.511+0000 7fc449ed8640 1 -- --> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] -- mon_getmap magic: 0 -- 0x7fc4440b3ff0 con 0x7fc44406e230 2024-03-28T14:08:11.517 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:11.511+0000 7fc43bfff640 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7fc44409e9a0 0x7fc44414dfc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:41700/0 (socket says 172.21.15.104:41700) 2024-03-28T14:08:11.517 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:11.511+0000 7fc43bfff640 1 -- 172.21.15.104:0/3725981003 learned_addr learned my addr 172.21.15.104:0/3725981003 (peer_addr_for_me v1:172.21.15.104:0/0) 2024-03-28T14:08:11.517 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:11.515+0000 7fc443fff640 1 --2- 172.21.15.104:0/3725981003 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fc44406e230 0x7fc44413fb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:11.517 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:11.515+0000 7fc442ffd640 1 -- 172.21.15.104:0/3725981003 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2916319528 0 0) 0x7fc44402c840 con 0x7fc44409e9a0 2024-03-28T14:08:11.517 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:11.515+0000 7fc442ffd640 1 -- 172.21.15.104:0/3725981003 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc430003530 con 0x7fc44409e9a0 2024-03-28T14:08:11.517 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:11.515+0000 7fc442ffd640 1 -- 172.21.15.104:0/3725981003 <== mon.0 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1905825234 0 0) 0x7fc430003530 con 0x7fc44409e9a0 2024-03-28T14:08:11.518 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:11.515+0000 7fc442ffd640 1 -- 172.21.15.104:0/3725981003 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fc44406e230 msgr2=0x7fc44413fb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T14:08:11.518 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:11.515+0000 7fc442ffd640 1 --2- 172.21.15.104:0/3725981003 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fc44406e230 0x7fc44413fb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:11.518 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:11.515+0000 7fc442ffd640 1 -- 172.21.15.104:0/3725981003 >> v1:172.21.15.171:6789/0 conn(0x7fc44402c460 legacy=0x7fc44413f440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:11.518 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:11.515+0000 7fc442ffd640 1 -- 172.21.15.104:0/3725981003 --> v1:172.21.15.104:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc4440b42d0 con 0x7fc44409e9a0 2024-03-28T14:08:11.518 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:11.515+0000 7fc442ffd640 1 -- 172.21.15.104:0/3725981003 <== mon.0 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3745580849 0 0) 0x7fc42c001680 con 0x7fc44409e9a0 2024-03-28T14:08:11.518 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:11.515+0000 7fc442ffd640 1 -- 172.21.15.104:0/3725981003 >> v1:172.21.15.104:6789/0 conn(0x7fc44409e9a0 legacy=0x7fc44414dfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:11.518 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:11.515+0000 7fc442ffd640 1 --2- 172.21.15.104:0/3725981003 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc4300039f0 0x7fc430003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:11.518 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:11.515+0000 7fc442ffd640 1 --2- 172.21.15.104:0/3725981003 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fc44406e230 0x7fc430004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:11.518 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:11.515+0000 7fc442ffd640 1 --2- 172.21.15.104:0/3725981003 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fc430004d50 0x7fc4300051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:11.518 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:11.515+0000 7fc442ffd640 1 -- 172.21.15.104:0/3725981003 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7fc4440b3ff0 con 0x7fc4300039f0 2024-03-28T14:08:11.518 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:11.515+0000 7fc442ffd640 1 -- 172.21.15.104:0/3725981003 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7fc43000ba80 con 0x7fc430004d50 2024-03-28T14:08:11.518 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:11.515+0000 7fc442ffd640 1 -- 172.21.15.104:0/3725981003 --> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] -- mon_getmap magic: 0 -- 0x7fc43000bc80 con 0x7fc44406e230 2024-03-28T14:08:11.518 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:11.515+0000 7fc4437fe640 1 --2- 172.21.15.104:0/3725981003 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc4300039f0 0x7fc430003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:11.519 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:11.515+0000 7fc43bfff640 1 --2- 172.21.15.104:0/3725981003 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fc44406e230 0x7fc430004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:11.519 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:11.515+0000 7fc443fff640 1 --2- 172.21.15.104:0/3725981003 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fc430004d50 0x7fc4300051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:11.519 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:11.515+0000 7fc4437fe640 1 -- 172.21.15.104:0/3725981003 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fc44406e230 msgr2=0x7fc430004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T14:08:11.519 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:11.515+0000 7fc4437fe640 1 --2- 172.21.15.104:0/3725981003 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fc44406e230 0x7fc430004810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:11.519 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:11.515+0000 7fc4437fe640 1 -- 172.21.15.104:0/3725981003 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fc430004d50 msgr2=0x7fc4300051d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T14:08:11.519 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:11.515+0000 7fc4437fe640 1 --2- 172.21.15.104:0/3725981003 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fc430004d50 0x7fc4300051d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:11.519 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:11.515+0000 7fc4437fe640 1 -- 172.21.15.104:0/3725981003 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc430003530 con 0x7fc4300039f0 2024-03-28T14:08:11.520 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:11.515+0000 7fc4437fe640 1 --2- 172.21.15.104:0/3725981003 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc4300039f0 0x7fc430003e10 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fc434001750 tx=0x7fc43401c3e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7352ebaac4925ef8 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:08:11.520 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:11.515+0000 7fc442ffd640 1 -- 172.21.15.104:0/3725981003 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4340013a0 con 0x7fc4300039f0 2024-03-28T14:08:11.520 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:11.515+0000 7fc442ffd640 1 -- 172.21.15.104:0/3725981003 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc4340052f0 con 0x7fc4300039f0 2024-03-28T14:08:11.520 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:11.515+0000 7fc442ffd640 1 -- 172.21.15.104:0/3725981003 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc434003840 con 0x7fc4300039f0 2024-03-28T14:08:11.520 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:11.519+0000 7fc449ed8640 1 -- 172.21.15.104:0/3725981003 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc4300039f0 msgr2=0x7fc430003e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:11.521 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:11.519+0000 7fc449ed8640 1 --2- 172.21.15.104:0/3725981003 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc4300039f0 0x7fc430003e10 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fc434001750 tx=0x7fc43401c3e0 comp rx=0 tx=0).stop 2024-03-28T14:08:11.521 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:11.519+0000 7fc443fff640 1 -- 172.21.15.104:0/3725981003 reap_dead start 2024-03-28T14:08:11.521 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:11.519+0000 7fc449ed8640 1 -- 172.21.15.104:0/3725981003 shutdown_connections 2024-03-28T14:08:11.521 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:11.519+0000 7fc449ed8640 1 -- 172.21.15.104:0/3725981003 >> 172.21.15.104:0/3725981003 conn(0x7fc4440b95f0 msgr2=0x7fc4440b99d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T14:08:11.521 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:11.519+0000 7fc449ed8640 1 -- 172.21.15.104:0/3725981003 shutdown_connections 2024-03-28T14:08:11.521 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:11.519+0000 7fc449ed8640 1 -- 172.21.15.104:0/3725981003 wait complete. 2024-03-28T14:08:11.521 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:11.519+0000 7fc449ed8640 1 Processor -- start 2024-03-28T14:08:11.521 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:11.519+0000 7fc449ed8640 1 -- start start 2024-03-28T14:08:11.523 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:11.519+0000 7fc449ed8640 1 --2- >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fc4300039f0 0x7fc44409b0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:11.523 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:11.519+0000 7fc449ed8640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc4440b3bd0 0x7fc444073530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:11.523 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:11.519+0000 7fc449ed8640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fc444071770 0x7fc4441de200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:11.523 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:11.519+0000 7fc449ed8640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7fc44409d890 con 0x7fc4440b3bd0 2024-03-28T14:08:11.523 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:11.519+0000 7fc449ed8640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7fc44409d710 con 0x7fc444071770 2024-03-28T14:08:11.523 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:11.519+0000 7fc449ed8640 1 -- --> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] -- mon_getmap magic: 0 -- 0x7fc44409da10 con 0x7fc4300039f0 2024-03-28T14:08:11.524 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:11.519+0000 7fc43bfff640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc4440b3bd0 0x7fc444073530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:11.524 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:11.519+0000 7fc43bfff640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc4440b3bd0 0x7fc444073530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:34928/0 (socket says 172.21.15.104:34928) 2024-03-28T14:08:11.524 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:11.519+0000 7fc43bfff640 1 -- 172.21.15.104:0/519619698 learned_addr learned my addr 172.21.15.104:0/519619698 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-03-28T14:08:11.524 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:11.519+0000 7fc4437fe640 1 --2- 172.21.15.104:0/519619698 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fc4300039f0 0x7fc44409b0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:11.524 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:11.519+0000 7fc443fff640 1 --2- 172.21.15.104:0/519619698 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fc444071770 0x7fc4441de200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:11.524 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:11.519+0000 7fc43bfff640 1 -- 172.21.15.104:0/519619698 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fc4300039f0 msgr2=0x7fc44409b0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:11.524 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:11.519+0000 7fc43bfff640 1 --2- 172.21.15.104:0/519619698 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fc4300039f0 0x7fc44409b0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:11.524 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:11.519+0000 7fc43bfff640 1 -- 172.21.15.104:0/519619698 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fc444071770 msgr2=0x7fc4441de200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:11.524 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:11.519+0000 7fc43bfff640 1 --2- 172.21.15.104:0/519619698 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fc444071770 0x7fc4441de200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:11.524 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:11.519+0000 7fc43bfff640 1 -- 172.21.15.104:0/519619698 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc444067c10 con 0x7fc4440b3bd0 2024-03-28T14:08:11.524 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:11.519+0000 7fc443fff640 1 --2- 172.21.15.104:0/519619698 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fc444071770 0x7fc4441de200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T14:08:11.524 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:11.519+0000 7fc4437fe640 1 --2- 172.21.15.104:0/519619698 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fc4300039f0 0x7fc44409b0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T14:08:11.524 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:11.519+0000 7fc43bfff640 1 --2- 172.21.15.104:0/519619698 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc4440b3bd0 0x7fc444073530 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fc42c00e730 tx=0x7fc42c00ebf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:08:11.524 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:11.519+0000 7fc440ff9640 1 -- 172.21.15.104:0/519619698 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc42c00ee40 con 0x7fc4440b3bd0 2024-03-28T14:08:11.525 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:11.519+0000 7fc440ff9640 1 -- 172.21.15.104:0/519619698 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc42c018070 con 0x7fc4440b3bd0 2024-03-28T14:08:11.525 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:11.519+0000 7fc449ed8640 1 -- 172.21.15.104:0/519619698 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc444070f70 con 0x7fc4440b3bd0 2024-03-28T14:08:11.525 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:11.519+0000 7fc440ff9640 1 -- 172.21.15.104:0/519619698 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc42c00ca90 con 0x7fc4440b3bd0 2024-03-28T14:08:11.525 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:11.519+0000 7fc449ed8640 1 -- 172.21.15.104:0/519619698 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc44413bfc0 con 0x7fc4440b3bd0 2024-03-28T14:08:11.530 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:11.523+0000 7fc440ff9640 1 -- 172.21.15.104:0/519619698 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) ==== 87803+0+0 (secure 0 0 0) 0x7fc42c00cc30 con 0x7fc4440b3bd0 2024-03-28T14:08:11.531 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:11.523+0000 7fc440ff9640 1 --2- 172.21.15.104:0/519619698 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7fc424067ff0 0x7fc42406a4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:11.531 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:11.523+0000 7fc440ff9640 1 -- 172.21.15.104:0/519619698 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7fc42c085fe0 con 0x7fc4440b3bd0 2024-03-28T14:08:11.531 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:11.523+0000 7fc4437fe640 1 --2- 172.21.15.104:0/519619698 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7fc424067ff0 0x7fc42406a4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:11.531 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:11.523+0000 7fc44d057000 1 Processor -- start 2024-03-28T14:08:11.531 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:11.523+0000 7fc44d057000 1 -- start start 2024-03-28T14:08:11.531 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:11.523+0000 7fc4437fe640 1 --2- 172.21.15.104:0/519619698 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7fc424067ff0 0x7fc42406a4b0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fc4300057f0 tx=0x7fc4340011a0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:08:11.531 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:11.523+0000 7fc44d057000 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55efb8c5ce00 0x55efb8c85fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:11.531 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:11.523+0000 7fc44d057000 1 --2- >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x55efb8c80e10 0x55efb8c811f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:11.531 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:11.523+0000 7fc44d057000 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55efb8c81730 0x55efb8c92d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:11.531 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:11.523+0000 7fc44d057000 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x55efb8aa7950 con 0x55efb8c81730 2024-03-28T14:08:11.531 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:11.523+0000 7fc44d057000 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x55efb8a44990 con 0x55efb8c5ce00 2024-03-28T14:08:11.531 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:11.523+0000 7fc44d057000 1 -- --> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] -- mon_getmap magic: 0 -- 0x55efb8aa3900 con 0x55efb8c80e10 2024-03-28T14:08:11.531 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:11.523+0000 7fc443fff640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55efb8c81730 0x55efb8c92d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:11.532 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:11.523+0000 7fc443fff640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55efb8c81730 0x55efb8c92d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:34944/0 (socket says 172.21.15.104:34944) 2024-03-28T14:08:11.532 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:11.523+0000 7fc443fff640 1 -- 172.21.15.104:0/1268284070 learned_addr learned my addr 172.21.15.104:0/1268284070 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-03-28T14:08:11.532 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:11.523+0000 7fc4437fe640 1 --2- 172.21.15.104:0/1268284070 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55efb8c5ce00 0x55efb8c85fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:11.532 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:11.523+0000 7fc43bfff640 1 --2- 172.21.15.104:0/1268284070 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x55efb8c80e10 0x55efb8c811f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:11.532 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:11.523+0000 7fc443fff640 1 -- 172.21.15.104:0/1268284070 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x55efb8c80e10 msgr2=0x55efb8c811f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T14:08:11.532 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:11.523+0000 7fc443fff640 1 --2- 172.21.15.104:0/1268284070 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x55efb8c80e10 0x55efb8c811f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:11.532 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:11.523+0000 7fc443fff640 1 -- 172.21.15.104:0/1268284070 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55efb8c5ce00 msgr2=0x55efb8c85fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T14:08:11.532 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:11.523+0000 7fc443fff640 1 --2- 172.21.15.104:0/1268284070 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55efb8c5ce00 0x55efb8c85fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:11.532 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:11.523+0000 7fc443fff640 1 -- 172.21.15.104:0/1268284070 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55efb8aec100 con 0x55efb8c81730 2024-03-28T14:08:11.532 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:11.523+0000 7fc4437fe640 1 --2- 172.21.15.104:0/1268284070 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55efb8c5ce00 0x55efb8c85fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T14:08:11.532 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:11.523+0000 7fc443fff640 1 --2- 172.21.15.104:0/1268284070 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55efb8c81730 0x55efb8c92d60 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fc44406e1c0 tx=0x7fc43c009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2cc485da014df03c server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:08:11.532 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:11.523+0000 7fc4397fa640 1 -- 172.21.15.104:0/1268284070 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc43c01a020 con 0x55efb8c81730 2024-03-28T14:08:11.532 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:11.523+0000 7fc4397fa640 1 -- 172.21.15.104:0/1268284070 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc43c016660 con 0x55efb8c81730 2024-03-28T14:08:11.532 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:11.523+0000 7fc4397fa640 1 -- 172.21.15.104:0/1268284070 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc43c016a50 con 0x55efb8c81730 2024-03-28T14:08:11.532 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:11.523+0000 7fc44d057000 1 -- 172.21.15.104:0/1268284070 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55efb8c81730 msgr2=0x55efb8c92d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:11.533 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:11.523+0000 7fc44d057000 1 --2- 172.21.15.104:0/1268284070 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55efb8c81730 0x55efb8c92d60 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fc44406e1c0 tx=0x7fc43c009750 comp rx=0 tx=0).stop 2024-03-28T14:08:11.533 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:11.523+0000 7fc44d057000 1 -- 172.21.15.104:0/1268284070 shutdown_connections 2024-03-28T14:08:11.533 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:11.523+0000 7fc44d057000 1 --2- 172.21.15.104:0/1268284070 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55efb8c81730 0x55efb8c92d60 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:11.533 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:11.523+0000 7fc44d057000 1 --2- 172.21.15.104:0/1268284070 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x55efb8c80e10 0x55efb8c811f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:11.533 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:11.523+0000 7fc44d057000 1 --2- 172.21.15.104:0/1268284070 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55efb8c5ce00 0x55efb8c85fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:11.533 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:11.523+0000 7fc44d057000 1 -- 172.21.15.104:0/1268284070 >> 172.21.15.104:0/1268284070 conn(0x55efb8b3fb80 msgr2=0x55efb8c85890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T14:08:11.533 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:11.523+0000 7fc44d057000 1 -- 172.21.15.104:0/1268284070 shutdown_connections 2024-03-28T14:08:11.533 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:11.523+0000 7fc44d057000 1 -- 172.21.15.104:0/1268284070 wait complete. 2024-03-28T14:08:11.533 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:11.527+0000 7fc44d057000 1 -- start start 2024-03-28T14:08:11.533 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:11.527+0000 7fc44d057000 1 Processor -- start 2024-03-28T14:08:11.533 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:11.527+0000 7fc44d057000 1 --2- >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x55efb8c5ce00 0x55efb8c92d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:11.533 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:11.527+0000 7fc44d057000 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55efb8c80e10 0x55efb8c932a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:11.533 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:11.527+0000 7fc44d057000 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55efb8c81730 0x55efb8c57cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:11.533 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:11.527+0000 7fc44d057000 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x55efb8aabf80 con 0x55efb8c81730 2024-03-28T14:08:11.534 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:11.527+0000 7fc44d057000 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x55efb8a653e0 con 0x55efb8c80e10 2024-03-28T14:08:11.534 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:11.527+0000 7fc44d057000 1 -- --> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] -- mon_getmap magic: 0 -- 0x55efb8ab4060 con 0x55efb8c5ce00 2024-03-28T14:08:11.534 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:11.527+0000 7fc44d057000 10 client.0 _ll_register_callbacks cb 0x55efb8b080f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T14:08:11.534 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:11.527+0000 7fc443fff640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55efb8c81730 0x55efb8c57cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:11.534 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:11.527+0000 7fc443fff640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55efb8c81730 0x55efb8c57cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:34960/0 (socket says 172.21.15.104:34960) 2024-03-28T14:08:11.534 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:11.527+0000 7fc443fff640 1 -- 172.21.15.104:0/172576327 learned_addr learned my addr 172.21.15.104:0/172576327 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-03-28T14:08:11.534 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:11.527+0000 7fc43bfff640 1 --2- 172.21.15.104:0/172576327 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55efb8c80e10 0x55efb8c932a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:11.534 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:11.527+0000 7fc4437fe640 1 --2- 172.21.15.104:0/172576327 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x55efb8c5ce00 0x55efb8c92d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:11.534 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:11.527+0000 7fc443fff640 1 -- 172.21.15.104:0/172576327 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x55efb8c5ce00 msgr2=0x55efb8c92d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T14:08:11.534 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:11.527+0000 7fc443fff640 1 --2- 172.21.15.104:0/172576327 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x55efb8c5ce00 0x55efb8c92d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:11.534 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:11.527+0000 7fc443fff640 1 -- 172.21.15.104:0/172576327 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55efb8c80e10 msgr2=0x55efb8c932a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T14:08:11.534 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:11.527+0000 7fc443fff640 1 --2- 172.21.15.104:0/172576327 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55efb8c80e10 0x55efb8c932a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:11.534 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:11.527+0000 7fc443fff640 1 -- 172.21.15.104:0/172576327 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55efb8a718c0 con 0x55efb8c81730 2024-03-28T14:08:11.534 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:11.527+0000 7fc43bfff640 1 --2- 172.21.15.104:0/172576327 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55efb8c80e10 0x55efb8c932a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T14:08:11.535 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:11.527+0000 7fc443fff640 1 --2- 172.21.15.104:0/172576327 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55efb8c81730 0x55efb8c57cb0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fc43c00b010 tx=0x7fc43c004a50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b474741541135502 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:08:11.535 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:11.527+0000 7fc4227fc640 10 client.0 ms_handle_connect on v2:172.21.15.104:3300/0 2024-03-28T14:08:11.535 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:11.527+0000 7fc4227fc640 1 -- 172.21.15.104:0/172576327 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc43c01f420 con 0x55efb8c81730 2024-03-28T14:08:11.535 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:11.527+0000 7fc4227fc640 1 -- 172.21.15.104:0/172576327 <== mon.0 v2:172.21.15.104:3300/0 2 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7fc43c01f5c0 con 0x55efb8c81730 2024-03-28T14:08:11.535 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:11.527+0000 7fc44d057000 1 -- 172.21.15.104:0/172576327 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc43c016660 con 0x55efb8c81730 2024-03-28T14:08:11.535 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:11.527+0000 7fc4227fc640 1 -- 172.21.15.104:0/172576327 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_get_version_reply(handle=1 version=7) ==== 24+0+0 (secure 0 0 0) 0x7fc43c01f800 con 0x55efb8c81730 2024-03-28T14:08:11.535 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:11.527+0000 7fc44d057000 10 client.4602 fetch_fsmap learned FSMap version 7 2024-03-28T14:08:11.535 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:11.527+0000 7fc44d057000 1 -- 172.21.15.104:0/172576327 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,fsmap=7,monmap=2+}) -- 0x55efb8a2fe80 con 0x55efb8c81730 2024-03-28T14:08:11.535 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:11.527+0000 7fc4227fc640 1 -- 172.21.15.104:0/172576327 <== mon.0 v2:172.21.15.104:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc43c016660 con 0x55efb8c81730 2024-03-28T14:08:11.535 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:11.527+0000 7fc4227fc640 1 -- 172.21.15.104:0/172576327 <== mon.0 v2:172.21.15.104:3300/0 5 ==== fsmap(e 7) ==== 2284+0+0 (secure 0 0 0) 0x7fc43c01f9a0 con 0x55efb8c81730 2024-03-28T14:08:11.535 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:11.527+0000 7fc44d057000 10 client.4602 fetch_fsmap finished waiting for FSMap version 7 2024-03-28T14:08:11.535 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:11.527+0000 7fc44d057000 10 client.4602 resolve_mds: resolved name 'a' to daemon mds.a 2024-03-28T14:08:11.535 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:11.527+0000 7fc44d057000 20 client.4602 populate_metadata read hostname 'smithi104' 2024-03-28T14:08:11.535 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:11.527+0000 7fc44d057000 1 --2- 172.21.15.104:0/172576327 >> [v2:172.21.15.104:6832/3184673325,v1:172.21.15.104:6833/3184673325] conn(0x55efb8c58a50 0x55efb8c58e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:11.536 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:11.527+0000 7fc44d057000 4 client.4602 mds_command: new command op to 4245 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T14:08:11.536 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:11.527+0000 7fc44d057000 1 -- 172.21.15.104:0/172576327 --> [v2:172.21.15.104:6832/3184673325,v1:172.21.15.104:6833/3184673325] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55efb8a42bc0 con 0x55efb8c58a50 2024-03-28T14:08:11.536 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:11.527+0000 7fc4437fe640 1 --2- 172.21.15.104:0/172576327 >> [v2:172.21.15.104:6832/3184673325,v1:172.21.15.104:6833/3184673325] conn(0x55efb8c58a50 0x55efb8c58e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:11.536 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:11.527+0000 7fc4437fe640 1 --2- 172.21.15.104:0/172576327 >> [v2:172.21.15.104:6832/3184673325,v1:172.21.15.104:6833/3184673325] conn(0x55efb8c58a50 0x55efb8c58e30 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ecb2f1a3fd63c57 server_cookie=448ab33ac9d00b50 in_seq=0 out_seq=0 2024-03-28T14:08:11.536 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:11.527+0000 7fc4227fc640 10 client.4602 ms_handle_connect on v2:172.21.15.104:6832/3184673325 2024-03-28T14:08:11.537 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:11.531+0000 7fc4227fc640 1 -- 172.21.15.104:0/172576327 <== mds.0 v2:172.21.15.104:6832/3184673325 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55efb8a42bc0 con 0x55efb8c58a50 2024-03-28T14:08:11.537 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:11.531+0000 7fc4227fc640 10 client.4602 handle_command_reply: tid=0 2024-03-28T14:08:11.537 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:11.531+0000 7fc44d057000 1 client.4602 shutdown 2024-03-28T14:08:11.537 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:11.531+0000 7fc44d057000 1 -- 172.21.15.104:0/172576327 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55efb8c81730 msgr2=0x55efb8c57cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:11.537 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:11.531+0000 7fc44d057000 1 --2- 172.21.15.104:0/172576327 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55efb8c81730 0x55efb8c57cb0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fc43c00b010 tx=0x7fc43c004a50 comp rx=0 tx=0).stop 2024-03-28T14:08:11.537 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:11.531+0000 7fc44d057000 1 -- 172.21.15.104:0/172576327 shutdown_connections 2024-03-28T14:08:11.537 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:11.531+0000 7fc44d057000 1 --2- 172.21.15.104:0/172576327 >> [v2:172.21.15.104:6832/3184673325,v1:172.21.15.104:6833/3184673325] conn(0x55efb8c58a50 0x55efb8c58e30 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:11.537 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:11.531+0000 7fc44d057000 1 --2- 172.21.15.104:0/172576327 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55efb8c81730 0x55efb8c57cb0 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:11.538 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:11.531+0000 7fc44d057000 1 --2- 172.21.15.104:0/172576327 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55efb8c80e10 0x55efb8c932a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:11.538 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:11.531+0000 7fc44d057000 1 --2- 172.21.15.104:0/172576327 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x55efb8c5ce00 0x55efb8c92d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:11.538 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:11.531+0000 7fc4227fc640 0 client.4602 ms_handle_reset on v2:172.21.15.104:6832/3184673325 2024-03-28T14:08:11.538 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:11.531+0000 7fc44d057000 1 -- 172.21.15.104:0/172576327 >> 172.21.15.104:0/172576327 conn(0x55efb8b3fb80 msgr2=0x55efb8c85890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T14:08:11.538 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:11.531+0000 7fc44d057000 1 -- 172.21.15.104:0/172576327 shutdown_connections 2024-03-28T14:08:11.538 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:11.531+0000 7fc44d057000 1 -- 172.21.15.104:0/172576327 wait complete. 2024-03-28T14:08:11.538 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:11.531+0000 7fc44d057000 20 client.4602 trim_cache size 0 max 16384 2024-03-28T14:08:11.550 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:11.547+0000 7fc44d057000 1 Processor -- start 2024-03-28T14:08:11.551 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:11.547+0000 7fc44d057000 1 -- start start 2024-03-28T14:08:11.551 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:11.547+0000 7fc44d057000 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55efb8c5ce00 0x55efb8c57880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:11.551 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:11.547+0000 7fc44d057000 1 --2- >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x55efb8c80e10 0x55efb8c98bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:11.551 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:11.547+0000 7fc44d057000 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55efb8c81730 0x55efb8c99110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:11.551 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:11.547+0000 7fc44d057000 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7fc4441eb7d0 con 0x55efb8c81730 2024-03-28T14:08:11.551 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:11.547+0000 7fc44d057000 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7fc4441ea8d0 con 0x55efb8c5ce00 2024-03-28T14:08:11.551 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:11.547+0000 7fc44d057000 1 -- --> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] -- mon_getmap magic: 0 -- 0x55efb8a3f7b0 con 0x55efb8c80e10 2024-03-28T14:08:11.551 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:11.547+0000 7fc443fff640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55efb8c81730 0x55efb8c99110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:11.551 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:11.547+0000 7fc443fff640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55efb8c81730 0x55efb8c99110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:34976/0 (socket says 172.21.15.104:34976) 2024-03-28T14:08:11.551 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:11.547+0000 7fc443fff640 1 -- 172.21.15.104:0/737019030 learned_addr learned my addr 172.21.15.104:0/737019030 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-03-28T14:08:11.551 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:11.547+0000 7fc443fff640 1 -- 172.21.15.104:0/737019030 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x55efb8c80e10 msgr2=0x55efb8c98bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T14:08:11.551 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:11.547+0000 7fc443fff640 1 --2- 172.21.15.104:0/737019030 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x55efb8c80e10 0x55efb8c98bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:11.551 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:11.547+0000 7fc443fff640 1 -- 172.21.15.104:0/737019030 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55efb8c5ce00 msgr2=0x55efb8c57880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T14:08:11.552 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:11.547+0000 7fc443fff640 1 --2- 172.21.15.104:0/737019030 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55efb8c5ce00 0x55efb8c57880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:11.552 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:11.547+0000 7fc443fff640 1 -- 172.21.15.104:0/737019030 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55efb8c81560 con 0x55efb8c81730 2024-03-28T14:08:11.552 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:11.547+0000 7fc443fff640 1 --2- 172.21.15.104:0/737019030 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55efb8c81730 0x55efb8c99110 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fc43c004560 tx=0x7fc43c022040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ff48e36c17792eb server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:08:11.552 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:11.547+0000 7fc438ff9640 1 -- 172.21.15.104:0/737019030 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc43c03c020 con 0x55efb8c81730 2024-03-28T14:08:11.552 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:11.547+0000 7fc438ff9640 1 -- 172.21.15.104:0/737019030 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc43c01fa20 con 0x55efb8c81730 2024-03-28T14:08:11.552 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:11.547+0000 7fc438ff9640 1 -- 172.21.15.104:0/737019030 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc43c01a040 con 0x55efb8c81730 2024-03-28T14:08:11.556 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:11.551+0000 7fc44d057000 1 -- 172.21.15.104:0/737019030 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55efb8c81730 msgr2=0x55efb8c99110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:11.557 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:11.551+0000 7fc44d057000 1 --2- 172.21.15.104:0/737019030 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55efb8c81730 0x55efb8c99110 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fc43c004560 tx=0x7fc43c022040 comp rx=0 tx=0).stop 2024-03-28T14:08:11.557 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:11.551+0000 7fc44d057000 1 -- 172.21.15.104:0/737019030 shutdown_connections 2024-03-28T14:08:11.557 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:11.551+0000 7fc44d057000 1 --2- 172.21.15.104:0/737019030 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55efb8c81730 0x55efb8c99110 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:11.557 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:11.551+0000 7fc44d057000 1 --2- 172.21.15.104:0/737019030 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x55efb8c80e10 0x55efb8c98bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:11.557 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:11.551+0000 7fc44d057000 1 --2- 172.21.15.104:0/737019030 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55efb8c5ce00 0x55efb8c57880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:11.557 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:11.551+0000 7fc44d057000 1 -- 172.21.15.104:0/737019030 >> 172.21.15.104:0/737019030 conn(0x55efb8b3fb80 msgr2=0x55efb8ca1ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T14:08:11.557 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:11.551+0000 7fc44d057000 1 -- 172.21.15.104:0/737019030 shutdown_connections 2024-03-28T14:08:11.557 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:11.551+0000 7fc44d057000 1 -- 172.21.15.104:0/737019030 wait complete. 2024-03-28T14:08:11.557 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:11.551+0000 7fc44d057000 1 -- start start 2024-03-28T14:08:11.557 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:11.551+0000 7fc44d057000 1 Processor -- start 2024-03-28T14:08:11.557 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:11.551+0000 7fc44d057000 1 --2- >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x55efb8c5ce00 0x55efb8c95910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:11.557 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:11.551+0000 7fc44d057000 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55efb8c80e10 0x55efb8c95e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:11.557 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:11.551+0000 7fc44d057000 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55efb8c81730 0x55efb8c96390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:11.558 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:11.551+0000 7fc44d057000 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x55efb8aabf80 con 0x55efb8c80e10 2024-03-28T14:08:11.558 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:11.551+0000 7fc44d057000 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x55efb8aa6c60 con 0x55efb8c81730 2024-03-28T14:08:11.558 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:11.551+0000 7fc44d057000 1 -- --> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] -- mon_getmap magic: 0 -- 0x55efb8b3c3c0 con 0x55efb8c5ce00 2024-03-28T14:08:11.558 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:11.551+0000 7fc44d057000 10 client.0 _ll_register_callbacks cb 0x7fc43c0039a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T14:08:11.558 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:11.551+0000 7fc43bfff640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55efb8c80e10 0x55efb8c95e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:11.558 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:11.551+0000 7fc43bfff640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55efb8c80e10 0x55efb8c95e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:34984/0 (socket says 172.21.15.104:34984) 2024-03-28T14:08:11.558 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:11.551+0000 7fc43bfff640 1 -- 172.21.15.104:0/629086071 learned_addr learned my addr 172.21.15.104:0/629086071 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-03-28T14:08:11.558 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:11.551+0000 7fc443fff640 1 --2- 172.21.15.104:0/629086071 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55efb8c81730 0x55efb8c96390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:11.558 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:11.551+0000 7fc4437fe640 1 --2- 172.21.15.104:0/629086071 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x55efb8c5ce00 0x55efb8c95910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:11.559 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:11.551+0000 7fc43bfff640 1 -- 172.21.15.104:0/629086071 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x55efb8c5ce00 msgr2=0x55efb8c95910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T14:08:11.559 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:11.551+0000 7fc43bfff640 1 --2- 172.21.15.104:0/629086071 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x55efb8c5ce00 0x55efb8c95910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:11.559 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:11.551+0000 7fc43bfff640 1 -- 172.21.15.104:0/629086071 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55efb8c81730 msgr2=0x55efb8c96390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T14:08:11.559 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:11.551+0000 7fc43bfff640 1 --2- 172.21.15.104:0/629086071 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55efb8c81730 0x55efb8c96390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:11.559 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:11.551+0000 7fc43bfff640 1 -- 172.21.15.104:0/629086071 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55efb8a62490 con 0x55efb8c80e10 2024-03-28T14:08:11.559 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:11.551+0000 7fc4437fe640 1 --2- 172.21.15.104:0/629086071 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x55efb8c5ce00 0x55efb8c95910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T14:08:11.559 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:11.551+0000 7fc43bfff640 1 --2- 172.21.15.104:0/629086071 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55efb8c80e10 0x55efb8c95e50 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fc42c052580 tx=0x7fc42c0568e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8d0ae572d8928876 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:08:11.559 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:11.551+0000 7fc4227fc640 10 client.0 ms_handle_connect on v2:172.21.15.104:3300/0 2024-03-28T14:08:11.559 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:11.551+0000 7fc4227fc640 1 -- 172.21.15.104:0/629086071 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc42c05ed00 con 0x55efb8c80e10 2024-03-28T14:08:11.559 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:11.551+0000 7fc4227fc640 1 -- 172.21.15.104:0/629086071 <== mon.0 v2:172.21.15.104:3300/0 2 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7fc42c06fdc0 con 0x55efb8c80e10 2024-03-28T14:08:11.559 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:11.551+0000 7fc44d057000 1 -- 172.21.15.104:0/629086071 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc43c01fa20 con 0x55efb8c80e10 2024-03-28T14:08:11.559 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:11.551+0000 7fc4227fc640 1 -- 172.21.15.104:0/629086071 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_get_version_reply(handle=1 version=7) ==== 24+0+0 (secure 0 0 0) 0x7fc42c05a120 con 0x55efb8c80e10 2024-03-28T14:08:11.559 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:11.551+0000 7fc44d057000 10 client.4608 fetch_fsmap learned FSMap version 7 2024-03-28T14:08:11.559 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:11.551+0000 7fc44d057000 1 -- 172.21.15.104:0/629086071 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,fsmap=7,monmap=2+}) -- 0x55efb8a60e80 con 0x55efb8c80e10 2024-03-28T14:08:11.560 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:11.551+0000 7fc4227fc640 1 -- 172.21.15.104:0/629086071 <== mon.0 v2:172.21.15.104:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc43c01fa20 con 0x55efb8c80e10 2024-03-28T14:08:11.560 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:11.551+0000 7fc4227fc640 1 -- 172.21.15.104:0/629086071 <== mon.0 v2:172.21.15.104:3300/0 5 ==== fsmap(e 7) ==== 2284+0+0 (secure 0 0 0) 0x7fc42c05e870 con 0x55efb8c80e10 2024-03-28T14:08:11.560 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:11.551+0000 7fc44d057000 10 client.4608 fetch_fsmap finished waiting for FSMap version 7 2024-03-28T14:08:11.560 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:11.551+0000 7fc44d057000 10 client.4608 resolve_mds: resolved name 'a' to daemon mds.a 2024-03-28T14:08:11.560 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:11.551+0000 7fc44d057000 20 client.4608 populate_metadata read hostname 'smithi104' 2024-03-28T14:08:11.560 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:11.551+0000 7fc44d057000 1 --2- 172.21.15.104:0/629086071 >> [v2:172.21.15.104:6832/3184673325,v1:172.21.15.104:6833/3184673325] conn(0x55efb8d018d0 0x55efb8d03cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:11.560 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:11.551+0000 7fc44d057000 4 client.4608 mds_command: new command op to 4245 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-28T14:08:11.560 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:11.551+0000 7fc44d057000 1 -- 172.21.15.104:0/629086071 --> [v2:172.21.15.104:6832/3184673325,v1:172.21.15.104:6833/3184673325] -- command(tid 0: {"prefix": "status"}) -- 0x55efb8ab4e90 con 0x55efb8d018d0 2024-03-28T14:08:11.560 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:11.551+0000 7fc4437fe640 1 --2- 172.21.15.104:0/629086071 >> [v2:172.21.15.104:6832/3184673325,v1:172.21.15.104:6833/3184673325] conn(0x55efb8d018d0 0x55efb8d03cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:11.560 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:11.551+0000 7fc4227fc640 10 client.4608 ms_handle_connect on v2:172.21.15.104:6832/3184673325 2024-03-28T14:08:11.560 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:11.551+0000 7fc4437fe640 1 --2- 172.21.15.104:0/629086071 >> [v2:172.21.15.104:6832/3184673325,v1:172.21.15.104:6833/3184673325] conn(0x55efb8d018d0 0x55efb8d03cc0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=34ffd9403e2187cf server_cookie=4e08758e629b3d55 in_seq=0 out_seq=0 2024-03-28T14:08:11.560 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:11.551+0000 7fc4227fc640 1 -- 172.21.15.104:0/629086071 <== mds.0 v2:172.21.15.104:6832/3184673325 1 ==== command_reply(tid 0: 0 ) ==== 8+0+324 (crc 0 0 0) 0x55efb8ab4e90 con 0x55efb8d018d0 2024-03-28T14:08:11.560 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:11.555+0000 7fc4227fc640 10 client.4608 handle_command_reply: tid=0 2024-03-28T14:08:11.560 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:11.555+0000 7fc44d057000 1 client.4608 shutdown 2024-03-28T14:08:11.561 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:11.555+0000 7fc44d057000 1 -- 172.21.15.104:0/629086071 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55efb8c80e10 msgr2=0x55efb8c95e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:11.561 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:11.555+0000 7fc44d057000 1 --2- 172.21.15.104:0/629086071 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55efb8c80e10 0x55efb8c95e50 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fc42c052580 tx=0x7fc42c0568e0 comp rx=0 tx=0).stop 2024-03-28T14:08:11.561 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:11.555+0000 7fc44d057000 1 -- 172.21.15.104:0/629086071 shutdown_connections 2024-03-28T14:08:11.561 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:11.555+0000 7fc44d057000 1 --2- 172.21.15.104:0/629086071 >> [v2:172.21.15.104:6832/3184673325,v1:172.21.15.104:6833/3184673325] conn(0x55efb8d018d0 0x55efb8d03cc0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:11.561 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:11.555+0000 7fc44d057000 1 --2- 172.21.15.104:0/629086071 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55efb8c81730 0x55efb8c96390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:11.561 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:11.555+0000 7fc44d057000 1 --2- 172.21.15.104:0/629086071 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55efb8c80e10 0x55efb8c95e50 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:11.561 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:11.555+0000 7fc4227fc640 0 client.4608 ms_handle_reset on v2:172.21.15.104:6832/3184673325 2024-03-28T14:08:11.561 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:11.555+0000 7fc44d057000 1 --2- 172.21.15.104:0/629086071 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x55efb8c5ce00 0x55efb8c95910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:11.561 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:11.555+0000 7fc44d057000 1 -- 172.21.15.104:0/629086071 >> 172.21.15.104:0/629086071 conn(0x55efb8b3fb80 msgr2=0x55efb8c57a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T14:08:11.561 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:11.555+0000 7fc44d057000 1 -- 172.21.15.104:0/629086071 shutdown_connections 2024-03-28T14:08:11.561 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:11.555+0000 7fc44d057000 1 -- 172.21.15.104:0/629086071 wait complete. 2024-03-28T14:08:11.561 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:11.555+0000 7fc44d057000 20 client.4608 trim_cache size 0 max 16384 2024-03-28T14:08:11.562 INFO:teuthology.orchestra.run.smithi104.stdout:{ 2024-03-28T14:08:11.562 INFO:teuthology.orchestra.run.smithi104.stdout: "cluster_fsid": "428a90f2-3351-435f-8fcf-f7e66c78a13e", 2024-03-28T14:08:11.562 INFO:teuthology.orchestra.run.smithi104.stdout: "whoami": 0, 2024-03-28T14:08:11.562 INFO:teuthology.orchestra.run.smithi104.stdout: "id": 4245, 2024-03-28T14:08:11.562 INFO:teuthology.orchestra.run.smithi104.stdout: "want_state": "up:active", 2024-03-28T14:08:11.562 INFO:teuthology.orchestra.run.smithi104.stdout: "state": "up:active", 2024-03-28T14:08:11.562 INFO:teuthology.orchestra.run.smithi104.stdout: "fs_name": "cephfs", 2024-03-28T14:08:11.562 INFO:teuthology.orchestra.run.smithi104.stdout: "rank_uptime": 11.868548009, 2024-03-28T14:08:11.562 INFO:teuthology.orchestra.run.smithi104.stdout: "mdsmap_epoch": 7, 2024-03-28T14:08:11.562 INFO:teuthology.orchestra.run.smithi104.stdout: "osdmap_epoch": 21, 2024-03-28T14:08:11.562 INFO:teuthology.orchestra.run.smithi104.stdout: "osdmap_epoch_barrier": 21, 2024-03-28T14:08:11.562 INFO:teuthology.orchestra.run.smithi104.stdout: "uptime": 18.811639209999999 2024-03-28T14:08:11.562 INFO:teuthology.orchestra.run.smithi104.stdout:} 2024-03-28T14:08:11.563 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:11.555+0000 7fc449ed8640 1 -- 172.21.15.104:0/519619698 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7fc424067ff0 msgr2=0x7fc42406a4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:11.563 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:11.555+0000 7fc449ed8640 1 --2- 172.21.15.104:0/519619698 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7fc424067ff0 0x7fc42406a4b0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fc4300057f0 tx=0x7fc4340011a0 comp rx=0 tx=0).stop 2024-03-28T14:08:11.563 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:11.555+0000 7fc449ed8640 1 -- 172.21.15.104:0/519619698 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc4440b3bd0 msgr2=0x7fc444073530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:11.564 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:11.555+0000 7fc449ed8640 1 --2- 172.21.15.104:0/519619698 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc4440b3bd0 0x7fc444073530 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fc42c00e730 tx=0x7fc42c00ebf0 comp rx=0 tx=0).stop 2024-03-28T14:08:11.564 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:11.555+0000 7fc449ed8640 1 -- 172.21.15.104:0/519619698 shutdown_connections 2024-03-28T14:08:11.564 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:11.555+0000 7fc449ed8640 1 --2- 172.21.15.104:0/519619698 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fc444071770 0x7fc4441de200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:11.564 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:11.555+0000 7fc449ed8640 1 --2- 172.21.15.104:0/519619698 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7fc424067ff0 0x7fc42406a4b0 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:11.564 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:11.555+0000 7fc449ed8640 1 --2- 172.21.15.104:0/519619698 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc4440b3bd0 0x7fc444073530 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:11.564 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:11.555+0000 7fc449ed8640 1 --2- 172.21.15.104:0/519619698 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fc4300039f0 0x7fc44409b0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:11.564 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:11.555+0000 7fc449ed8640 1 -- 172.21.15.104:0/519619698 >> 172.21.15.104:0/519619698 conn(0x7fc4440b95f0 msgr2=0x7fc44413d980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T14:08:11.564 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:11.555+0000 7fc449ed8640 1 -- 172.21.15.104:0/519619698 shutdown_connections 2024-03-28T14:08:11.564 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:11.555+0000 7fc449ed8640 1 -- 172.21.15.104:0/519619698 wait complete. 2024-03-28T14:08:11.578 INFO:teuthology.run_tasks:Running task sequential... 2024-03-28T14:08:11.589 INFO:teuthology.task.sequential:In sequential, running task sequential... 2024-03-28T14:08:11.589 INFO:teuthology.task.sequential:In sequential, running task print... 2024-03-28T14:08:11.589 INFO:teuthology.task.print:Enabling mgr modules 2024-03-28T14:08:11.589 INFO:teuthology.run_tasks:Running task ceph-fuse... 2024-03-28T14:08:11.624 INFO:tasks.ceph_fuse:Running ceph_fuse task... 2024-03-28T14:08:11.624 INFO:tasks.ceph_fuse:config is {'client.0': {}} 2024-03-28T14:08:11.625 INFO:tasks.ceph_fuse:client.0 config is {} 2024-03-28T14:08:11.625 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-03-28T14:08:11.625 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-03-28T14:08:11.625 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T14:08:11.625 DEBUG:teuthology.orchestra.run.smithi104:> ip netns list 2024-03-28T14:08:11.631 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T14:08:11.631 DEBUG:teuthology.orchestra.run.smithi104:> sudo ip link delete ceph-brx 2024-03-28T14:08:11.684 INFO:teuthology.orchestra.run.smithi104.stderr:Cannot find device "ceph-brx" 2024-03-28T14:08:11.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-28T14:08:11.686 INFO:tasks.ceph_fuse:Mounting ceph-fuse clients... 2024-03-28T14:08:11.686 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-03-28T14:08:11.852 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:11.851+0000 7fdf0e658640 1 Processor -- start 2024-03-28T14:08:11.852 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:11.851+0000 7fdf0e658640 1 -- start start 2024-03-28T14:08:11.853 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:11.851+0000 7fdf0e658640 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fdf08141b00 0x7fdf08153090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:11.853 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:11.851+0000 7fdf0e658640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdf080b5050 con 0x7fdf0814b640 2024-03-28T14:08:11.853 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:11.851+0000 7fdf0e658640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdf080bddc0 con 0x7fdf080bd9e0 2024-03-28T14:08:11.853 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:11.851+0000 7fdf0e658640 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 -- 0x7fdf080b70b0 con 0x7fdf08141b00 2024-03-28T14:08:11.853 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:11.851+0000 7fdf077fe640 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7fdf0814b640 0x7fdf081413f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:41716/0 (socket says 172.21.15.104:41716) 2024-03-28T14:08:11.853 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:11.851+0000 7fdf077fe640 1 -- 172.21.15.104:0/3124656514 learned_addr learned my addr 172.21.15.104:0/3124656514 (peer_addr_for_me v1:172.21.15.104:0/0) 2024-03-28T14:08:11.854 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:11.851+0000 7fdf0cbf5640 1 --2- 172.21.15.104:0/3124656514 >> v2:172.21.15.104:3300/0 conn(0x7fdf08141b00 0x7fdf08153090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:11.854 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:11.851+0000 7fdf06ffd640 1 -- 172.21.15.104:0/3124656514 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2842384243 0 0) 0x7fdf080b5050 con 0x7fdf0814b640 2024-03-28T14:08:11.854 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:11.851+0000 7fdf06ffd640 1 -- 172.21.15.104:0/3124656514 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fdef4003530 con 0x7fdf0814b640 2024-03-28T14:08:11.854 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:11.851+0000 7fdf0cbf5640 1 -- 172.21.15.104:0/3124656514 >> v1:172.21.15.171:6789/0 conn(0x7fdf080bd9e0 legacy=0x7fdf0814af30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:11.854 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:11.851+0000 7fdf0cbf5640 1 -- 172.21.15.104:0/3124656514 >> v1:172.21.15.104:6789/0 conn(0x7fdf0814b640 legacy=0x7fdf081413f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:11.854 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:11.851+0000 7fdf0cbf5640 1 -- 172.21.15.104:0/3124656514 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdf080b52f0 con 0x7fdf08141b00 2024-03-28T14:08:11.855 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:11.851+0000 7fdf0cbf5640 1 --2- 172.21.15.104:0/3124656514 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fdf08141b00 0x7fdf08153090 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fdf000097c0 tx=0x7fdf0001c350 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1738056960f6e677 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:08:11.855 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:11.855+0000 7fdf06ffd640 1 -- 172.21.15.104:0/3124656514 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf000b0070 con 0x7fdf08141b00 2024-03-28T14:08:11.855 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:11.855+0000 7fdf06ffd640 1 -- 172.21.15.104:0/3124656514 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdf000ab600 con 0x7fdf08141b00 2024-03-28T14:08:11.856 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:11.855+0000 7fdf06ffd640 1 -- 172.21.15.104:0/3124656514 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf000ab9d0 con 0x7fdf08141b00 2024-03-28T14:08:11.856 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:11.855+0000 7fdf0e658640 1 -- 172.21.15.104:0/3124656514 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fdf08141b00 msgr2=0x7fdf08153090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:11.857 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:11.855+0000 7fdf0e658640 1 --2- 172.21.15.104:0/3124656514 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fdf08141b00 0x7fdf08153090 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fdf000097c0 tx=0x7fdf0001c350 comp rx=0 tx=0).stop 2024-03-28T14:08:11.857 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:11.855+0000 7fdf0e658640 1 -- 172.21.15.104:0/3124656514 shutdown_connections 2024-03-28T14:08:11.857 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:11.855+0000 7fdf0e658640 1 --2- 172.21.15.104:0/3124656514 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fdf08141b00 0x7fdf08153090 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:11.857 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:11.855+0000 7fdf0e658640 1 -- 172.21.15.104:0/3124656514 >> 172.21.15.104:0/3124656514 conn(0x7fdf08073530 msgr2=0x7fdf080bd070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T14:08:11.857 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:11.855+0000 7fdf0e658640 1 -- 172.21.15.104:0/3124656514 shutdown_connections 2024-03-28T14:08:11.857 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:11.855+0000 7fdf0e658640 1 -- 172.21.15.104:0/3124656514 wait complete. 2024-03-28T14:08:11.859 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:11.855+0000 7fdf0e658640 1 Processor -- start 2024-03-28T14:08:11.859 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:11.855+0000 7fdf0e658640 1 -- start start 2024-03-28T14:08:11.859 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:11.855+0000 7fdf0e658640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fdf080bd9e0 0x7fdf0814a680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:11.859 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:11.855+0000 7fdf0e658640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fdf08141b00 0x7fdf081dfdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:11.859 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:11.855+0000 7fdf0e658640 1 --2- >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fdf0814b640 0x7fdf081332a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:11.859 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:11.855+0000 7fdf0e658640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7fdf080b4ac0 con 0x7fdf080bd9e0 2024-03-28T14:08:11.859 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:11.855+0000 7fdf0e658640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7fdf080b4940 con 0x7fdf08141b00 2024-03-28T14:08:11.859 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:11.855+0000 7fdf0e658640 1 -- --> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] -- mon_getmap magic: 0 -- 0x7fdf08155df0 con 0x7fdf0814b640 2024-03-28T14:08:11.859 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:11.855+0000 7fdf07fff640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fdf080bd9e0 0x7fdf0814a680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:11.860 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:11.855+0000 7fdf07fff640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fdf080bd9e0 0x7fdf0814a680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:35002/0 (socket says 172.21.15.104:35002) 2024-03-28T14:08:11.860 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:11.855+0000 7fdf07fff640 1 -- 172.21.15.104:0/2635858692 learned_addr learned my addr 172.21.15.104:0/2635858692 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-03-28T14:08:11.860 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:11.855+0000 7fdf077fe640 1 --2- 172.21.15.104:0/2635858692 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fdf08141b00 0x7fdf081dfdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:11.860 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:11.855+0000 7fdf07fff640 1 -- 172.21.15.104:0/2635858692 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fdf0814b640 msgr2=0x7fdf081332a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:11.860 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:11.855+0000 7fdf07fff640 1 --2- 172.21.15.104:0/2635858692 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fdf0814b640 0x7fdf081332a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:11.860 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:11.855+0000 7fdf07fff640 1 -- 172.21.15.104:0/2635858692 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fdf08141b00 msgr2=0x7fdf081dfdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:11.860 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:11.855+0000 7fdf07fff640 1 --2- 172.21.15.104:0/2635858692 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fdf08141b00 0x7fdf081dfdd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:11.860 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:11.855+0000 7fdf07fff640 1 -- 172.21.15.104:0/2635858692 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdf0813f2d0 con 0x7fdf080bd9e0 2024-03-28T14:08:11.860 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:11.855+0000 7fdf077fe640 1 --2- 172.21.15.104:0/2635858692 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fdf08141b00 0x7fdf081dfdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T14:08:11.860 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:11.855+0000 7fdf07fff640 1 --2- 172.21.15.104:0/2635858692 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fdf080bd9e0 0x7fdf0814a680 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fdef000c7c0 tx=0x7fdef000cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:08:11.860 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:11.855+0000 7fdf04ff9640 1 -- 172.21.15.104:0/2635858692 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdef0016020 con 0x7fdf080bd9e0 2024-03-28T14:08:11.860 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:11.855+0000 7fdf04ff9640 1 -- 172.21.15.104:0/2635858692 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdef0006d60 con 0x7fdf080bd9e0 2024-03-28T14:08:11.860 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:11.855+0000 7fdf04ff9640 1 -- 172.21.15.104:0/2635858692 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdef0003bc0 con 0x7fdf080bd9e0 2024-03-28T14:08:11.860 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:11.855+0000 7fdf0e658640 1 -- 172.21.15.104:0/2635858692 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdf0813f820 con 0x7fdf080bd9e0 2024-03-28T14:08:11.861 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:11.855+0000 7fdf0e658640 1 -- 172.21.15.104:0/2635858692 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdf08138a80 con 0x7fdf080bd9e0 2024-03-28T14:08:11.862 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:11.859+0000 7fdf0e658640 1 -- 172.21.15.104:0/2635858692 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdf08138890 con 0x7fdf080bd9e0 2024-03-28T14:08:11.862 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:11.859+0000 7fdf04ff9640 1 -- 172.21.15.104:0/2635858692 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) ==== 87803+0+0 (secure 0 0 0) 0x7fdef000a850 con 0x7fdf080bd9e0 2024-03-28T14:08:11.862 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:11.859+0000 7fdf04ff9640 1 --2- 172.21.15.104:0/2635858692 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7fdee8067ff0 0x7fdee806a4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:11.862 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:11.859+0000 7fdf04ff9640 1 -- 172.21.15.104:0/2635858692 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7fdef00861e0 con 0x7fdf080bd9e0 2024-03-28T14:08:11.862 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:11.859+0000 7fdf077fe640 1 --2- 172.21.15.104:0/2635858692 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7fdee8067ff0 0x7fdee806a4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:11.863 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:11.859+0000 7fdf077fe640 1 --2- 172.21.15.104:0/2635858692 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7fdee8067ff0 0x7fdee806a4b0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fdf081475e0 tx=0x7fdef8009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:08:11.870 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:11.867+0000 7fdf04ff9640 1 -- 172.21.15.104:0/2635858692 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179117 (secure 0 0 0) 0x7fdef0020020 con 0x7fdf080bd9e0 2024-03-28T14:08:12.074 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:12.071+0000 7fdf0e658640 1 -- 172.21.15.104:0/2635858692 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7fdf080687c0 con 0x7fdf080bd9e0 2024-03-28T14:08:12.074 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:12.071+0000 7fdf04ff9640 1 -- 172.21.15.104:0/2635858692 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v7) ==== 53+0+73 (secure 0 0 0) 0x7fdef0019310 con 0x7fdf080bd9e0 2024-03-28T14:08:12.075 INFO:teuthology.orchestra.run.smithi104.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-03-28T14:08:12.077 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:12.075+0000 7fdf0e658640 1 -- 172.21.15.104:0/2635858692 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7fdee8067ff0 msgr2=0x7fdee806a4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:12.077 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:12.075+0000 7fdf0e658640 1 --2- 172.21.15.104:0/2635858692 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7fdee8067ff0 0x7fdee806a4b0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fdf081475e0 tx=0x7fdef8009040 comp rx=0 tx=0).stop 2024-03-28T14:08:12.077 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:12.075+0000 7fdf0e658640 1 -- 172.21.15.104:0/2635858692 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fdf080bd9e0 msgr2=0x7fdf0814a680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:12.077 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:12.075+0000 7fdf0e658640 1 --2- 172.21.15.104:0/2635858692 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fdf080bd9e0 0x7fdf0814a680 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fdef000c7c0 tx=0x7fdef000cc80 comp rx=0 tx=0).stop 2024-03-28T14:08:12.077 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:12.075+0000 7fdf0e658640 1 -- 172.21.15.104:0/2635858692 shutdown_connections 2024-03-28T14:08:12.077 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:12.075+0000 7fdf0e658640 1 --2- 172.21.15.104:0/2635858692 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fdf0814b640 0x7fdf081332a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:12.077 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:12.075+0000 7fdf0e658640 1 --2- 172.21.15.104:0/2635858692 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fdf08141b00 0x7fdf081dfdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:12.077 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:12.075+0000 7fdf0e658640 1 --2- 172.21.15.104:0/2635858692 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7fdee8067ff0 0x7fdee806a4b0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:12.077 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:12.075+0000 7fdf0e658640 1 --2- 172.21.15.104:0/2635858692 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fdf080bd9e0 0x7fdf0814a680 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:12.078 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:12.075+0000 7fdf0e658640 1 -- 172.21.15.104:0/2635858692 >> 172.21.15.104:0/2635858692 conn(0x7fdf08073530 msgr2=0x7fdf081411c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T14:08:12.078 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:12.075+0000 7fdf0e658640 1 -- 172.21.15.104:0/2635858692 shutdown_connections 2024-03-28T14:08:12.078 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:12.075+0000 7fdf0e658640 1 -- 172.21.15.104:0/2635858692 wait complete. 2024-03-28T14:08:12.093 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-03-28T14:08:12.093 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-28T14:08:12.093 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi104.front.sepia.ceph.com 2024-03-28T14:08:12.094 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-03-28T14:08:12.094 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-03-28T14:08:12.094 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-03-28T14:08:12.094 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-03-28T14:08:12.094 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-28T14:08:12.094 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-28T14:08:12.094 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T14:08:12.094 DEBUG:teuthology.orchestra.run.smithi104:> ip addr 2024-03-28T14:08:12.100 INFO:teuthology.orchestra.run.smithi104.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-28T14:08:12.100 INFO:teuthology.orchestra.run.smithi104.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-28T14:08:12.100 INFO:teuthology.orchestra.run.smithi104.stdout: inet 127.0.0.1/8 scope host lo 2024-03-28T14:08:12.100 INFO:teuthology.orchestra.run.smithi104.stdout: valid_lft forever preferred_lft forever 2024-03-28T14:08:12.100 INFO:teuthology.orchestra.run.smithi104.stdout: inet6 ::1/128 scope host 2024-03-28T14:08:12.100 INFO:teuthology.orchestra.run.smithi104.stdout: valid_lft forever preferred_lft forever 2024-03-28T14:08:12.100 INFO:teuthology.orchestra.run.smithi104.stdout:2: eno1: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-03-28T14:08:12.100 INFO:teuthology.orchestra.run.smithi104.stdout: link/ether 0c:c4:7a:d8:a1:1c brd ff:ff:ff:ff:ff:ff 2024-03-28T14:08:12.100 INFO:teuthology.orchestra.run.smithi104.stdout: altname enp7s0f0 2024-03-28T14:08:12.101 INFO:teuthology.orchestra.run.smithi104.stdout:3: eno2: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-03-28T14:08:12.101 INFO:teuthology.orchestra.run.smithi104.stdout: link/ether 0c:c4:7a:d8:a1:1d brd ff:ff:ff:ff:ff:ff 2024-03-28T14:08:12.101 INFO:teuthology.orchestra.run.smithi104.stdout: altname enp7s0f1 2024-03-28T14:08:12.101 INFO:teuthology.orchestra.run.smithi104.stdout:4: enp3s0f0: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-03-28T14:08:12.101 INFO:teuthology.orchestra.run.smithi104.stdout: link/ether 0c:c4:7a:8f:ce:88 brd ff:ff:ff:ff:ff:ff 2024-03-28T14:08:12.101 INFO:teuthology.orchestra.run.smithi104.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-28T14:08:12.101 INFO:teuthology.orchestra.run.smithi104.stdout: link/ether 0c:c4:7a:8f:ce:89 brd ff:ff:ff:ff:ff:ff 2024-03-28T14:08:12.101 INFO:teuthology.orchestra.run.smithi104.stdout: inet 172.21.15.104/20 metric 100 brd 172.21.15.255 scope global dynamic enp3s0f1 2024-03-28T14:08:12.101 INFO:teuthology.orchestra.run.smithi104.stdout: valid_lft 42500sec preferred_lft 42500sec 2024-03-28T14:08:12.101 INFO:teuthology.orchestra.run.smithi104.stdout: inet6 fe80::ec4:7aff:fe8f:ce89/64 scope link 2024-03-28T14:08:12.101 INFO:teuthology.orchestra.run.smithi104.stdout: valid_lft forever preferred_lft forever 2024-03-28T14:08:12.101 INFO:teuthology.orchestra.run.smithi104.stdout:6: docker0: mtu 1500 qdisc noqueue state DOWN group default 2024-03-28T14:08:12.101 INFO:teuthology.orchestra.run.smithi104.stdout: link/ether 02:42:4f:83:9c:9e brd ff:ff:ff:ff:ff:ff 2024-03-28T14:08:12.102 INFO:teuthology.orchestra.run.smithi104.stdout: inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0 2024-03-28T14:08:12.102 INFO:teuthology.orchestra.run.smithi104.stdout: valid_lft forever preferred_lft forever 2024-03-28T14:08:12.102 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-03-28T14:08:12.102 DEBUG:teuthology.orchestra.run.smithi104:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-28T14:08:12.102 DEBUG:teuthology.orchestra.run.smithi104:> set -e 2024-03-28T14:08:12.102 DEBUG:teuthology.orchestra.run.smithi104:> sudo ip link add name ceph-brx type bridge 2024-03-28T14:08:12.102 DEBUG:teuthology.orchestra.run.smithi104:> sudo ip addr flush dev ceph-brx 2024-03-28T14:08:12.103 DEBUG:teuthology.orchestra.run.smithi104:> sudo ip link set ceph-brx up 2024-03-28T14:08:12.103 DEBUG:teuthology.orchestra.run.smithi104:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-03-28T14:08:12.103 DEBUG:teuthology.orchestra.run.smithi104:> ') 2024-03-28T14:08:12.179 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:12 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-28T14:08:12.210 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:12 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-28T14:08:12.216 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T14:08:12.216 DEBUG:teuthology.orchestra.run.smithi104:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-28T14:08:12.271 INFO:teuthology.orchestra.run.smithi104.stdout:1 2024-03-28T14:08:12.272 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T14:08:12.272 DEBUG:teuthology.orchestra.run.smithi104:> route 2024-03-28T14:08:12.346 INFO:teuthology.orchestra.run.smithi104.stdout:Kernel IP routing table 2024-03-28T14:08:12.346 INFO:teuthology.orchestra.run.smithi104.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-28T14:08:12.346 INFO:teuthology.orchestra.run.smithi104.stdout:default _gateway 0.0.0.0 UG 0 0 0 enp3s0f1 2024-03-28T14:08:12.346 INFO:teuthology.orchestra.run.smithi104.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-03-28T14:08:12.347 INFO:teuthology.orchestra.run.smithi104.stdout:158.69.67.47 _gateway 255.255.255.255 UGH 100 0 0 enp3s0f1 2024-03-28T14:08:12.347 INFO:teuthology.orchestra.run.smithi104.stdout:172.17.0.0 0.0.0.0 255.255.0.0 U 0 0 0 docker0 2024-03-28T14:08:12.347 INFO:teuthology.orchestra.run.smithi104.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-03-28T14:08:12.347 INFO:teuthology.orchestra.run.smithi104.stdout:ns1.front.sepia 0.0.0.0 255.255.255.255 UH 100 0 0 enp3s0f1 2024-03-28T14:08:12.347 INFO:teuthology.orchestra.run.smithi104.stdout:172.21.0.2 0.0.0.0 255.255.255.255 UH 100 0 0 enp3s0f1 2024-03-28T14:08:12.347 INFO:teuthology.orchestra.run.smithi104.stdout:_gateway 0.0.0.0 255.255.255.255 UH 100 0 0 enp3s0f1 2024-03-28T14:08:12.347 INFO:teuthology.orchestra.run.smithi104.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-28T14:08:12.347 DEBUG:teuthology.orchestra.run.smithi104:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-28T14:08:12.348 DEBUG:teuthology.orchestra.run.smithi104:> set -e 2024-03-28T14:08:12.348 DEBUG:teuthology.orchestra.run.smithi104:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-03-28T14:08:12.348 DEBUG:teuthology.orchestra.run.smithi104:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-03-28T14:08:12.348 DEBUG:teuthology.orchestra.run.smithi104:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-03-28T14:08:12.348 DEBUG:teuthology.orchestra.run.smithi104:> ') 2024-03-28T14:08:12.427 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:12 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-28T14:08:12.447 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:12 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-28T14:08:12.452 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T14:08:12.453 DEBUG:teuthology.orchestra.run.smithi104:> ip netns list 2024-03-28T14:08:12.501 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T14:08:12.501 DEBUG:teuthology.orchestra.run.smithi104:> ip netns list-id 2024-03-28T14:08:12.548 DEBUG:teuthology.orchestra.run.smithi104:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-28T14:08:12.548 DEBUG:teuthology.orchestra.run.smithi104:> set -e 2024-03-28T14:08:12.548 DEBUG:teuthology.orchestra.run.smithi104:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-28T14:08:12.548 DEBUG:teuthology.orchestra.run.smithi104:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.0 0 2024-03-28T14:08:12.548 DEBUG:teuthology.orchestra.run.smithi104:> ') 2024-03-28T14:08:12.633 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:12 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-28T14:08:12.649 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:12 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-28T14:08:12.654 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.0' with 192.168.0.1/16 2024-03-28T14:08:12.655 DEBUG:teuthology.orchestra.run.smithi104:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-28T14:08:12.655 DEBUG:teuthology.orchestra.run.smithi104:> set -e 2024-03-28T14:08:12.655 DEBUG:teuthology.orchestra.run.smithi104:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.0 type veth peer name brx.0 2024-03-28T14:08:12.655 DEBUG:teuthology.orchestra.run.smithi104:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-03-28T14:08:12.655 DEBUG:teuthology.orchestra.run.smithi104:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set veth0 up 2024-03-28T14:08:12.655 DEBUG:teuthology.orchestra.run.smithi104:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set lo up 2024-03-28T14:08:12.655 DEBUG:teuthology.orchestra.run.smithi104:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip route add default via 192.168.255.254 2024-03-28T14:08:12.655 DEBUG:teuthology.orchestra.run.smithi104:> ') 2024-03-28T14:08:12.733 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:12 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-28T14:08:12.790 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:12 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-28T14:08:12.795 DEBUG:teuthology.orchestra.run.smithi104:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-28T14:08:12.796 DEBUG:teuthology.orchestra.run.smithi104:> set -e 2024-03-28T14:08:12.796 DEBUG:teuthology.orchestra.run.smithi104:> sudo ip link set brx.0 up 2024-03-28T14:08:12.796 DEBUG:teuthology.orchestra.run.smithi104:> sudo ip link set dev brx.0 master ceph-brx 2024-03-28T14:08:12.796 DEBUG:teuthology.orchestra.run.smithi104:> ') 2024-03-28T14:08:12.872 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:12 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-28T14:08:12.887 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:12 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-28T14:08:12.893 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2024-03-28T14:08:12.893 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-28T14:08:12.893 DEBUG:teuthology.orchestra.run.smithi104:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-03-28T14:08:12.939 INFO:teuthology.orchestra.run.smithi104.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-03-28T14:08:12.939 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-28T14:08:12.939 DEBUG:teuthology.orchestra.run.smithi104:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-03-28T14:08:12.987 DEBUG:teuthology.orchestra.run.smithi104:> sudo modprobe fuse 2024-03-28T14:08:13.037 DEBUG:teuthology.orchestra.run.smithi104:> cat /proc/self/mounts | awk '{print $2}' 2024-03-28T14:08:13.084 INFO:teuthology.orchestra.run.smithi104.stdout:/sys 2024-03-28T14:08:13.084 INFO:teuthology.orchestra.run.smithi104.stdout:/proc 2024-03-28T14:08:13.084 INFO:teuthology.orchestra.run.smithi104.stdout:/dev 2024-03-28T14:08:13.084 INFO:teuthology.orchestra.run.smithi104.stdout:/dev/pts 2024-03-28T14:08:13.084 INFO:teuthology.orchestra.run.smithi104.stdout:/run 2024-03-28T14:08:13.084 INFO:teuthology.orchestra.run.smithi104.stdout:/ 2024-03-28T14:08:13.084 INFO:teuthology.orchestra.run.smithi104.stdout:/sys/kernel/security 2024-03-28T14:08:13.084 INFO:teuthology.orchestra.run.smithi104.stdout:/dev/shm 2024-03-28T14:08:13.084 INFO:teuthology.orchestra.run.smithi104.stdout:/run/lock 2024-03-28T14:08:13.084 INFO:teuthology.orchestra.run.smithi104.stdout:/sys/fs/cgroup 2024-03-28T14:08:13.085 INFO:teuthology.orchestra.run.smithi104.stdout:/sys/fs/pstore 2024-03-28T14:08:13.085 INFO:teuthology.orchestra.run.smithi104.stdout:/sys/fs/bpf 2024-03-28T14:08:13.085 INFO:teuthology.orchestra.run.smithi104.stdout:/proc/sys/fs/binfmt_misc 2024-03-28T14:08:13.085 INFO:teuthology.orchestra.run.smithi104.stdout:/dev/hugepages 2024-03-28T14:08:13.085 INFO:teuthology.orchestra.run.smithi104.stdout:/dev/mqueue 2024-03-28T14:08:13.085 INFO:teuthology.orchestra.run.smithi104.stdout:/sys/kernel/debug 2024-03-28T14:08:13.085 INFO:teuthology.orchestra.run.smithi104.stdout:/sys/kernel/tracing 2024-03-28T14:08:13.085 INFO:teuthology.orchestra.run.smithi104.stdout:/proc/fs/nfsd 2024-03-28T14:08:13.085 INFO:teuthology.orchestra.run.smithi104.stdout:/sys/fs/fuse/connections 2024-03-28T14:08:13.085 INFO:teuthology.orchestra.run.smithi104.stdout:/sys/kernel/config 2024-03-28T14:08:13.085 INFO:teuthology.orchestra.run.smithi104.stdout:/run/credentials/systemd-sysusers.service 2024-03-28T14:08:13.085 INFO:teuthology.orchestra.run.smithi104.stdout:/run/rpc_pipefs 2024-03-28T14:08:13.085 INFO:teuthology.orchestra.run.smithi104.stdout:/run/user/1000 2024-03-28T14:08:13.085 INFO:teuthology.orchestra.run.smithi104.stdout:/var/lib/ceph 2024-03-28T14:08:13.085 INFO:teuthology.orchestra.run.smithi104.stdout:/var/lib/ceph/osd/ceph-0 2024-03-28T14:08:13.086 INFO:teuthology.orchestra.run.smithi104.stdout:/var/lib/ceph/osd/ceph-1 2024-03-28T14:08:13.086 INFO:teuthology.orchestra.run.smithi104.stdout:/var/lib/ceph/osd/ceph-2 2024-03-28T14:08:13.086 INFO:teuthology.orchestra.run.smithi104.stdout:/var/lib/ceph/osd/ceph-3 2024-03-28T14:08:13.086 INFO:teuthology.orchestra.run.smithi104.stdout:/run/netns 2024-03-28T14:08:13.086 INFO:teuthology.orchestra.run.smithi104.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-28T14:08:13.086 INFO:teuthology.orchestra.run.smithi104.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-28T14:08:13.087 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T14:08:13.087 DEBUG:teuthology.orchestra.run.smithi104:> ls /sys/fs/fuse/connections 2024-03-28T14:08:13.135 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-03-28T14:08:13.135 DEBUG:teuthology.orchestra.run.smithi104:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2024-03-28T14:08:13.182 DEBUG:teuthology.orchestra.run.smithi104:> sudo modprobe fuse 2024-03-28T14:08:13.194 DEBUG:teuthology.orchestra.run.smithi104:> cat /proc/self/mounts | awk '{print $2}' 2024-03-28T14:08:13.240 INFO:teuthology.orchestra.run.smithi104.stdout:/sys 2024-03-28T14:08:13.241 INFO:teuthology.orchestra.run.smithi104.stdout:/proc 2024-03-28T14:08:13.241 INFO:teuthology.orchestra.run.smithi104.stdout:/dev 2024-03-28T14:08:13.241 INFO:teuthology.orchestra.run.smithi104.stdout:/dev/pts 2024-03-28T14:08:13.241 INFO:teuthology.orchestra.run.smithi104.stdout:/run 2024-03-28T14:08:13.241 INFO:teuthology.orchestra.run.smithi104.stdout:/ 2024-03-28T14:08:13.241 INFO:teuthology.orchestra.run.smithi104.stdout:/sys/kernel/security 2024-03-28T14:08:13.241 INFO:teuthology.orchestra.run.smithi104.stdout:/dev/shm 2024-03-28T14:08:13.241 INFO:teuthology.orchestra.run.smithi104.stdout:/run/lock 2024-03-28T14:08:13.241 INFO:teuthology.orchestra.run.smithi104.stdout:/sys/fs/cgroup 2024-03-28T14:08:13.241 INFO:teuthology.orchestra.run.smithi104.stdout:/sys/fs/pstore 2024-03-28T14:08:13.241 INFO:teuthology.orchestra.run.smithi104.stdout:/sys/fs/bpf 2024-03-28T14:08:13.241 INFO:teuthology.orchestra.run.smithi104.stdout:/proc/sys/fs/binfmt_misc 2024-03-28T14:08:13.242 INFO:teuthology.orchestra.run.smithi104.stdout:/dev/hugepages 2024-03-28T14:08:13.242 INFO:teuthology.orchestra.run.smithi104.stdout:/dev/mqueue 2024-03-28T14:08:13.242 INFO:teuthology.orchestra.run.smithi104.stdout:/sys/kernel/debug 2024-03-28T14:08:13.242 INFO:teuthology.orchestra.run.smithi104.stdout:/sys/kernel/tracing 2024-03-28T14:08:13.242 INFO:teuthology.orchestra.run.smithi104.stdout:/proc/fs/nfsd 2024-03-28T14:08:13.242 INFO:teuthology.orchestra.run.smithi104.stdout:/sys/fs/fuse/connections 2024-03-28T14:08:13.242 INFO:teuthology.orchestra.run.smithi104.stdout:/sys/kernel/config 2024-03-28T14:08:13.242 INFO:teuthology.orchestra.run.smithi104.stdout:/run/credentials/systemd-sysusers.service 2024-03-28T14:08:13.242 INFO:teuthology.orchestra.run.smithi104.stdout:/run/rpc_pipefs 2024-03-28T14:08:13.242 INFO:teuthology.orchestra.run.smithi104.stdout:/run/user/1000 2024-03-28T14:08:13.242 INFO:teuthology.orchestra.run.smithi104.stdout:/var/lib/ceph 2024-03-28T14:08:13.242 INFO:teuthology.orchestra.run.smithi104.stdout:/var/lib/ceph/osd/ceph-0 2024-03-28T14:08:13.242 INFO:teuthology.orchestra.run.smithi104.stdout:/var/lib/ceph/osd/ceph-1 2024-03-28T14:08:13.242 INFO:teuthology.orchestra.run.smithi104.stdout:/var/lib/ceph/osd/ceph-2 2024-03-28T14:08:13.243 INFO:teuthology.orchestra.run.smithi104.stdout:/var/lib/ceph/osd/ceph-3 2024-03-28T14:08:13.243 INFO:teuthology.orchestra.run.smithi104.stdout:/run/netns 2024-03-28T14:08:13.243 INFO:teuthology.orchestra.run.smithi104.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-28T14:08:13.243 INFO:teuthology.orchestra.run.smithi104.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-28T14:08:13.243 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T14:08:13.243 DEBUG:teuthology.orchestra.run.smithi104:> ls /sys/fs/fuse/connections 2024-03-28T14:08:13.257 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi104.stderr:2024-03-28T14:08:13.255+0000 7fb66cd18580 -1 init, newargv = 0x55e6465fc2e0 newargc=15 2024-03-28T14:08:13.257 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi104.stderr:ceph-fuse[34782]: starting ceph client 2024-03-28T14:08:13.263 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi104.stderr:ceph-fuse[34782]: starting fuse 2024-03-28T14:08:14.362 DEBUG:teuthology.orchestra.run.smithi104:> sudo modprobe fuse 2024-03-28T14:08:14.377 DEBUG:teuthology.orchestra.run.smithi104:> cat /proc/self/mounts | awk '{print $2}' 2024-03-28T14:08:14.425 INFO:teuthology.orchestra.run.smithi104.stdout:/sys 2024-03-28T14:08:14.425 INFO:teuthology.orchestra.run.smithi104.stdout:/proc 2024-03-28T14:08:14.425 INFO:teuthology.orchestra.run.smithi104.stdout:/dev 2024-03-28T14:08:14.425 INFO:teuthology.orchestra.run.smithi104.stdout:/dev/pts 2024-03-28T14:08:14.425 INFO:teuthology.orchestra.run.smithi104.stdout:/run 2024-03-28T14:08:14.425 INFO:teuthology.orchestra.run.smithi104.stdout:/ 2024-03-28T14:08:14.426 INFO:teuthology.orchestra.run.smithi104.stdout:/sys/kernel/security 2024-03-28T14:08:14.426 INFO:teuthology.orchestra.run.smithi104.stdout:/dev/shm 2024-03-28T14:08:14.426 INFO:teuthology.orchestra.run.smithi104.stdout:/run/lock 2024-03-28T14:08:14.426 INFO:teuthology.orchestra.run.smithi104.stdout:/sys/fs/cgroup 2024-03-28T14:08:14.426 INFO:teuthology.orchestra.run.smithi104.stdout:/sys/fs/pstore 2024-03-28T14:08:14.426 INFO:teuthology.orchestra.run.smithi104.stdout:/sys/fs/bpf 2024-03-28T14:08:14.426 INFO:teuthology.orchestra.run.smithi104.stdout:/proc/sys/fs/binfmt_misc 2024-03-28T14:08:14.426 INFO:teuthology.orchestra.run.smithi104.stdout:/dev/hugepages 2024-03-28T14:08:14.426 INFO:teuthology.orchestra.run.smithi104.stdout:/dev/mqueue 2024-03-28T14:08:14.426 INFO:teuthology.orchestra.run.smithi104.stdout:/sys/kernel/debug 2024-03-28T14:08:14.426 INFO:teuthology.orchestra.run.smithi104.stdout:/sys/kernel/tracing 2024-03-28T14:08:14.426 INFO:teuthology.orchestra.run.smithi104.stdout:/proc/fs/nfsd 2024-03-28T14:08:14.426 INFO:teuthology.orchestra.run.smithi104.stdout:/sys/fs/fuse/connections 2024-03-28T14:08:14.427 INFO:teuthology.orchestra.run.smithi104.stdout:/sys/kernel/config 2024-03-28T14:08:14.427 INFO:teuthology.orchestra.run.smithi104.stdout:/run/credentials/systemd-sysusers.service 2024-03-28T14:08:14.427 INFO:teuthology.orchestra.run.smithi104.stdout:/run/rpc_pipefs 2024-03-28T14:08:14.427 INFO:teuthology.orchestra.run.smithi104.stdout:/run/user/1000 2024-03-28T14:08:14.427 INFO:teuthology.orchestra.run.smithi104.stdout:/var/lib/ceph 2024-03-28T14:08:14.427 INFO:teuthology.orchestra.run.smithi104.stdout:/var/lib/ceph/osd/ceph-0 2024-03-28T14:08:14.427 INFO:teuthology.orchestra.run.smithi104.stdout:/var/lib/ceph/osd/ceph-1 2024-03-28T14:08:14.427 INFO:teuthology.orchestra.run.smithi104.stdout:/var/lib/ceph/osd/ceph-2 2024-03-28T14:08:14.427 INFO:teuthology.orchestra.run.smithi104.stdout:/var/lib/ceph/osd/ceph-3 2024-03-28T14:08:14.427 INFO:teuthology.orchestra.run.smithi104.stdout:/run/netns 2024-03-28T14:08:14.427 INFO:teuthology.orchestra.run.smithi104.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-28T14:08:14.427 INFO:teuthology.orchestra.run.smithi104.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-28T14:08:14.427 INFO:teuthology.orchestra.run.smithi104.stdout:/home/ubuntu/cephtest/mnt.0 2024-03-28T14:08:14.428 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T14:08:14.428 DEBUG:teuthology.orchestra.run.smithi104:> ls /sys/fs/fuse/connections 2024-03-28T14:08:14.471 INFO:teuthology.orchestra.run.smithi104.stdout:45 2024-03-28T14:08:14.471 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [45] 2024-03-28T14:08:14.471 DEBUG:teuthology.orchestra.run.smithi104:> sudo stdin-killer -- python3 -c ' 2024-03-28T14:08:14.471 DEBUG:teuthology.orchestra.run.smithi104:> import glob 2024-03-28T14:08:14.472 DEBUG:teuthology.orchestra.run.smithi104:> import re 2024-03-28T14:08:14.472 DEBUG:teuthology.orchestra.run.smithi104:> import os 2024-03-28T14:08:14.472 DEBUG:teuthology.orchestra.run.smithi104:> import subprocess 2024-03-28T14:08:14.472 DEBUG:teuthology.orchestra.run.smithi104:> 2024-03-28T14:08:14.472 DEBUG:teuthology.orchestra.run.smithi104:> def _find_admin_socket(client_name): 2024-03-28T14:08:14.472 DEBUG:teuthology.orchestra.run.smithi104:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2024-03-28T14:08:14.472 DEBUG:teuthology.orchestra.run.smithi104:> files = glob.glob(asok_path) 2024-03-28T14:08:14.472 DEBUG:teuthology.orchestra.run.smithi104:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2024-03-28T14:08:14.472 DEBUG:teuthology.orchestra.run.smithi104:> 2024-03-28T14:08:14.472 DEBUG:teuthology.orchestra.run.smithi104:> # Given a non-glob path, it better be there 2024-03-28T14:08:14.472 DEBUG:teuthology.orchestra.run.smithi104:> if "*" not in asok_path: 2024-03-28T14:08:14.472 DEBUG:teuthology.orchestra.run.smithi104:> assert(len(files) == 1) 2024-03-28T14:08:14.472 DEBUG:teuthology.orchestra.run.smithi104:> return files[0] 2024-03-28T14:08:14.472 DEBUG:teuthology.orchestra.run.smithi104:> 2024-03-28T14:08:14.472 DEBUG:teuthology.orchestra.run.smithi104:> for f in files: 2024-03-28T14:08:14.473 DEBUG:teuthology.orchestra.run.smithi104:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-03-28T14:08:14.473 DEBUG:teuthology.orchestra.run.smithi104:> if os.path.exists("/proc/{0}".format(pid)): 2024-03-28T14:08:14.473 DEBUG:teuthology.orchestra.run.smithi104:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-03-28T14:08:14.473 DEBUG:teuthology.orchestra.run.smithi104:> contents = proc_f.read() 2024-03-28T14:08:14.473 DEBUG:teuthology.orchestra.run.smithi104:> if mountpoint in contents: 2024-03-28T14:08:14.473 DEBUG:teuthology.orchestra.run.smithi104:> return f 2024-03-28T14:08:14.473 DEBUG:teuthology.orchestra.run.smithi104:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-03-28T14:08:14.473 DEBUG:teuthology.orchestra.run.smithi104:> 2024-03-28T14:08:14.473 DEBUG:teuthology.orchestra.run.smithi104:> print(_find_admin_socket("client.0")) 2024-03-28T14:08:14.473 DEBUG:teuthology.orchestra.run.smithi104:> ' 2024-03-28T14:08:14.596 INFO:teuthology.orchestra.run.smithi104.stdout:/var/run/ceph/ceph-client.0.34782.asok 2024-03-28T14:08:14.599 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:14 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-28T14:08:14.607 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.34782.asok 2024-03-28T14:08:14.607 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T14:08:14.607 DEBUG:teuthology.orchestra.run.smithi104:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.34782.asok status 2024-03-28T14:08:14.757 INFO:teuthology.orchestra.run.smithi104.stdout:{ 2024-03-28T14:08:14.757 INFO:teuthology.orchestra.run.smithi104.stdout: "metadata": { 2024-03-28T14:08:14.757 INFO:teuthology.orchestra.run.smithi104.stdout: "ceph_sha1": "e0e48ba6ee8c8b4c8f38d3b879d6eafbbb1efc15", 2024-03-28T14:08:14.757 INFO:teuthology.orchestra.run.smithi104.stdout: "ceph_version": "ceph version 19.0.0-2499-ge0e48ba6 (e0e48ba6ee8c8b4c8f38d3b879d6eafbbb1efc15) squid (dev)", 2024-03-28T14:08:14.757 INFO:teuthology.orchestra.run.smithi104.stdout: "entity_id": "0", 2024-03-28T14:08:14.757 INFO:teuthology.orchestra.run.smithi104.stdout: "hostname": "smithi104", 2024-03-28T14:08:14.757 INFO:teuthology.orchestra.run.smithi104.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2024-03-28T14:08:14.758 INFO:teuthology.orchestra.run.smithi104.stdout: "pid": "34782", 2024-03-28T14:08:14.758 INFO:teuthology.orchestra.run.smithi104.stdout: "root": "/" 2024-03-28T14:08:14.758 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-03-28T14:08:14.758 INFO:teuthology.orchestra.run.smithi104.stdout: "dentry_count": 0, 2024-03-28T14:08:14.758 INFO:teuthology.orchestra.run.smithi104.stdout: "dentry_pinned_count": 0, 2024-03-28T14:08:14.758 INFO:teuthology.orchestra.run.smithi104.stdout: "id": 4626, 2024-03-28T14:08:14.758 INFO:teuthology.orchestra.run.smithi104.stdout: "inst": { 2024-03-28T14:08:14.758 INFO:teuthology.orchestra.run.smithi104.stdout: "name": { 2024-03-28T14:08:14.758 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "client", 2024-03-28T14:08:14.758 INFO:teuthology.orchestra.run.smithi104.stdout: "num": 4626 2024-03-28T14:08:14.758 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-03-28T14:08:14.758 INFO:teuthology.orchestra.run.smithi104.stdout: "addr": { 2024-03-28T14:08:14.758 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "v1", 2024-03-28T14:08:14.759 INFO:teuthology.orchestra.run.smithi104.stdout: "addr": "192.168.0.1:0", 2024-03-28T14:08:14.759 INFO:teuthology.orchestra.run.smithi104.stdout: "nonce": 1134458285 2024-03-28T14:08:14.759 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-03-28T14:08:14.759 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-03-28T14:08:14.759 INFO:teuthology.orchestra.run.smithi104.stdout: "addr": { 2024-03-28T14:08:14.759 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "v1", 2024-03-28T14:08:14.759 INFO:teuthology.orchestra.run.smithi104.stdout: "addr": "192.168.0.1:0", 2024-03-28T14:08:14.759 INFO:teuthology.orchestra.run.smithi104.stdout: "nonce": 1134458285 2024-03-28T14:08:14.759 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-03-28T14:08:14.759 INFO:teuthology.orchestra.run.smithi104.stdout: "inst_str": "client.4626 192.168.0.1:0/1134458285", 2024-03-28T14:08:14.759 INFO:teuthology.orchestra.run.smithi104.stdout: "addr_str": "192.168.0.1:0/1134458285", 2024-03-28T14:08:14.759 INFO:teuthology.orchestra.run.smithi104.stdout: "inode_count": 1, 2024-03-28T14:08:14.759 INFO:teuthology.orchestra.run.smithi104.stdout: "mds_epoch": 7, 2024-03-28T14:08:14.760 INFO:teuthology.orchestra.run.smithi104.stdout: "osd_epoch": 21, 2024-03-28T14:08:14.760 INFO:teuthology.orchestra.run.smithi104.stdout: "osd_epoch_barrier": 0, 2024-03-28T14:08:14.760 INFO:teuthology.orchestra.run.smithi104.stdout: "blocklisted": false, 2024-03-28T14:08:14.760 INFO:teuthology.orchestra.run.smithi104.stdout: "fs_name": "cephfs" 2024-03-28T14:08:14.760 INFO:teuthology.orchestra.run.smithi104.stdout:} 2024-03-28T14:08:14.769 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T14:08:14.770 DEBUG:teuthology.orchestra.run.smithi104:> stat --file-system '--printf=%T 2024-03-28T14:08:14.770 DEBUG:teuthology.orchestra.run.smithi104:> ' -- /home/ubuntu/cephtest/mnt.0 2024-03-28T14:08:14.775 INFO:teuthology.orchestra.run.smithi104.stdout:fuseblk 2024-03-28T14:08:14.776 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2024-03-28T14:08:14.776 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T14:08:14.776 DEBUG:teuthology.orchestra.run.smithi104:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-03-28T14:08:14.831 INFO:teuthology.run_tasks:Running task cephfs_test_runner... 2024-03-28T14:08:14.850 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-28T14:08:14.995 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:14.995+0000 7f2aa6132640 1 Processor -- start 2024-03-28T14:08:14.996 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:14.995+0000 7f2aa6132640 1 -- start start 2024-03-28T14:08:14.998 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:14.995+0000 7f2aa6132640 1 --2- >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f2aa0072bf0 0x7f2aa00b1e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:14.998 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:14.995+0000 7f2aa6132640 1 --2- >> v2:172.21.15.171:3300/0 conn(0x7f2aa0145ce0 0x7f2aa0146150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:14.999 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:14.995+0000 7f2aa6132640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2aa0072020 con 0x7f2aa0071be0 2024-03-28T14:08:14.999 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:14.995+0000 7f2aa6132640 1 -- --> v2:172.21.15.171:3300/0 -- mon_getmap magic: 0 -- 0x7f2aa0073250 con 0x7f2aa0145ce0 2024-03-28T14:08:14.999 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:14.995+0000 7f2aa6132640 1 -- --> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] -- mon_getmap magic: 0 -- 0x7f2aa0072fd0 con 0x7f2aa0072bf0 2024-03-28T14:08:14.999 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:14.995+0000 7f2a97fff640 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f2aa0071be0 0x7f2aa00b12f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:41720/0 (socket says 172.21.15.104:41720) 2024-03-28T14:08:14.999 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:14.995+0000 7f2a97fff640 1 -- 172.21.15.104:0/2354130198 learned_addr learned my addr 172.21.15.104:0/2354130198 (peer_addr_for_me v1:172.21.15.104:0/0) 2024-03-28T14:08:14.999 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:14.995+0000 7f2a9f7fe640 1 --2- 172.21.15.104:0/2354130198 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f2aa0072bf0 0x7f2aa00b1e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:14.999 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:14.995+0000 7f2a9ffff640 1 --2- 172.21.15.104:0/2354130198 >> v2:172.21.15.171:3300/0 conn(0x7f2aa0145ce0 0x7f2aa0146150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:14.999 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:14.995+0000 7f2a9effd640 1 -- 172.21.15.104:0/2354130198 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2752504376 0 0) 0x7f2aa0072020 con 0x7f2aa0071be0 2024-03-28T14:08:14.999 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:14.995+0000 7f2a9effd640 1 -- 172.21.15.104:0/2354130198 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2a8c003530 con 0x7f2aa0071be0 2024-03-28T14:08:14.999 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:14.995+0000 7f2a9effd640 1 -- 172.21.15.104:0/2354130198 <== mon.0 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2673338824 0 0) 0x7f2a8c003530 con 0x7f2aa0071be0 2024-03-28T14:08:14.999 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:14.995+0000 7f2a9effd640 1 -- 172.21.15.104:0/2354130198 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f2aa0072bf0 msgr2=0x7f2aa00b1e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T14:08:14.999 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:14.995+0000 7f2a9effd640 1 --2- 172.21.15.104:0/2354130198 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f2aa0072bf0 0x7f2aa00b1e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:14.999 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:14.995+0000 7f2a9effd640 1 -- 172.21.15.104:0/2354130198 >> v2:172.21.15.171:3300/0 conn(0x7f2aa0145ce0 msgr2=0x7f2aa0146150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T14:08:15.000 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:14.995+0000 7f2a9effd640 1 --2- 172.21.15.104:0/2354130198 >> v2:172.21.15.171:3300/0 conn(0x7f2aa0145ce0 0x7f2aa0146150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:15.000 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:14.995+0000 7f2a9effd640 1 -- 172.21.15.104:0/2354130198 --> v1:172.21.15.104:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2aa00734a0 con 0x7f2aa0071be0 2024-03-28T14:08:15.000 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:14.995+0000 7f2a9effd640 1 -- 172.21.15.104:0/2354130198 <== mon.0 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3745580849 0 0) 0x7f2a88001680 con 0x7f2aa0071be0 2024-03-28T14:08:15.000 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:14.995+0000 7f2a9effd640 1 -- 172.21.15.104:0/2354130198 >> v1:172.21.15.104:6789/0 conn(0x7f2aa0071be0 legacy=0x7f2aa00b12f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:15.000 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:14.995+0000 7f2a9effd640 1 --2- 172.21.15.104:0/2354130198 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2a8c0039c0 0x7f2a8c003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:15.000 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:14.995+0000 7f2a9effd640 1 --2- 172.21.15.104:0/2354130198 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f2aa0072bf0 0x7f2a8c004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:15.000 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:14.995+0000 7f2a9effd640 1 --2- 172.21.15.104:0/2354130198 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f2a8c004d50 0x7f2a8c0051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:15.000 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:14.995+0000 7f2a9effd640 1 -- 172.21.15.104:0/2354130198 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f2aa0073250 con 0x7f2a8c0039c0 2024-03-28T14:08:15.000 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:14.995+0000 7f2a9effd640 1 -- 172.21.15.104:0/2354130198 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f2aa0072fd0 con 0x7f2a8c004d50 2024-03-28T14:08:15.000 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:14.995+0000 7f2a9effd640 1 -- 172.21.15.104:0/2354130198 --> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] -- mon_getmap magic: 0 -- 0x7f2a8c00ba50 con 0x7f2aa0072bf0 2024-03-28T14:08:15.000 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:14.995+0000 7f2a9f7fe640 1 --2- 172.21.15.104:0/2354130198 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2a8c0039c0 0x7f2a8c003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:15.001 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:14.995+0000 7f2a9ffff640 1 --2- 172.21.15.104:0/2354130198 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f2a8c004d50 0x7f2a8c0051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:15.001 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:14.995+0000 7f2a97fff640 1 --2- 172.21.15.104:0/2354130198 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f2aa0072bf0 0x7f2a8c004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:15.001 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:14.995+0000 7f2a9f7fe640 1 -- 172.21.15.104:0/2354130198 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f2aa0072bf0 msgr2=0x7f2a8c004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T14:08:15.001 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:14.995+0000 7f2a9f7fe640 1 --2- 172.21.15.104:0/2354130198 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f2aa0072bf0 0x7f2a8c004810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:15.001 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:14.995+0000 7f2a9f7fe640 1 -- 172.21.15.104:0/2354130198 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f2a8c004d50 msgr2=0x7f2a8c0051d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T14:08:15.001 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:14.995+0000 7f2a9f7fe640 1 --2- 172.21.15.104:0/2354130198 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f2a8c004d50 0x7f2a8c0051d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:15.001 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:14.995+0000 7f2a9f7fe640 1 -- 172.21.15.104:0/2354130198 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2a8c003530 con 0x7f2a8c0039c0 2024-03-28T14:08:15.001 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:14.995+0000 7f2a9ffff640 1 --2- 172.21.15.104:0/2354130198 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f2a8c004d50 0x7f2a8c0051d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T14:08:15.004 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:14.999+0000 7f2a9f7fe640 1 --2- 172.21.15.104:0/2354130198 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2a8c0039c0 0x7f2a8c003de0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f2a90009ae0 tx=0x7f2a9001ca70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=495cd5e0b649063e server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:08:15.005 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:14.999+0000 7f2a9effd640 1 -- 172.21.15.104:0/2354130198 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a900096b0 con 0x7f2a8c0039c0 2024-03-28T14:08:15.005 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:14.999+0000 7f2a9effd640 1 -- 172.21.15.104:0/2354130198 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2a900a6450 con 0x7f2a8c0039c0 2024-03-28T14:08:15.005 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:14.999+0000 7f2a9effd640 1 -- 172.21.15.104:0/2354130198 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a900a6840 con 0x7f2a8c0039c0 2024-03-28T14:08:15.005 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:14.999+0000 7f2aa6132640 1 -- 172.21.15.104:0/2354130198 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2a8c0039c0 msgr2=0x7f2a8c003de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:15.005 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:14.999+0000 7f2aa6132640 1 --2- 172.21.15.104:0/2354130198 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2a8c0039c0 0x7f2a8c003de0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f2a90009ae0 tx=0x7f2a9001ca70 comp rx=0 tx=0).stop 2024-03-28T14:08:15.005 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:14.999+0000 7f2a9ffff640 1 -- 172.21.15.104:0/2354130198 reap_dead start 2024-03-28T14:08:15.005 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:14.999+0000 7f2aa6132640 1 -- 172.21.15.104:0/2354130198 shutdown_connections 2024-03-28T14:08:15.005 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:14.999+0000 7f2aa6132640 1 -- 172.21.15.104:0/2354130198 >> 172.21.15.104:0/2354130198 conn(0x7f2aa00b7b10 msgr2=0x7f2aa00b7f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T14:08:15.005 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:14.999+0000 7f2aa6132640 1 -- 172.21.15.104:0/2354130198 shutdown_connections 2024-03-28T14:08:15.005 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:14.999+0000 7f2aa6132640 1 -- 172.21.15.104:0/2354130198 wait complete. 2024-03-28T14:08:15.005 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:14.999+0000 7f2aa6132640 1 Processor -- start 2024-03-28T14:08:15.005 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:14.999+0000 7f2aa6132640 1 -- start start 2024-03-28T14:08:15.005 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:14.999+0000 7f2aa6132640 1 --2- >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f2a8c0039c0 0x7f2aa014e4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:15.006 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:14.999+0000 7f2aa6132640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f2aa0146750 0x7f2aa013f300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:15.006 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:14.999+0000 7f2aa6132640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2aa013f840 0x7f2aa00b4250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:15.006 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:14.999+0000 7f2aa6132640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f2aa0150820 con 0x7f2aa013f840 2024-03-28T14:08:15.006 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:14.999+0000 7f2aa6132640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f2aa01506a0 con 0x7f2aa0146750 2024-03-28T14:08:15.006 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:14.999+0000 7f2aa6132640 1 -- --> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] -- mon_getmap magic: 0 -- 0x7f2aa01509a0 con 0x7f2a8c0039c0 2024-03-28T14:08:15.006 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:14.999+0000 7f2a9ffff640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2aa013f840 0x7f2aa00b4250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:15.006 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:14.999+0000 7f2a9ffff640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2aa013f840 0x7f2aa00b4250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:35026/0 (socket says 172.21.15.104:35026) 2024-03-28T14:08:15.006 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:14.999+0000 7f2a9ffff640 1 -- 172.21.15.104:0/3182638196 learned_addr learned my addr 172.21.15.104:0/3182638196 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-03-28T14:08:15.006 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:14.999+0000 7f2a9f7fe640 1 --2- 172.21.15.104:0/3182638196 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f2a8c0039c0 0x7f2aa014e4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:15.006 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:14.999+0000 7f2a97fff640 1 --2- 172.21.15.104:0/3182638196 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f2aa0146750 0x7f2aa013f300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:15.006 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:14.999+0000 7f2a9ffff640 1 -- 172.21.15.104:0/3182638196 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f2a8c0039c0 msgr2=0x7f2aa014e4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:15.007 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:14.999+0000 7f2a9ffff640 1 --2- 172.21.15.104:0/3182638196 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f2a8c0039c0 0x7f2aa014e4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:15.007 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:14.999+0000 7f2a9ffff640 1 -- 172.21.15.104:0/3182638196 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f2aa0146750 msgr2=0x7f2aa013f300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:15.007 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:14.999+0000 7f2a9ffff640 1 --2- 172.21.15.104:0/3182638196 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f2aa0146750 0x7f2aa013f300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:15.007 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:14.999+0000 7f2a9ffff640 1 -- 172.21.15.104:0/3182638196 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2aa00b5170 con 0x7f2aa013f840 2024-03-28T14:08:15.007 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:14.999+0000 7f2a9ffff640 1 --2- 172.21.15.104:0/3182638196 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2aa013f840 0x7f2aa00b4250 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f2aa00b7450 tx=0x7f2a9800f520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:08:15.007 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:14.999+0000 7f2a9cff9640 1 -- 172.21.15.104:0/3182638196 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a98016420 con 0x7f2aa013f840 2024-03-28T14:08:15.007 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:15.003+0000 7f2a9cff9640 1 -- 172.21.15.104:0/3182638196 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2a9801b070 con 0x7f2aa013f840 2024-03-28T14:08:15.007 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:15.003+0000 7f2a9cff9640 1 -- 172.21.15.104:0/3182638196 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a980167e0 con 0x7f2aa013f840 2024-03-28T14:08:15.007 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:15.003+0000 7f2aa6132640 1 -- 172.21.15.104:0/3182638196 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2aa0146c30 con 0x7f2aa013f840 2024-03-28T14:08:15.007 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:15.003+0000 7f2aa6132640 1 -- 172.21.15.104:0/3182638196 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2aa00b3600 con 0x7f2aa013f840 2024-03-28T14:08:15.007 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:15.003+0000 7f2aa6132640 1 -- 172.21.15.104:0/3182638196 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2aa00687c0 con 0x7f2aa013f840 2024-03-28T14:08:15.008 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:15.003+0000 7f2a9cff9640 1 -- 172.21.15.104:0/3182638196 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) ==== 87803+0+0 (secure 0 0 0) 0x7f2a98016980 con 0x7f2aa013f840 2024-03-28T14:08:15.008 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:15.003+0000 7f2a9cff9640 1 --2- 172.21.15.104:0/3182638196 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f2a80067fc0 0x7f2a8006a480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:15.008 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:15.003+0000 7f2a9cff9640 1 -- 172.21.15.104:0/3182638196 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f2a98087dc0 con 0x7f2aa013f840 2024-03-28T14:08:15.009 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:15.003+0000 7f2a9f7fe640 1 --2- 172.21.15.104:0/3182638196 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f2a80067fc0 0x7f2a8006a480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:15.009 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:15.007+0000 7f2a9f7fe640 1 --2- 172.21.15.104:0/3182638196 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f2a80067fc0 0x7f2a8006a480 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f2a900ad000 tx=0x7f2a900046d0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:08:15.009 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:15.007+0000 7f2a9cff9640 1 -- 172.21.15.104:0/3182638196 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179117 (secure 0 0 0) 0x7f2a980586e0 con 0x7f2aa013f840 2024-03-28T14:08:15.171 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:15.167+0000 7f2aa6132640 1 -- 172.21.15.104:0/3182638196 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f2aa00b8eb0 con 0x7f2aa013f840 2024-03-28T14:08:15.173 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:15.171+0000 7f2a9cff9640 1 -- 172.21.15.104:0/3182638196 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) ==== 74+0+16629 (secure 0 0 0) 0x7f2a98057fe0 con 0x7f2aa013f840 2024-03-28T14:08:15.173 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-03-28T14:08:15.174 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":21,"fsid":"428a90f2-3351-435f-8fcf-f7e66c78a13e","created":"2024-03-28T14:07:34.862458+0000","modified":"2024-03-28T14:08:00.671587+0000","last_up_change":"2024-03-28T14:07:46.028122+0000","last_in_change":"2024-03-28T14:07:37.157486+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-28T14:07:47.622988+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-28T14:07:57.147450+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-28T14:07:57.990975+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"1cb70540-4e42-402f-a446-aaec4ca83007","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6800","nonce":3745690918},{"type":"v1","addr":"172.21.15.104:6801","nonce":3745690918}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":3745690918},{"type":"v1","addr":"172.21.15.104:6803","nonce":3745690918}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":3745690918},{"type":"v1","addr":"172.21.15.104:6807","nonce":3745690918}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":3745690918},{"type":"v1","addr":"172.21.15.104:6805","nonce":3745690918}]},"public_addr":"172.21.15.104:6801/3745690918","cluster_addr":"172.21.15.104:6803/3745690918","heartbeat_back_addr":"172.21.15.104:6807/3745690918","heartbeat_front_addr":"172.21.15.104:6805/3745690918","state":["exists","up"]},{"osd":1,"uuid":"3ab7afdd-b980-418c-81f2-004e16de3dca","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":19438674},{"type":"v1","addr":"172.21.15.104:6810","nonce":19438674}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":19438674},{"type":"v1","addr":"172.21.15.104:6814","nonce":19438674}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":19438674},{"type":"v1","addr":"172.21.15.104:6823","nonce":19438674}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":19438674},{"type":"v1","addr":"172.21.15.104:6819","nonce":19438674}]},"public_addr":"172.21.15.104:6810/19438674","cluster_addr":"172.21.15.104:6814/19438674","heartbeat_back_addr":"172.21.15.104:6823/19438674","heartbeat_front_addr":"172.21.15.104:6819/19438674","state":["exists","up"]},{"osd":2,"uuid":"3e2651c8-ecae-4a9e-a053-27240a79eac2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":2352012626},{"type":"v1","addr":"172.21.15.104:6825","nonce":2352012626}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6826","nonce":2352012626},{"type":"v1","addr":"172.21.15.104:6827","nonce":2352012626}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":2352012626},{"type":"v1","addr":"172.21.15.104:6831","nonce":2352012626}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6828","nonce":2352012626},{"type":"v1","addr":"172.21.15.104:6829","nonce":2352012626}]},"public_addr":"172.21.15.104:6825/2352012626","cluster_addr":"172.21.15.104:6827/2352012626","heartbeat_back_addr":"172.21.15.104:6831/2352012626","heartbeat_front_addr":"172.21.15.104:6829/2352012626","state":["exists","up"]},{"osd":3,"uuid":"925c7e0a-38ef-47cd-b288-8bec8456fb17","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6809","nonce":3568347381},{"type":"v1","addr":"172.21.15.104:6811","nonce":3568347381}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6813","nonce":3568347381},{"type":"v1","addr":"172.21.15.104:6815","nonce":3568347381}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":3568347381},{"type":"v1","addr":"172.21.15.104:6822","nonce":3568347381}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6817","nonce":3568347381},{"type":"v1","addr":"172.21.15.104:6818","nonce":3568347381}]},"public_addr":"172.21.15.104:6811/3568347381","cluster_addr":"172.21.15.104:6815/3568347381","heartbeat_back_addr":"172.21.15.104:6822/3568347381","heartbeat_front_addr":"172.21.15.104:6818/3568347381","state":["exists","up"]},{"osd":4,"uuid":"4a8fac86-e595-4a0e-96e3-57edb3cc3f57","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6800","nonce":2107001077},{"type":"v1","addr":"172.21.15.171:6801","nonce":2107001077}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6802","nonce":2107001077},{"type":"v1","addr":"172.21.15.171:6803","nonce":2107001077}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6806","nonce":2107001077},{"type":"v1","addr":"172.21.15.171:6807","nonce":2107001077}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6804","nonce":2107001077},{"type":"v1","addr":"172.21.15.171:6805","nonce":2107001077}]},"public_addr":"172.21.15.171:6801/2107001077","cluster_addr":"172.21.15.171:6803/2107001077","heartbeat_back_addr":"172.21.15.171:6807/2107001077","heartbeat_front_addr":"172.21.15.171:6805/2107001077","state":["exists","up"]},{"osd":5,"uuid":"e48d50b0-beae-40ae-b15a-9a84dbfe88ab","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6808","nonce":2934826925},{"type":"v1","addr":"172.21.15.171:6810","nonce":2934826925}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6812","nonce":2934826925},{"type":"v1","addr":"172.21.15.171:6814","nonce":2934826925}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6819","nonce":2934826925},{"type":"v1","addr":"172.21.15.171:6821","nonce":2934826925}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6816","nonce":2934826925},{"type":"v1","addr":"172.21.15.171:6817","nonce":2934826925}]},"public_addr":"172.21.15.171:6810/2934826925","cluster_addr":"172.21.15.171:6814/2934826925","heartbeat_back_addr":"172.21.15.171:6821/2934826925","heartbeat_front_addr":"172.21.15.171:6817/2934826925","state":["exists","up"]},{"osd":6,"uuid":"5e7c3292-9936-4f35-b267-aed9d4e49323","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6824","nonce":436742919},{"type":"v1","addr":"172.21.15.171:6825","nonce":436742919}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6826","nonce":436742919},{"type":"v1","addr":"172.21.15.171:6827","nonce":436742919}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6830","nonce":436742919},{"type":"v1","addr":"172.21.15.171:6831","nonce":436742919}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6828","nonce":436742919},{"type":"v1","addr":"172.21.15.171:6829","nonce":436742919}]},"public_addr":"172.21.15.171:6825/436742919","cluster_addr":"172.21.15.171:6827/436742919","heartbeat_back_addr":"172.21.15.171:6831/436742919","heartbeat_front_addr":"172.21.15.171:6829/436742919","state":["exists","up"]},{"osd":7,"uuid":"339b8414-d7f0-4fb0-a446-efaf635cb7ff","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6809","nonce":2055631175},{"type":"v1","addr":"172.21.15.171:6811","nonce":2055631175}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6813","nonce":2055631175},{"type":"v1","addr":"172.21.15.171:6815","nonce":2055631175}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6822","nonce":2055631175},{"type":"v1","addr":"172.21.15.171:6823","nonce":2055631175}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6818","nonce":2055631175},{"type":"v1","addr":"172.21.15.171:6820","nonce":2055631175}]},"public_addr":"172.21.15.171:6811/2055631175","cluster_addr":"172.21.15.171:6815/2055631175","heartbeat_back_addr":"172.21.15.171:6823/2055631175","heartbeat_front_addr":"172.21.15.171:6820/2055631175","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T14:07:42.668881+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T14:07:42.708397+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T14:07:42.634191+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T14:07:42.733662+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T14:07:42.651541+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T14:07:42.783181+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T14:07:42.766310+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T14:07:42.670797+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-28T14:08:15.177 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:15.175+0000 7f2aa6132640 1 -- 172.21.15.104:0/3182638196 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f2a80067fc0 msgr2=0x7f2a8006a480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:15.177 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:15.175+0000 7f2aa6132640 1 --2- 172.21.15.104:0/3182638196 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f2a80067fc0 0x7f2a8006a480 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f2a900ad000 tx=0x7f2a900046d0 comp rx=0 tx=0).stop 2024-03-28T14:08:15.177 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:15.175+0000 7f2aa6132640 1 -- 172.21.15.104:0/3182638196 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2aa013f840 msgr2=0x7f2aa00b4250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:15.178 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:15.175+0000 7f2aa6132640 1 --2- 172.21.15.104:0/3182638196 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2aa013f840 0x7f2aa00b4250 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f2aa00b7450 tx=0x7f2a9800f520 comp rx=0 tx=0).stop 2024-03-28T14:08:15.178 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:15.175+0000 7f2aa6132640 1 -- 172.21.15.104:0/3182638196 shutdown_connections 2024-03-28T14:08:15.178 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:15.175+0000 7f2aa6132640 1 --2- 172.21.15.104:0/3182638196 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f2a80067fc0 0x7f2a8006a480 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:15.178 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:15.175+0000 7f2aa6132640 1 --2- 172.21.15.104:0/3182638196 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2aa013f840 0x7f2aa00b4250 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:15.178 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:15.175+0000 7f2aa6132640 1 --2- 172.21.15.104:0/3182638196 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f2aa0146750 0x7f2aa013f300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:15.179 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:15.175+0000 7f2aa6132640 1 --2- 172.21.15.104:0/3182638196 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f2a8c0039c0 0x7f2aa014e4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:15.179 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:15.175+0000 7f2aa6132640 1 -- 172.21.15.104:0/3182638196 >> 172.21.15.104:0/3182638196 conn(0x7f2aa00b7b10 msgr2=0x7f2aa00b96b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T14:08:15.179 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:15.175+0000 7f2aa6132640 1 -- 172.21.15.104:0/3182638196 shutdown_connections 2024-03-28T14:08:15.179 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:15.175+0000 7f2aa6132640 1 -- 172.21.15.104:0/3182638196 wait complete. 2024-03-28T14:08:15.193 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-28T14:07:47.622988+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-28T14:07:57.147450+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-28T14:07:57.990975+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-28T14:08:15.194 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-28T14:08:15.312 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:15.311+0000 7f5156c9f640 1 Processor -- start 2024-03-28T14:08:15.312 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:15.311+0000 7f5156c9f640 1 -- start start 2024-03-28T14:08:15.313 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:15.311+0000 7f5156c9f640 1 --2- >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f515006b830 0x7f5150072ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:15.313 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:15.311+0000 7f5156c9f640 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f51500b54c0 0x7f515014ffe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:15.313 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:15.311+0000 7f5156c9f640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f515006bc10 con 0x7f51500b4f70 2024-03-28T14:08:15.313 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:15.311+0000 7f5156c9f640 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 -- 0x7f51500b6060 con 0x7f51500b54c0 2024-03-28T14:08:15.313 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:15.311+0000 7f5156c9f640 1 -- --> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] -- mon_getmap magic: 0 -- 0x7f51500b5a10 con 0x7f515006b830 2024-03-28T14:08:15.314 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:15.311+0000 7f515523c640 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f51500b54c0 0x7f515014ffe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:15.314 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:15.311+0000 7f514ffff640 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f51500b4f70 0x7f515013f650 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:41732/0 (socket says 172.21.15.104:41732) 2024-03-28T14:08:15.314 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:15.311+0000 7f514ffff640 1 -- 172.21.15.104:0/3925941454 learned_addr learned my addr 172.21.15.104:0/3925941454 (peer_addr_for_me v1:172.21.15.104:0/0) 2024-03-28T14:08:15.314 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:15.311+0000 7f5154a3b640 1 --2- 172.21.15.104:0/3925941454 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f515006b830 0x7f5150072ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:15.314 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:15.311+0000 7f515523c640 1 -- 172.21.15.104:0/3925941454 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f515006b830 msgr2=0x7f5150072ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T14:08:15.314 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:15.311+0000 7f515523c640 1 --2- 172.21.15.104:0/3925941454 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f515006b830 0x7f5150072ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:15.314 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:15.311+0000 7f515523c640 1 -- 172.21.15.104:0/3925941454 >> v1:172.21.15.104:6789/0 conn(0x7f51500b4f70 legacy=0x7f515013f650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:15.314 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:15.311+0000 7f515523c640 1 -- 172.21.15.104:0/3925941454 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f51500b5cf0 con 0x7f51500b54c0 2024-03-28T14:08:15.314 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:15.311+0000 7f5154a3b640 1 --2- 172.21.15.104:0/3925941454 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f515006b830 0x7f5150072ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T14:08:15.315 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:15.311+0000 7f515523c640 1 --2- 172.21.15.104:0/3925941454 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f51500b54c0 0x7f515014ffe0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f5148009d90 tx=0x7f514801c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f02031fde2e984c server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:08:15.315 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:15.315+0000 7f514f7fe640 1 -- 172.21.15.104:0/3925941454 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51480b2070 con 0x7f51500b54c0 2024-03-28T14:08:15.315 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:15.315+0000 7f514f7fe640 1 -- 172.21.15.104:0/3925941454 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f51480a59c0 con 0x7f51500b54c0 2024-03-28T14:08:15.315 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:15.315+0000 7f514f7fe640 1 -- 172.21.15.104:0/3925941454 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51480a5d90 con 0x7f51500b54c0 2024-03-28T14:08:15.316 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:15.315+0000 7f5156c9f640 1 -- 172.21.15.104:0/3925941454 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f51500b54c0 msgr2=0x7f515014ffe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:15.316 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:15.315+0000 7f5156c9f640 1 --2- 172.21.15.104:0/3925941454 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f51500b54c0 0x7f515014ffe0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f5148009d90 tx=0x7f514801c8e0 comp rx=0 tx=0).stop 2024-03-28T14:08:15.316 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:15.315+0000 7f5156c9f640 1 -- 172.21.15.104:0/3925941454 shutdown_connections 2024-03-28T14:08:15.316 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:15.315+0000 7f5156c9f640 1 --2- 172.21.15.104:0/3925941454 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f51500b54c0 0x7f515014ffe0 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:15.316 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:15.315+0000 7f5156c9f640 1 --2- 172.21.15.104:0/3925941454 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f515006b830 0x7f5150072ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:15.316 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:15.315+0000 7f5156c9f640 1 -- 172.21.15.104:0/3925941454 >> 172.21.15.104:0/3925941454 conn(0x7f51500ba400 msgr2=0x7f51500ba810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T14:08:15.317 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:15.315+0000 7f5156c9f640 1 -- 172.21.15.104:0/3925941454 shutdown_connections 2024-03-28T14:08:15.317 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:15.315+0000 7f5156c9f640 1 -- 172.21.15.104:0/3925941454 wait complete. 2024-03-28T14:08:15.318 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:15.315+0000 7f5156c9f640 1 Processor -- start 2024-03-28T14:08:15.319 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:15.315+0000 7f5156c9f640 1 -- start start 2024-03-28T14:08:15.319 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:15.315+0000 7f5156c9f640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f515006b830 0x7f515013d2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:15.319 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:15.315+0000 7f5156c9f640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f51500b4f70 0x7f5150145d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:15.319 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:15.315+0000 7f5156c9f640 1 --2- >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f51500b54c0 0x7f51501462b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:15.319 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:15.315+0000 7f5156c9f640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f5150073560 con 0x7f515006b830 2024-03-28T14:08:15.319 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:15.315+0000 7f5156c9f640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f51500733e0 con 0x7f51500b4f70 2024-03-28T14:08:15.319 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:15.315+0000 7f5156c9f640 1 -- --> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] -- mon_getmap magic: 0 -- 0x7f5150152d40 con 0x7f51500b54c0 2024-03-28T14:08:15.319 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:15.315+0000 7f5154a3b640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f515006b830 0x7f515013d2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:15.319 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:15.315+0000 7f5154a3b640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f515006b830 0x7f515013d2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:35050/0 (socket says 172.21.15.104:35050) 2024-03-28T14:08:15.319 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:15.315+0000 7f5154a3b640 1 -- 172.21.15.104:0/528465514 learned_addr learned my addr 172.21.15.104:0/528465514 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-03-28T14:08:15.319 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:15.315+0000 7f5154a3b640 1 -- 172.21.15.104:0/528465514 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f51500b54c0 msgr2=0x7f51501462b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:15.319 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:15.315+0000 7f5154a3b640 1 --2- 172.21.15.104:0/528465514 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f51500b54c0 0x7f51501462b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:15.319 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:15.315+0000 7f5154a3b640 1 -- 172.21.15.104:0/528465514 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f51500b4f70 msgr2=0x7f5150145d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:15.320 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:15.315+0000 7f5154a3b640 1 --2- 172.21.15.104:0/528465514 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f51500b4f70 0x7f5150145d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:15.320 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:15.315+0000 7f5154a3b640 1 -- 172.21.15.104:0/528465514 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f51501e1ad0 con 0x7f515006b830 2024-03-28T14:08:15.320 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:15.315+0000 7f5154a3b640 1 --2- 172.21.15.104:0/528465514 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f515006b830 0x7f515013d2c0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f514400d800 tx=0x7f514400dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:08:15.320 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:15.315+0000 7f514d7fa640 1 -- 172.21.15.104:0/528465514 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5144018020 con 0x7f515006b830 2024-03-28T14:08:15.320 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:15.315+0000 7f514d7fa640 1 -- 172.21.15.104:0/528465514 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f51440096c0 con 0x7f515006b830 2024-03-28T14:08:15.320 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:15.315+0000 7f5156c9f640 1 -- 172.21.15.104:0/528465514 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f51501e1db0 con 0x7f515006b830 2024-03-28T14:08:15.320 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:15.315+0000 7f514d7fa640 1 -- 172.21.15.104:0/528465514 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5144009ab0 con 0x7f515006b830 2024-03-28T14:08:15.320 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:15.315+0000 7f5156c9f640 1 -- 172.21.15.104:0/528465514 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f51501435b0 con 0x7f515006b830 2024-03-28T14:08:15.321 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:15.319+0000 7f5156c9f640 1 -- 172.21.15.104:0/528465514 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f51500687c0 con 0x7f515006b830 2024-03-28T14:08:15.321 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:15.319+0000 7f514d7fa640 1 -- 172.21.15.104:0/528465514 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) ==== 87803+0+0 (secure 0 0 0) 0x7f514401b020 con 0x7f515006b830 2024-03-28T14:08:15.321 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:15.319+0000 7f514d7fa640 1 --2- 172.21.15.104:0/528465514 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f5134067fc0 0x7f513406a480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:15.322 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:15.319+0000 7f514d7fa640 1 -- 172.21.15.104:0/528465514 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f5144086a40 con 0x7f515006b830 2024-03-28T14:08:15.322 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:15.319+0000 7f514ffff640 1 --2- 172.21.15.104:0/528465514 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f5134067fc0 0x7f513406a480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:15.322 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:15.319+0000 7f514ffff640 1 --2- 172.21.15.104:0/528465514 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f5134067fc0 0x7f513406a480 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f513c000f70 tx=0x7f513c007a90 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:08:15.329 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:15.327+0000 7f514d7fa640 1 -- 172.21.15.104:0/528465514 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179117 (secure 0 0 0) 0x7f5144022020 con 0x7f515006b830 2024-03-28T14:08:15.489 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:15.487+0000 7f5156c9f640 1 -- 172.21.15.104:0/528465514 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f51500b8560 con 0x7f515006b830 2024-03-28T14:08:15.490 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:15.487+0000 7f514d7fa640 1 -- 172.21.15.104:0/528465514 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v21) ==== 93+0+10 (secure 0 0 0) 0x7f5144086e20 con 0x7f515006b830 2024-03-28T14:08:15.490 INFO:teuthology.orchestra.run.smithi104.stdout:pg_num: 1 2024-03-28T14:08:15.492 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:15.491+0000 7f5156c9f640 1 -- 172.21.15.104:0/528465514 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f5134067fc0 msgr2=0x7f513406a480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:15.492 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:15.491+0000 7f5156c9f640 1 --2- 172.21.15.104:0/528465514 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f5134067fc0 0x7f513406a480 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f513c000f70 tx=0x7f513c007a90 comp rx=0 tx=0).stop 2024-03-28T14:08:15.492 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:15.491+0000 7f5156c9f640 1 -- 172.21.15.104:0/528465514 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f515006b830 msgr2=0x7f515013d2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:15.492 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:15.491+0000 7f5156c9f640 1 --2- 172.21.15.104:0/528465514 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f515006b830 0x7f515013d2c0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f514400d800 tx=0x7f514400dcc0 comp rx=0 tx=0).stop 2024-03-28T14:08:15.492 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:15.491+0000 7f5156c9f640 1 -- 172.21.15.104:0/528465514 shutdown_connections 2024-03-28T14:08:15.492 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:15.491+0000 7f5156c9f640 1 --2- 172.21.15.104:0/528465514 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f51500b54c0 0x7f51501462b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:15.492 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:15.491+0000 7f5156c9f640 1 --2- 172.21.15.104:0/528465514 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f51500b4f70 0x7f5150145d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:15.493 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:15.491+0000 7f5156c9f640 1 --2- 172.21.15.104:0/528465514 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f5134067fc0 0x7f513406a480 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:15.493 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:15.491+0000 7f5156c9f640 1 --2- 172.21.15.104:0/528465514 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f515006b830 0x7f515013d2c0 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:15.493 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:15.491+0000 7f5156c9f640 1 -- 172.21.15.104:0/528465514 >> 172.21.15.104:0/528465514 conn(0x7f51500ba400 msgr2=0x7f51501481f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T14:08:15.493 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:15.491+0000 7f5156c9f640 1 -- 172.21.15.104:0/528465514 shutdown_connections 2024-03-28T14:08:15.493 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:15.491+0000 7f5156c9f640 1 -- 172.21.15.104:0/528465514 wait complete. 2024-03-28T14:08:15.508 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-28T14:08:15.627 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:15.627+0000 7fec7ce82640 1 Processor -- start 2024-03-28T14:08:15.628 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:15.627+0000 7fec7ce82640 1 -- start start 2024-03-28T14:08:15.629 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:15.627+0000 7fec7ce82640 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fec780af9b0 0x7fec780b0010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:15.629 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:15.627+0000 7fec7ce82640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fec780b6780 con 0x7fec781445e0 2024-03-28T14:08:15.629 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:15.627+0000 7fec7ce82640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fec7806dde0 con 0x7fec780b63a0 2024-03-28T14:08:15.629 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:15.627+0000 7fec7ce82640 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 -- 0x7fec780af610 con 0x7fec780af9b0 2024-03-28T14:08:15.629 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:15.627+0000 7fec75d9b640 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fec780af9b0 0x7fec780b0010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:15.629 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:15.627+0000 7fec76d9d640 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7fec781445e0 0x7fec781449f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:41740/0 (socket says 172.21.15.104:41740) 2024-03-28T14:08:15.629 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:15.627+0000 7fec76d9d640 1 -- 172.21.15.104:0/689412345 learned_addr learned my addr 172.21.15.104:0/689412345 (peer_addr_for_me v1:172.21.15.104:0/0) 2024-03-28T14:08:15.629 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:15.627+0000 7fec75d9b640 1 -- 172.21.15.104:0/689412345 >> v1:172.21.15.171:6789/0 conn(0x7fec780b63a0 legacy=0x7fec78141670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T14:08:15.629 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:15.627+0000 7fec75d9b640 1 -- 172.21.15.104:0/689412345 >> v1:172.21.15.104:6789/0 conn(0x7fec781445e0 legacy=0x7fec781449f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:15.629 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:15.627+0000 7fec7559a640 1 -- 172.21.15.104:0/689412345 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1239448650 0 0) 0x7fec780b6780 con 0x7fec781445e0 2024-03-28T14:08:15.629 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:15.627+0000 7fec75d9b640 1 -- 172.21.15.104:0/689412345 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fec7806e0c0 con 0x7fec780af9b0 2024-03-28T14:08:15.632 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:15.631+0000 7fec75d9b640 1 --2- 172.21.15.104:0/689412345 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fec780af9b0 0x7fec780b0010 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fec6c00ba60 tx=0x7fec6c01eaa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ff5e455b8ec1f8ad server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:08:15.632 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:15.631+0000 7fec7559a640 1 -- 172.21.15.104:0/689412345 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec6c0b4070 con 0x7fec780af9b0 2024-03-28T14:08:15.632 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:15.631+0000 7fec7559a640 1 -- 172.21.15.104:0/689412345 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fec6c0a7870 con 0x7fec780af9b0 2024-03-28T14:08:15.632 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:15.631+0000 7fec7559a640 1 -- 172.21.15.104:0/689412345 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec6c0a7c40 con 0x7fec780af9b0 2024-03-28T14:08:15.633 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:15.631+0000 7fec7ce82640 1 -- 172.21.15.104:0/689412345 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fec780af9b0 msgr2=0x7fec780b0010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:15.633 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:15.631+0000 7fec7ce82640 1 --2- 172.21.15.104:0/689412345 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fec780af9b0 0x7fec780b0010 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fec6c00ba60 tx=0x7fec6c01eaa0 comp rx=0 tx=0).stop 2024-03-28T14:08:15.633 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:15.631+0000 7fec7ce82640 1 -- 172.21.15.104:0/689412345 shutdown_connections 2024-03-28T14:08:15.633 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:15.631+0000 7fec7ce82640 1 --2- 172.21.15.104:0/689412345 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fec780af9b0 0x7fec780b0010 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:15.633 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:15.631+0000 7fec7ce82640 1 -- 172.21.15.104:0/689412345 >> 172.21.15.104:0/689412345 conn(0x7fec780ba400 msgr2=0x7fec780ba810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T14:08:15.634 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:15.631+0000 7fec7ce82640 1 -- 172.21.15.104:0/689412345 shutdown_connections 2024-03-28T14:08:15.634 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:15.631+0000 7fec7ce82640 1 -- 172.21.15.104:0/689412345 wait complete. 2024-03-28T14:08:15.641 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:15.631+0000 7fec7ce82640 1 Processor -- start 2024-03-28T14:08:15.641 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:15.631+0000 7fec7ce82640 1 -- start start 2024-03-28T14:08:15.641 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:15.635+0000 7fec7ce82640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fec780af9b0 0x7fec78139510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:15.641 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:15.635+0000 7fec7ce82640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fec780b63a0 0x7fec78138780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:15.641 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:15.635+0000 7fec7ce82640 1 --2- >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fec781445e0 0x7fec780b2450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:15.641 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:15.635+0000 7fec7ce82640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7fec7806c170 con 0x7fec780b63a0 2024-03-28T14:08:15.641 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:15.635+0000 7fec7ce82640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7fec7806bff0 con 0x7fec780af9b0 2024-03-28T14:08:15.641 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:15.635+0000 7fec7ce82640 1 -- --> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] -- mon_getmap magic: 0 -- 0x7fec7806c2f0 con 0x7fec781445e0 2024-03-28T14:08:15.641 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:15.635+0000 7fec75d9b640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fec780b63a0 0x7fec78138780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:15.641 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:15.635+0000 7fec75d9b640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fec780b63a0 0x7fec78138780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:35068/0 (socket says 172.21.15.104:35068) 2024-03-28T14:08:15.641 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:15.635+0000 7fec75d9b640 1 -- 172.21.15.104:0/1197754200 learned_addr learned my addr 172.21.15.104:0/1197754200 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-03-28T14:08:15.642 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:15.635+0000 7fec76d9d640 1 --2- 172.21.15.104:0/1197754200 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fec781445e0 0x7fec780b2450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:15.642 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:15.635+0000 7fec75d9b640 1 -- 172.21.15.104:0/1197754200 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fec781445e0 msgr2=0x7fec780b2450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:15.642 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:15.635+0000 7fec7659c640 1 --2- 172.21.15.104:0/1197754200 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fec780af9b0 0x7fec78139510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:15.642 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:15.635+0000 7fec75d9b640 1 --2- 172.21.15.104:0/1197754200 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fec781445e0 0x7fec780b2450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:15.642 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:15.635+0000 7fec75d9b640 1 -- 172.21.15.104:0/1197754200 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fec780af9b0 msgr2=0x7fec78139510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:15.642 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:15.635+0000 7fec75d9b640 1 --2- 172.21.15.104:0/1197754200 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fec780af9b0 0x7fec78139510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:15.642 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:15.635+0000 7fec75d9b640 1 -- 172.21.15.104:0/1197754200 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fec78139e70 con 0x7fec780b63a0 2024-03-28T14:08:15.642 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:15.635+0000 7fec75d9b640 1 --2- 172.21.15.104:0/1197754200 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fec780b63a0 0x7fec78138780 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fec6c00ba60 tx=0x7fec6c01b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:08:15.642 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:15.635+0000 7fec66ffd640 1 -- 172.21.15.104:0/1197754200 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec6c0bf070 con 0x7fec780b63a0 2024-03-28T14:08:15.642 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:15.635+0000 7fec66ffd640 1 -- 172.21.15.104:0/1197754200 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fec6c004220 con 0x7fec780b63a0 2024-03-28T14:08:15.642 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:15.635+0000 7fec7ce82640 1 -- 172.21.15.104:0/1197754200 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fec781df4d0 con 0x7fec780b63a0 2024-03-28T14:08:15.642 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:15.635+0000 7fec66ffd640 1 -- 172.21.15.104:0/1197754200 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec6c0b4040 con 0x7fec780b63a0 2024-03-28T14:08:15.642 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:15.635+0000 7fec7ce82640 1 -- 172.21.15.104:0/1197754200 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fec78139a50 con 0x7fec780b63a0 2024-03-28T14:08:15.643 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:15.635+0000 7fec7ce82640 1 -- 172.21.15.104:0/1197754200 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fec781df2e0 con 0x7fec780b63a0 2024-03-28T14:08:15.643 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:15.635+0000 7fec66ffd640 1 -- 172.21.15.104:0/1197754200 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) ==== 87803+0+0 (secure 0 0 0) 0x7fec6c0a7730 con 0x7fec780b63a0 2024-03-28T14:08:15.643 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:15.635+0000 7fec66ffd640 1 --2- 172.21.15.104:0/1197754200 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7fec58067fc0 0x7fec5806a480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:15.643 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:15.635+0000 7fec66ffd640 1 -- 172.21.15.104:0/1197754200 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7fec6c1235d0 con 0x7fec780b63a0 2024-03-28T14:08:15.643 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:15.635+0000 7fec7659c640 1 --2- 172.21.15.104:0/1197754200 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7fec58067fc0 0x7fec5806a480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:15.643 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:15.639+0000 7fec7659c640 1 --2- 172.21.15.104:0/1197754200 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7fec58067fc0 0x7fec5806a480 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fec68007fd0 tx=0x7fec68009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:08:15.643 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:15.639+0000 7fec66ffd640 1 -- 172.21.15.104:0/1197754200 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179117 (secure 0 0 0) 0x7fec6c018720 con 0x7fec780b63a0 2024-03-28T14:08:15.830 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:15.827+0000 7fec7ce82640 1 -- 172.21.15.104:0/1197754200 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fec780687c0 con 0x7fec780b63a0 2024-03-28T14:08:15.830 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:15.827+0000 7fec66ffd640 1 -- 172.21.15.104:0/1197754200 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v21) ==== 104+0+11 (secure 0 0 0) 0x7fec6c0f3f30 con 0x7fec780b63a0 2024-03-28T14:08:15.830 INFO:teuthology.orchestra.run.smithi104.stdout:pg_num: 32 2024-03-28T14:08:15.832 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:15.831+0000 7fec7ce82640 1 -- 172.21.15.104:0/1197754200 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7fec58067fc0 msgr2=0x7fec5806a480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:15.832 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:15.831+0000 7fec7ce82640 1 --2- 172.21.15.104:0/1197754200 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7fec58067fc0 0x7fec5806a480 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fec68007fd0 tx=0x7fec68009040 comp rx=0 tx=0).stop 2024-03-28T14:08:15.832 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:15.831+0000 7fec7ce82640 1 -- 172.21.15.104:0/1197754200 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fec780b63a0 msgr2=0x7fec78138780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:15.833 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:15.831+0000 7fec7ce82640 1 --2- 172.21.15.104:0/1197754200 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fec780b63a0 0x7fec78138780 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fec6c00ba60 tx=0x7fec6c01b840 comp rx=0 tx=0).stop 2024-03-28T14:08:15.833 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:15.831+0000 7fec7ce82640 1 -- 172.21.15.104:0/1197754200 shutdown_connections 2024-03-28T14:08:15.833 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:15.831+0000 7fec7ce82640 1 --2- 172.21.15.104:0/1197754200 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fec781445e0 0x7fec780b2450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:15.833 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:15.831+0000 7fec7ce82640 1 --2- 172.21.15.104:0/1197754200 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7fec58067fc0 0x7fec5806a480 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:15.833 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:15.831+0000 7fec7ce82640 1 --2- 172.21.15.104:0/1197754200 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fec780b63a0 0x7fec78138780 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:15.833 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:15.831+0000 7fec7ce82640 1 --2- 172.21.15.104:0/1197754200 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fec780af9b0 0x7fec78139510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:15.833 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:15.831+0000 7fec7ce82640 1 -- 172.21.15.104:0/1197754200 >> 172.21.15.104:0/1197754200 conn(0x7fec780ba400 msgr2=0x7fec78067330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T14:08:15.833 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:15.831+0000 7fec7ce82640 1 -- 172.21.15.104:0/1197754200 shutdown_connections 2024-03-28T14:08:15.833 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:15.831+0000 7fec7ce82640 1 -- 172.21.15.104:0/1197754200 wait complete. 2024-03-28T14:08:15.848 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-28T14:08:15.964 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:15.963+0000 7fae9745f640 1 Processor -- start 2024-03-28T14:08:15.964 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:15.963+0000 7fae9745f640 1 -- start start 2024-03-28T14:08:15.965 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:15.963+0000 7fae9745f640 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fae900b7a40 0x7fae9014b840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:15.965 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:15.963+0000 7fae9745f640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fae900b51e0 con 0x7fae900b4e00 2024-03-28T14:08:15.965 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:15.963+0000 7fae9745f640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fae9014ad60 con 0x7fae90141500 2024-03-28T14:08:15.965 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:15.963+0000 7fae9745f640 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 -- 0x7fae9014b040 con 0x7fae900b7a40 2024-03-28T14:08:15.965 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:15.963+0000 7fae949fa640 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fae900b7a40 0x7fae9014b840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:15.965 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:15.963+0000 7fae951fb640 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7fae900b4e00 0x7fae900b5c60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:41750/0 (socket says 172.21.15.104:41750) 2024-03-28T14:08:15.965 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:15.963+0000 7fae951fb640 1 -- 172.21.15.104:0/232336090 learned_addr learned my addr 172.21.15.104:0/232336090 (peer_addr_for_me v1:172.21.15.104:0/0) 2024-03-28T14:08:15.965 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:15.963+0000 7fae949fa640 1 -- 172.21.15.104:0/232336090 >> v1:172.21.15.171:6789/0 conn(0x7fae90141500 legacy=0x7fae90141930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T14:08:15.965 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:15.963+0000 7fae949fa640 1 -- 172.21.15.104:0/232336090 >> v1:172.21.15.104:6789/0 conn(0x7fae900b4e00 legacy=0x7fae900b5c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:15.965 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:15.963+0000 7fae87fff640 1 -- 172.21.15.104:0/232336090 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 937608616 0 0) 0x7fae900b51e0 con 0x7fae900b4e00 2024-03-28T14:08:15.965 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:15.963+0000 7fae949fa640 1 -- 172.21.15.104:0/232336090 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fae9014b320 con 0x7fae900b7a40 2024-03-28T14:08:15.968 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:15.967+0000 7fae949fa640 1 --2- 172.21.15.104:0/232336090 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fae900b7a40 0x7fae9014b840 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fae800097c0 tx=0x7fae8001c3d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=490dd1ceb05ca2b9 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:08:15.968 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:15.967+0000 7fae87fff640 1 -- 172.21.15.104:0/232336090 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae800ab5f0 con 0x7fae900b7a40 2024-03-28T14:08:15.968 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:15.967+0000 7fae87fff640 1 -- 172.21.15.104:0/232336090 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fae800b0070 con 0x7fae900b7a40 2024-03-28T14:08:15.968 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:15.967+0000 7fae87fff640 1 -- 172.21.15.104:0/232336090 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae800ab9b0 con 0x7fae900b7a40 2024-03-28T14:08:15.969 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:15.967+0000 7fae9745f640 1 -- 172.21.15.104:0/232336090 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fae900b7a40 msgr2=0x7fae9014b840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:15.969 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:15.967+0000 7fae9745f640 1 --2- 172.21.15.104:0/232336090 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fae900b7a40 0x7fae9014b840 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fae800097c0 tx=0x7fae8001c3d0 comp rx=0 tx=0).stop 2024-03-28T14:08:15.969 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:15.967+0000 7fae9745f640 1 -- 172.21.15.104:0/232336090 shutdown_connections 2024-03-28T14:08:15.969 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:15.967+0000 7fae9745f640 1 --2- 172.21.15.104:0/232336090 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fae900b7a40 0x7fae9014b840 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:15.969 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:15.967+0000 7fae9745f640 1 -- 172.21.15.104:0/232336090 >> 172.21.15.104:0/232336090 conn(0x7fae900b9b40 msgr2=0x7fae900b9f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T14:08:15.970 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:15.967+0000 7fae9745f640 1 -- 172.21.15.104:0/232336090 shutdown_connections 2024-03-28T14:08:15.970 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:15.967+0000 7fae9745f640 1 -- 172.21.15.104:0/232336090 wait complete. 2024-03-28T14:08:15.970 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:15.967+0000 7fae9745f640 1 Processor -- start 2024-03-28T14:08:15.970 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:15.967+0000 7fae9745f640 1 -- start start 2024-03-28T14:08:15.972 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:15.967+0000 7fae9745f640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fae900b4e00 0x7fae9013a600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:15.972 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:15.967+0000 7fae9745f640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fae900b7a40 0x7fae901e01b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:15.972 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:15.967+0000 7fae9745f640 1 --2- >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fae90141500 0x7fae901e3720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:15.972 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:15.967+0000 7fae9745f640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7fae90155c10 con 0x7fae900b4e00 2024-03-28T14:08:15.972 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:15.967+0000 7fae9745f640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7fae90155a90 con 0x7fae900b7a40 2024-03-28T14:08:15.972 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:15.967+0000 7fae9745f640 1 -- --> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] -- mon_getmap magic: 0 -- 0x7fae90155d90 con 0x7fae90141500 2024-03-28T14:08:15.973 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:15.967+0000 7fae951fb640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fae900b4e00 0x7fae9013a600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:15.973 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:15.967+0000 7fae951fb640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fae900b4e00 0x7fae9013a600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:35082/0 (socket says 172.21.15.104:35082) 2024-03-28T14:08:15.973 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:15.967+0000 7fae951fb640 1 -- 172.21.15.104:0/1306426636 learned_addr learned my addr 172.21.15.104:0/1306426636 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-03-28T14:08:15.973 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:15.967+0000 7fae949fa640 1 --2- 172.21.15.104:0/1306426636 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fae900b7a40 0x7fae901e01b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:15.973 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:15.967+0000 7fae951fb640 1 -- 172.21.15.104:0/1306426636 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fae90141500 msgr2=0x7fae901e3720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:15.973 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:15.967+0000 7fae959fc640 1 --2- 172.21.15.104:0/1306426636 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fae90141500 0x7fae901e3720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:15.973 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:15.971+0000 7fae951fb640 1 --2- 172.21.15.104:0/1306426636 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fae90141500 0x7fae901e3720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:15.973 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:15.971+0000 7fae951fb640 1 -- 172.21.15.104:0/1306426636 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fae900b7a40 msgr2=0x7fae901e01b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:15.973 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:15.971+0000 7fae951fb640 1 --2- 172.21.15.104:0/1306426636 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fae900b7a40 0x7fae901e01b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:15.973 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:15.971+0000 7fae951fb640 1 -- 172.21.15.104:0/1306426636 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fae901420c0 con 0x7fae900b4e00 2024-03-28T14:08:15.973 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:15.971+0000 7fae949fa640 1 --2- 172.21.15.104:0/1306426636 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fae900b7a40 0x7fae901e01b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T14:08:15.973 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:15.971+0000 7fae951fb640 1 --2- 172.21.15.104:0/1306426636 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fae900b4e00 0x7fae9013a600 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fae8c0011f0 tx=0x7fae8c00c9b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:08:15.973 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:15.971+0000 7fae85ffb640 1 -- 172.21.15.104:0/1306426636 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae8c018070 con 0x7fae900b4e00 2024-03-28T14:08:15.974 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:15.971+0000 7fae85ffb640 1 -- 172.21.15.104:0/1306426636 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fae8c0074e0 con 0x7fae900b4e00 2024-03-28T14:08:15.974 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:15.971+0000 7fae9745f640 1 -- 172.21.15.104:0/1306426636 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fae901423a0 con 0x7fae900b4e00 2024-03-28T14:08:15.974 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:15.971+0000 7fae85ffb640 1 -- 172.21.15.104:0/1306426636 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae8c004a10 con 0x7fae900b4e00 2024-03-28T14:08:15.974 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:15.971+0000 7fae9745f640 1 -- 172.21.15.104:0/1306426636 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fae901472b0 con 0x7fae900b4e00 2024-03-28T14:08:15.975 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:15.971+0000 7fae9745f640 1 -- 172.21.15.104:0/1306426636 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fae9002cc50 con 0x7fae900b4e00 2024-03-28T14:08:15.975 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:15.971+0000 7fae85ffb640 1 -- 172.21.15.104:0/1306426636 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) ==== 87803+0+0 (secure 0 0 0) 0x7fae8c01b070 con 0x7fae900b4e00 2024-03-28T14:08:15.975 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:15.971+0000 7fae85ffb640 1 --2- 172.21.15.104:0/1306426636 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7fae74067ff0 0x7fae7406a4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:15.975 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:15.971+0000 7fae85ffb640 1 -- 172.21.15.104:0/1306426636 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7fae8c0857a0 con 0x7fae900b4e00 2024-03-28T14:08:15.975 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:15.971+0000 7fae949fa640 1 --2- 172.21.15.104:0/1306426636 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7fae74067ff0 0x7fae7406a4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:15.975 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:15.971+0000 7fae949fa640 1 --2- 172.21.15.104:0/1306426636 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7fae74067ff0 0x7fae7406a4b0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fae800af000 tx=0x7fae80008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:08:15.978 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:15.975+0000 7fae85ffb640 1 -- 172.21.15.104:0/1306426636 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179117 (secure 0 0 0) 0x7fae8c055fe0 con 0x7fae900b4e00 2024-03-28T14:08:16.137 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:16.135+0000 7fae9745f640 1 -- 172.21.15.104:0/1306426636 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fae900bc6b0 con 0x7fae900b4e00 2024-03-28T14:08:16.138 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:16.135+0000 7fae85ffb640 1 -- 172.21.15.104:0/1306426636 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v21) ==== 100+0+11 (secure 0 0 0) 0x7fae8c05ae20 con 0x7fae900b4e00 2024-03-28T14:08:16.138 INFO:teuthology.orchestra.run.smithi104.stdout:pg_num: 64 2024-03-28T14:08:16.141 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:16.139+0000 7fae9745f640 1 -- 172.21.15.104:0/1306426636 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7fae74067ff0 msgr2=0x7fae7406a4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:16.141 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:16.139+0000 7fae9745f640 1 --2- 172.21.15.104:0/1306426636 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7fae74067ff0 0x7fae7406a4b0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fae800af000 tx=0x7fae80008040 comp rx=0 tx=0).stop 2024-03-28T14:08:16.141 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:16.139+0000 7fae9745f640 1 -- 172.21.15.104:0/1306426636 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fae900b4e00 msgr2=0x7fae9013a600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:16.141 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:16.139+0000 7fae9745f640 1 --2- 172.21.15.104:0/1306426636 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fae900b4e00 0x7fae9013a600 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fae8c0011f0 tx=0x7fae8c00c9b0 comp rx=0 tx=0).stop 2024-03-28T14:08:16.141 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:16.139+0000 7fae9745f640 1 -- 172.21.15.104:0/1306426636 shutdown_connections 2024-03-28T14:08:16.142 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:16.139+0000 7fae9745f640 1 --2- 172.21.15.104:0/1306426636 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fae90141500 0x7fae901e3720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:16.142 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:16.139+0000 7fae9745f640 1 --2- 172.21.15.104:0/1306426636 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fae900b7a40 0x7fae901e01b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:16.142 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:16.139+0000 7fae9745f640 1 --2- 172.21.15.104:0/1306426636 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7fae74067ff0 0x7fae7406a4b0 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:16.142 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:16.139+0000 7fae9745f640 1 --2- 172.21.15.104:0/1306426636 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fae900b4e00 0x7fae9013a600 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:16.142 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:16.139+0000 7fae9745f640 1 -- 172.21.15.104:0/1306426636 >> 172.21.15.104:0/1306426636 conn(0x7fae900b9b40 msgr2=0x7fae9013be00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T14:08:16.142 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:16.139+0000 7fae9745f640 1 -- 172.21.15.104:0/1306426636 shutdown_connections 2024-03-28T14:08:16.142 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:16.139+0000 7fae9745f640 1 -- 172.21.15.104:0/1306426636 wait complete. 2024-03-28T14:08:16.164 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-28T14:08:16.279 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:16.275+0000 7f5a788f2640 1 Processor -- start 2024-03-28T14:08:16.279 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:16.275+0000 7f5a788f2640 1 -- start start 2024-03-28T14:08:16.280 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:16.279+0000 7f5a788f2640 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f5a74071e00 0x7f5a74149400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:16.281 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:16.279+0000 7f5a788f2640 1 --2- >> v2:172.21.15.171:3300/0 conn(0x7f5a740b1f60 0x7f5a7414ff70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:16.281 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:16.279+0000 7f5a788f2640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5a74148c40 con 0x7f5a740b1550 2024-03-28T14:08:16.281 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:16.279+0000 7f5a788f2640 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 -- 0x7f5a740721e0 con 0x7f5a74071e00 2024-03-28T14:08:16.281 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:16.279+0000 7f5a788f2640 1 -- --> v2:172.21.15.171:3300/0 -- mon_getmap magic: 0 -- 0x7f5a740b1a30 con 0x7f5a740b1f60 2024-03-28T14:08:16.281 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:16.279+0000 7f5a7259c640 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f5a74071e00 0x7f5a74149400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:16.281 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:16.279+0000 7f5a7259c640 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f5a74071e00 0x7f5a74149400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:35096/0 (socket says 172.21.15.104:35096) 2024-03-28T14:08:16.281 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:16.279+0000 7f5a7259c640 1 -- 172.21.15.104:0/756809779 learned_addr learned my addr 172.21.15.104:0/756809779 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-03-28T14:08:16.281 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:16.279+0000 7f5a72d9d640 1 --2- 172.21.15.104:0/756809779 >> v2:172.21.15.171:3300/0 conn(0x7f5a740b1f60 0x7f5a7414ff70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:16.281 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:16.279+0000 7f5a7259c640 1 -- 172.21.15.104:0/756809779 >> v2:172.21.15.171:3300/0 conn(0x7f5a740b1f60 msgr2=0x7f5a7414ff70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T14:08:16.281 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:16.279+0000 7f5a7259c640 1 --2- 172.21.15.104:0/756809779 >> v2:172.21.15.171:3300/0 conn(0x7f5a740b1f60 0x7f5a7414ff70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:16.281 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:16.279+0000 7f5a7259c640 1 -- 172.21.15.104:0/756809779 >> v1:172.21.15.171:6789/0 conn(0x7f5a740b1550 legacy=0x7f5a74145ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T14:08:16.281 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:16.279+0000 7f5a7259c640 1 -- 172.21.15.104:0/756809779 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5a74148ee0 con 0x7f5a74071e00 2024-03-28T14:08:16.283 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:16.279+0000 7f5a7259c640 1 --2- 172.21.15.104:0/756809779 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5a74071e00 0x7f5a74149400 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f5a740b8b00 tx=0x7f5a6401c3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=93563b253733ac04 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:08:16.283 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:16.279+0000 7f5a7159a640 1 -- 172.21.15.104:0/756809779 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a640b0070 con 0x7f5a74071e00 2024-03-28T14:08:16.283 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:16.279+0000 7f5a7159a640 1 -- 172.21.15.104:0/756809779 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5a640ab600 con 0x7f5a74071e00 2024-03-28T14:08:16.283 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:16.279+0000 7f5a7159a640 1 -- 172.21.15.104:0/756809779 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a640ab9d0 con 0x7f5a74071e00 2024-03-28T14:08:16.290 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:16.283+0000 7f5a788f2640 1 -- 172.21.15.104:0/756809779 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5a74071e00 msgr2=0x7f5a74149400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:16.290 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:16.283+0000 7f5a788f2640 1 --2- 172.21.15.104:0/756809779 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5a74071e00 0x7f5a74149400 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f5a740b8b00 tx=0x7f5a6401c3c0 comp rx=0 tx=0).stop 2024-03-28T14:08:16.290 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:16.283+0000 7f5a788f2640 1 -- 172.21.15.104:0/756809779 shutdown_connections 2024-03-28T14:08:16.290 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:16.283+0000 7f5a788f2640 1 --2- 172.21.15.104:0/756809779 >> v2:172.21.15.171:3300/0 conn(0x7f5a740b1f60 0x7f5a7414ff70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:16.290 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:16.283+0000 7f5a788f2640 1 --2- 172.21.15.104:0/756809779 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5a74071e00 0x7f5a74149400 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:16.290 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:16.283+0000 7f5a788f2640 1 -- 172.21.15.104:0/756809779 >> 172.21.15.104:0/756809779 conn(0x7f5a740b7b10 msgr2=0x7f5a740b7f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T14:08:16.291 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:16.283+0000 7f5a788f2640 1 -- 172.21.15.104:0/756809779 shutdown_connections 2024-03-28T14:08:16.291 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:16.283+0000 7f5a788f2640 1 -- 172.21.15.104:0/756809779 wait complete. 2024-03-28T14:08:16.291 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:16.283+0000 7f5a788f2640 1 Processor -- start 2024-03-28T14:08:16.291 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:16.283+0000 7f5a788f2640 1 -- start start 2024-03-28T14:08:16.291 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:16.283+0000 7f5a788f2640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5a74071e00 0x7f5a74141f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:16.291 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:16.283+0000 7f5a788f2640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f5a740b1550 0x7f5a741de460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:16.292 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:16.283+0000 7f5a788f2640 1 --2- >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f5a740b1f60 0x7f5a741de9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:16.292 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:16.283+0000 7f5a788f2640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f5a74072540 con 0x7f5a74071e00 2024-03-28T14:08:16.292 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:16.283+0000 7f5a788f2640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f5a740726c0 con 0x7f5a740b1550 2024-03-28T14:08:16.292 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:16.283+0000 7f5a788f2640 1 -- --> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] -- mon_getmap magic: 0 -- 0x7f5a74152cd0 con 0x7f5a740b1f60 2024-03-28T14:08:16.292 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:16.283+0000 7f5a7259c640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5a74071e00 0x7f5a74141f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:16.293 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:16.283+0000 7f5a7259c640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5a74071e00 0x7f5a74141f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:35102/0 (socket says 172.21.15.104:35102) 2024-03-28T14:08:16.293 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:16.283+0000 7f5a7259c640 1 -- 172.21.15.104:0/2937789756 learned_addr learned my addr 172.21.15.104:0/2937789756 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-03-28T14:08:16.293 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:16.283+0000 7f5a71d9b640 1 --2- 172.21.15.104:0/2937789756 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f5a740b1550 0x7f5a741de460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:16.293 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:16.283+0000 7f5a7259c640 1 -- 172.21.15.104:0/2937789756 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f5a740b1f60 msgr2=0x7f5a741de9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:16.293 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:16.283+0000 7f5a72d9d640 1 --2- 172.21.15.104:0/2937789756 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f5a740b1f60 0x7f5a741de9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:16.293 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:16.283+0000 7f5a7259c640 1 --2- 172.21.15.104:0/2937789756 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f5a740b1f60 0x7f5a741de9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:16.293 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:16.283+0000 7f5a7259c640 1 -- 172.21.15.104:0/2937789756 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f5a740b1550 msgr2=0x7f5a741de460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:16.293 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:16.283+0000 7f5a7259c640 1 --2- 172.21.15.104:0/2937789756 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f5a740b1550 0x7f5a741de460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:16.293 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:16.283+0000 7f5a7259c640 1 -- 172.21.15.104:0/2937789756 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5a7409afd0 con 0x7f5a74071e00 2024-03-28T14:08:16.293 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:16.283+0000 7f5a71d9b640 1 --2- 172.21.15.104:0/2937789756 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f5a740b1550 0x7f5a741de460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T14:08:16.293 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:16.283+0000 7f5a7259c640 1 --2- 172.21.15.104:0/2937789756 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5a74071e00 0x7f5a74141f50 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f5a6401aa70 tx=0x7f5a6401ac00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:08:16.294 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:16.283+0000 7f5a62ffd640 1 -- 172.21.15.104:0/2937789756 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a640ba020 con 0x7f5a74071e00 2024-03-28T14:08:16.294 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:16.283+0000 7f5a62ffd640 1 -- 172.21.15.104:0/2937789756 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5a640b6640 con 0x7f5a74071e00 2024-03-28T14:08:16.294 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:16.283+0000 7f5a62ffd640 1 -- 172.21.15.104:0/2937789756 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a640b0040 con 0x7f5a74071e00 2024-03-28T14:08:16.294 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:16.283+0000 7f5a788f2640 1 -- 172.21.15.104:0/2937789756 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5a740b5230 con 0x7f5a74071e00 2024-03-28T14:08:16.294 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:16.283+0000 7f5a788f2640 1 -- 172.21.15.104:0/2937789756 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5a74140900 con 0x7f5a74071e00 2024-03-28T14:08:16.294 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:16.287+0000 7f5a788f2640 1 -- 172.21.15.104:0/2937789756 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5a740687c0 con 0x7f5a74071e00 2024-03-28T14:08:16.294 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:16.287+0000 7f5a62ffd640 1 -- 172.21.15.104:0/2937789756 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) ==== 87803+0+0 (secure 0 0 0) 0x7f5a640bd020 con 0x7f5a74071e00 2024-03-28T14:08:16.294 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:16.287+0000 7f5a62ffd640 1 --2- 172.21.15.104:0/2937789756 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f5a54067ff0 0x7f5a5406a4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:16.294 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:16.287+0000 7f5a62ffd640 1 -- 172.21.15.104:0/2937789756 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f5a64120750 con 0x7f5a74071e00 2024-03-28T14:08:16.294 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:16.287+0000 7f5a71d9b640 1 --2- 172.21.15.104:0/2937789756 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f5a54067ff0 0x7f5a5406a4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:16.295 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:16.287+0000 7f5a71d9b640 1 --2- 172.21.15.104:0/2937789756 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f5a54067ff0 0x7f5a5406a4b0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f5a740a1590 tx=0x7f5a68009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:08:16.295 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:16.291+0000 7f5a62ffd640 1 -- 172.21.15.104:0/2937789756 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179117 (secure 0 0 0) 0x7f5a640c4030 con 0x7f5a74071e00 2024-03-28T14:08:16.459 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:16.455+0000 7f5a788f2640 1 -- 172.21.15.104:0/2937789756 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f5a7409acf0 con 0x7f5a74071e00 2024-03-28T14:08:16.459 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:16.455+0000 7f5a62ffd640 1 -- 172.21.15.104:0/2937789756 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) ==== 74+0+16629 (secure 0 0 0) 0x7f5a640160d0 con 0x7f5a74071e00 2024-03-28T14:08:16.460 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-03-28T14:08:16.460 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":21,"fsid":"428a90f2-3351-435f-8fcf-f7e66c78a13e","created":"2024-03-28T14:07:34.862458+0000","modified":"2024-03-28T14:08:00.671587+0000","last_up_change":"2024-03-28T14:07:46.028122+0000","last_in_change":"2024-03-28T14:07:37.157486+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-28T14:07:47.622988+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-28T14:07:57.147450+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-28T14:07:57.990975+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"1cb70540-4e42-402f-a446-aaec4ca83007","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6800","nonce":3745690918},{"type":"v1","addr":"172.21.15.104:6801","nonce":3745690918}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":3745690918},{"type":"v1","addr":"172.21.15.104:6803","nonce":3745690918}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":3745690918},{"type":"v1","addr":"172.21.15.104:6807","nonce":3745690918}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":3745690918},{"type":"v1","addr":"172.21.15.104:6805","nonce":3745690918}]},"public_addr":"172.21.15.104:6801/3745690918","cluster_addr":"172.21.15.104:6803/3745690918","heartbeat_back_addr":"172.21.15.104:6807/3745690918","heartbeat_front_addr":"172.21.15.104:6805/3745690918","state":["exists","up"]},{"osd":1,"uuid":"3ab7afdd-b980-418c-81f2-004e16de3dca","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":19438674},{"type":"v1","addr":"172.21.15.104:6810","nonce":19438674}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":19438674},{"type":"v1","addr":"172.21.15.104:6814","nonce":19438674}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":19438674},{"type":"v1","addr":"172.21.15.104:6823","nonce":19438674}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":19438674},{"type":"v1","addr":"172.21.15.104:6819","nonce":19438674}]},"public_addr":"172.21.15.104:6810/19438674","cluster_addr":"172.21.15.104:6814/19438674","heartbeat_back_addr":"172.21.15.104:6823/19438674","heartbeat_front_addr":"172.21.15.104:6819/19438674","state":["exists","up"]},{"osd":2,"uuid":"3e2651c8-ecae-4a9e-a053-27240a79eac2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":2352012626},{"type":"v1","addr":"172.21.15.104:6825","nonce":2352012626}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6826","nonce":2352012626},{"type":"v1","addr":"172.21.15.104:6827","nonce":2352012626}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":2352012626},{"type":"v1","addr":"172.21.15.104:6831","nonce":2352012626}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6828","nonce":2352012626},{"type":"v1","addr":"172.21.15.104:6829","nonce":2352012626}]},"public_addr":"172.21.15.104:6825/2352012626","cluster_addr":"172.21.15.104:6827/2352012626","heartbeat_back_addr":"172.21.15.104:6831/2352012626","heartbeat_front_addr":"172.21.15.104:6829/2352012626","state":["exists","up"]},{"osd":3,"uuid":"925c7e0a-38ef-47cd-b288-8bec8456fb17","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6809","nonce":3568347381},{"type":"v1","addr":"172.21.15.104:6811","nonce":3568347381}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6813","nonce":3568347381},{"type":"v1","addr":"172.21.15.104:6815","nonce":3568347381}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":3568347381},{"type":"v1","addr":"172.21.15.104:6822","nonce":3568347381}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6817","nonce":3568347381},{"type":"v1","addr":"172.21.15.104:6818","nonce":3568347381}]},"public_addr":"172.21.15.104:6811/3568347381","cluster_addr":"172.21.15.104:6815/3568347381","heartbeat_back_addr":"172.21.15.104:6822/3568347381","heartbeat_front_addr":"172.21.15.104:6818/3568347381","state":["exists","up"]},{"osd":4,"uuid":"4a8fac86-e595-4a0e-96e3-57edb3cc3f57","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6800","nonce":2107001077},{"type":"v1","addr":"172.21.15.171:6801","nonce":2107001077}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6802","nonce":2107001077},{"type":"v1","addr":"172.21.15.171:6803","nonce":2107001077}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6806","nonce":2107001077},{"type":"v1","addr":"172.21.15.171:6807","nonce":2107001077}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6804","nonce":2107001077},{"type":"v1","addr":"172.21.15.171:6805","nonce":2107001077}]},"public_addr":"172.21.15.171:6801/2107001077","cluster_addr":"172.21.15.171:6803/2107001077","heartbeat_back_addr":"172.21.15.171:6807/2107001077","heartbeat_front_addr":"172.21.15.171:6805/2107001077","state":["exists","up"]},{"osd":5,"uuid":"e48d50b0-beae-40ae-b15a-9a84dbfe88ab","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6808","nonce":2934826925},{"type":"v1","addr":"172.21.15.171:6810","nonce":2934826925}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6812","nonce":2934826925},{"type":"v1","addr":"172.21.15.171:6814","nonce":2934826925}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6819","nonce":2934826925},{"type":"v1","addr":"172.21.15.171:6821","nonce":2934826925}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6816","nonce":2934826925},{"type":"v1","addr":"172.21.15.171:6817","nonce":2934826925}]},"public_addr":"172.21.15.171:6810/2934826925","cluster_addr":"172.21.15.171:6814/2934826925","heartbeat_back_addr":"172.21.15.171:6821/2934826925","heartbeat_front_addr":"172.21.15.171:6817/2934826925","state":["exists","up"]},{"osd":6,"uuid":"5e7c3292-9936-4f35-b267-aed9d4e49323","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6824","nonce":436742919},{"type":"v1","addr":"172.21.15.171:6825","nonce":436742919}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6826","nonce":436742919},{"type":"v1","addr":"172.21.15.171:6827","nonce":436742919}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6830","nonce":436742919},{"type":"v1","addr":"172.21.15.171:6831","nonce":436742919}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6828","nonce":436742919},{"type":"v1","addr":"172.21.15.171:6829","nonce":436742919}]},"public_addr":"172.21.15.171:6825/436742919","cluster_addr":"172.21.15.171:6827/436742919","heartbeat_back_addr":"172.21.15.171:6831/436742919","heartbeat_front_addr":"172.21.15.171:6829/436742919","state":["exists","up"]},{"osd":7,"uuid":"339b8414-d7f0-4fb0-a446-efaf635cb7ff","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6809","nonce":2055631175},{"type":"v1","addr":"172.21.15.171:6811","nonce":2055631175}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6813","nonce":2055631175},{"type":"v1","addr":"172.21.15.171:6815","nonce":2055631175}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6822","nonce":2055631175},{"type":"v1","addr":"172.21.15.171:6823","nonce":2055631175}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6818","nonce":2055631175},{"type":"v1","addr":"172.21.15.171:6820","nonce":2055631175}]},"public_addr":"172.21.15.171:6811/2055631175","cluster_addr":"172.21.15.171:6815/2055631175","heartbeat_back_addr":"172.21.15.171:6823/2055631175","heartbeat_front_addr":"172.21.15.171:6820/2055631175","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T14:07:42.668881+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T14:07:42.708397+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T14:07:42.634191+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T14:07:42.733662+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T14:07:42.651541+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T14:07:42.783181+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T14:07:42.766310+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T14:07:42.670797+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-28T14:08:16.461 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:16.455+0000 7f5a788f2640 1 -- 172.21.15.104:0/2937789756 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f5a54067ff0 msgr2=0x7f5a5406a4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:16.461 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:16.455+0000 7f5a788f2640 1 --2- 172.21.15.104:0/2937789756 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f5a54067ff0 0x7f5a5406a4b0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f5a740a1590 tx=0x7f5a68009040 comp rx=0 tx=0).stop 2024-03-28T14:08:16.461 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:16.455+0000 7f5a788f2640 1 -- 172.21.15.104:0/2937789756 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5a74071e00 msgr2=0x7f5a74141f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:16.461 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:16.455+0000 7f5a788f2640 1 --2- 172.21.15.104:0/2937789756 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5a74071e00 0x7f5a74141f50 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f5a6401aa70 tx=0x7f5a6401ac00 comp rx=0 tx=0).stop 2024-03-28T14:08:16.461 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:16.455+0000 7f5a788f2640 1 -- 172.21.15.104:0/2937789756 shutdown_connections 2024-03-28T14:08:16.461 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:16.455+0000 7f5a788f2640 1 --2- 172.21.15.104:0/2937789756 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f5a740b1f60 0x7f5a741de9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:16.461 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:16.455+0000 7f5a788f2640 1 --2- 172.21.15.104:0/2937789756 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f5a740b1550 0x7f5a741de460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:16.461 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:16.455+0000 7f5a788f2640 1 --2- 172.21.15.104:0/2937789756 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f5a54067ff0 0x7f5a5406a4b0 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:16.461 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:16.455+0000 7f5a788f2640 1 --2- 172.21.15.104:0/2937789756 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5a74071e00 0x7f5a74141f50 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:16.461 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:16.455+0000 7f5a788f2640 1 -- 172.21.15.104:0/2937789756 >> 172.21.15.104:0/2937789756 conn(0x7f5a740b7b10 msgr2=0x7f5a740b9eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T14:08:16.461 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:16.459+0000 7f5a788f2640 1 -- 172.21.15.104:0/2937789756 shutdown_connections 2024-03-28T14:08:16.461 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:16.459+0000 7f5a788f2640 1 -- 172.21.15.104:0/2937789756 wait complete. 2024-03-28T14:08:16.476 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-28T14:07:47.622988+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-28T14:07:57.147450+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-28T14:07:57.990975+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-28T14:08:16.477 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-28T14:08:16.596 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:16.595+0000 7f2ede4c5640 1 Processor -- start 2024-03-28T14:08:16.597 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:16.595+0000 7f2ede4c5640 1 -- start start 2024-03-28T14:08:16.599 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:16.595+0000 7f2ede4c5640 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f2ed8145210 0x7f2ed80b4fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:16.599 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:16.595+0000 7f2ede4c5640 1 --2- >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f2ed80b7940 0x7f2ed8141540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:16.599 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:16.595+0000 7f2ede4c5640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2ed802c840 con 0x7f2ed802c460 2024-03-28T14:08:16.600 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:16.595+0000 7f2ede4c5640 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 -- 0x7f2ed80b7df0 con 0x7f2ed8145210 2024-03-28T14:08:16.600 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:16.595+0000 7f2ede4c5640 1 -- --> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] -- mon_getmap magic: 0 -- 0x7f2ed80b5690 con 0x7f2ed80b7940 2024-03-28T14:08:16.600 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:16.595+0000 7f2ed77fe640 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f2ed8145210 0x7f2ed80b4fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:16.600 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:16.595+0000 7f2ed7fff640 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f2ed802c460 0x7f2ed8144b00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:46100/0 (socket says 172.21.15.104:46100) 2024-03-28T14:08:16.600 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:16.595+0000 7f2ed7fff640 1 -- 172.21.15.104:0/572598539 learned_addr learned my addr 172.21.15.104:0/572598539 (peer_addr_for_me v1:172.21.15.104:0/0) 2024-03-28T14:08:16.601 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:16.595+0000 7f2edca62640 1 --2- 172.21.15.104:0/572598539 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f2ed80b7940 0x7f2ed8141540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:16.601 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:16.595+0000 7f2ed77fe640 1 -- 172.21.15.104:0/572598539 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f2ed80b7940 msgr2=0x7f2ed8141540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T14:08:16.601 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:16.595+0000 7f2ed77fe640 1 --2- 172.21.15.104:0/572598539 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f2ed80b7940 0x7f2ed8141540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:16.601 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:16.595+0000 7f2ed77fe640 1 -- 172.21.15.104:0/572598539 >> v1:172.21.15.104:6789/0 conn(0x7f2ed802c460 legacy=0x7f2ed8144b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:16.601 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:16.595+0000 7f2ed77fe640 1 -- 172.21.15.104:0/572598539 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2ed81445e0 con 0x7f2ed8145210 2024-03-28T14:08:16.607 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:16.599+0000 7f2ed77fe640 1 --2- 172.21.15.104:0/572598539 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2ed8145210 0x7f2ed80b4fe0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f2ec80097c0 tx=0x7f2ec801c3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a761b38ce12184fe server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:08:16.607 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:16.599+0000 7f2ed6ffd640 1 -- 172.21.15.104:0/572598539 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ec80b0070 con 0x7f2ed8145210 2024-03-28T14:08:16.607 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:16.599+0000 7f2ed6ffd640 1 -- 172.21.15.104:0/572598539 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2ec80ab600 con 0x7f2ed8145210 2024-03-28T14:08:16.607 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:16.599+0000 7f2ed6ffd640 1 -- 172.21.15.104:0/572598539 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ec80ab9d0 con 0x7f2ed8145210 2024-03-28T14:08:16.607 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:16.599+0000 7f2ede4c5640 1 -- 172.21.15.104:0/572598539 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2ed8145210 msgr2=0x7f2ed80b4fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:16.607 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:16.599+0000 7f2ede4c5640 1 --2- 172.21.15.104:0/572598539 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2ed8145210 0x7f2ed80b4fe0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f2ec80097c0 tx=0x7f2ec801c3c0 comp rx=0 tx=0).stop 2024-03-28T14:08:16.608 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:16.599+0000 7f2ede4c5640 1 -- 172.21.15.104:0/572598539 shutdown_connections 2024-03-28T14:08:16.608 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:16.599+0000 7f2ede4c5640 1 --2- 172.21.15.104:0/572598539 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f2ed80b7940 0x7f2ed8141540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:16.608 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:16.599+0000 7f2ede4c5640 1 --2- 172.21.15.104:0/572598539 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2ed8145210 0x7f2ed80b4fe0 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:16.608 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:16.599+0000 7f2ede4c5640 1 -- 172.21.15.104:0/572598539 >> 172.21.15.104:0/572598539 conn(0x7f2ed80b9660 msgr2=0x7f2ed80b9a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T14:08:16.608 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:16.599+0000 7f2ede4c5640 1 -- 172.21.15.104:0/572598539 shutdown_connections 2024-03-28T14:08:16.608 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:16.599+0000 7f2ede4c5640 1 -- 172.21.15.104:0/572598539 wait complete. 2024-03-28T14:08:16.608 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:16.599+0000 7f2ede4c5640 1 Processor -- start 2024-03-28T14:08:16.608 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:16.599+0000 7f2ede4c5640 1 -- start start 2024-03-28T14:08:16.608 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:16.599+0000 7f2ede4c5640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f2ed802c460 0x7f2ed81de5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:16.608 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:16.599+0000 7f2ede4c5640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2ed80b7940 0x7f2ed8072710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:16.608 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:16.599+0000 7f2ede4c5640 1 --2- >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f2ed8145210 0x7f2ed8072c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:16.608 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:16.599+0000 7f2ede4c5640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f2ed8141c00 con 0x7f2ed80b7940 2024-03-28T14:08:16.608 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:16.599+0000 7f2ede4c5640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f2ed8141a80 con 0x7f2ed802c460 2024-03-28T14:08:16.609 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:16.599+0000 7f2ede4c5640 1 -- --> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] -- mon_getmap magic: 0 -- 0x7f2ed8141d80 con 0x7f2ed8145210 2024-03-28T14:08:16.609 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:16.599+0000 7f2ed77fe640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2ed80b7940 0x7f2ed8072710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:16.609 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:16.599+0000 7f2ed77fe640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2ed80b7940 0x7f2ed8072710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:42288/0 (socket says 172.21.15.104:42288) 2024-03-28T14:08:16.609 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:16.599+0000 7f2ed77fe640 1 -- 172.21.15.104:0/4099191150 learned_addr learned my addr 172.21.15.104:0/4099191150 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-03-28T14:08:16.609 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:16.599+0000 7f2edca62640 1 --2- 172.21.15.104:0/4099191150 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f2ed8145210 0x7f2ed8072c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:16.609 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:16.599+0000 7f2ed7fff640 1 --2- 172.21.15.104:0/4099191150 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f2ed802c460 0x7f2ed81de5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:16.609 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:16.599+0000 7f2ed77fe640 1 -- 172.21.15.104:0/4099191150 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f2ed8145210 msgr2=0x7f2ed8072c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:16.609 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:16.599+0000 7f2ed77fe640 1 --2- 172.21.15.104:0/4099191150 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f2ed8145210 0x7f2ed8072c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:16.609 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:16.599+0000 7f2ed77fe640 1 -- 172.21.15.104:0/4099191150 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f2ed802c460 msgr2=0x7f2ed81de5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:16.609 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:16.599+0000 7f2ed77fe640 1 --2- 172.21.15.104:0/4099191150 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f2ed802c460 0x7f2ed81de5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:16.609 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:16.599+0000 7f2ed77fe640 1 -- 172.21.15.104:0/4099191150 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2ed81ce140 con 0x7f2ed80b7940 2024-03-28T14:08:16.609 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:16.603+0000 7f2ed77fe640 1 --2- 172.21.15.104:0/4099191150 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2ed80b7940 0x7f2ed8072710 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f2ec801cc60 tx=0x7f2ec8019160 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:08:16.609 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:16.603+0000 7f2ed4ff9640 1 -- 172.21.15.104:0/4099191150 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ec80b0040 con 0x7f2ed80b7940 2024-03-28T14:08:16.610 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:16.603+0000 7f2ed4ff9640 1 -- 172.21.15.104:0/4099191150 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2ec80bd070 con 0x7f2ed80b7940 2024-03-28T14:08:16.610 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:16.603+0000 7f2ed4ff9640 1 -- 172.21.15.104:0/4099191150 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ec80aac40 con 0x7f2ed80b7940 2024-03-28T14:08:16.610 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:16.603+0000 7f2ede4c5640 1 -- 172.21.15.104:0/4099191150 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2ed81cc310 con 0x7f2ed80b7940 2024-03-28T14:08:16.610 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:16.603+0000 7f2ede4c5640 1 -- 172.21.15.104:0/4099191150 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2ed81cc830 con 0x7f2ed80b7940 2024-03-28T14:08:16.610 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:16.603+0000 7f2ede4c5640 1 -- 172.21.15.104:0/4099191150 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2ed80687c0 con 0x7f2ed80b7940 2024-03-28T14:08:16.610 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:16.603+0000 7f2ed4ff9640 1 -- 172.21.15.104:0/4099191150 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) ==== 87803+0+0 (secure 0 0 0) 0x7f2ec80b3070 con 0x7f2ed80b7940 2024-03-28T14:08:16.610 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:16.603+0000 7f2ed4ff9640 1 --2- 172.21.15.104:0/4099191150 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f2ec4067fc0 0x7f2ec406a480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:16.610 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:16.603+0000 7f2ed4ff9640 1 -- 172.21.15.104:0/4099191150 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f2ec81209e0 con 0x7f2ed80b7940 2024-03-28T14:08:16.610 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:16.603+0000 7f2ed7fff640 1 --2- 172.21.15.104:0/4099191150 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f2ec4067fc0 0x7f2ec406a480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:16.610 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:16.603+0000 7f2ed7fff640 1 --2- 172.21.15.104:0/4099191150 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f2ec4067fc0 0x7f2ec406a480 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f2ecc001ec0 tx=0x7f2ecc00b040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:08:16.611 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:16.607+0000 7f2ed4ff9640 1 -- 172.21.15.104:0/4099191150 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179117 (secure 0 0 0) 0x7f2ec8015fb0 con 0x7f2ed80b7940 2024-03-28T14:08:16.768 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:16.767+0000 7f2ede4c5640 1 -- 172.21.15.104:0/4099191150 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f2ed81cc4e0 con 0x7f2ed80b7940 2024-03-28T14:08:16.768 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:16.767+0000 7f2ed4ff9640 1 -- 172.21.15.104:0/4099191150 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v21) ==== 93+0+10 (secure 0 0 0) 0x7f2ec80f1200 con 0x7f2ed80b7940 2024-03-28T14:08:16.769 INFO:teuthology.orchestra.run.smithi104.stdout:pg_num: 1 2024-03-28T14:08:16.771 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:16.767+0000 7f2ede4c5640 1 -- 172.21.15.104:0/4099191150 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f2ec4067fc0 msgr2=0x7f2ec406a480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:16.771 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:16.767+0000 7f2ede4c5640 1 --2- 172.21.15.104:0/4099191150 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f2ec4067fc0 0x7f2ec406a480 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f2ecc001ec0 tx=0x7f2ecc00b040 comp rx=0 tx=0).stop 2024-03-28T14:08:16.771 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:16.767+0000 7f2ede4c5640 1 -- 172.21.15.104:0/4099191150 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2ed80b7940 msgr2=0x7f2ed8072710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:16.771 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:16.767+0000 7f2ede4c5640 1 --2- 172.21.15.104:0/4099191150 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2ed80b7940 0x7f2ed8072710 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f2ec801cc60 tx=0x7f2ec8019160 comp rx=0 tx=0).stop 2024-03-28T14:08:16.771 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:16.767+0000 7f2ede4c5640 1 -- 172.21.15.104:0/4099191150 shutdown_connections 2024-03-28T14:08:16.772 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:16.767+0000 7f2ede4c5640 1 --2- 172.21.15.104:0/4099191150 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f2ed8145210 0x7f2ed8072c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:16.772 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:16.767+0000 7f2ede4c5640 1 --2- 172.21.15.104:0/4099191150 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f2ec4067fc0 0x7f2ec406a480 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:16.772 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:16.767+0000 7f2ede4c5640 1 --2- 172.21.15.104:0/4099191150 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2ed80b7940 0x7f2ed8072710 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:16.772 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:16.767+0000 7f2ede4c5640 1 --2- 172.21.15.104:0/4099191150 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f2ed802c460 0x7f2ed81de5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:16.772 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:16.767+0000 7f2ede4c5640 1 -- 172.21.15.104:0/4099191150 >> 172.21.15.104:0/4099191150 conn(0x7f2ed80b9660 msgr2=0x7f2ed8067330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T14:08:16.772 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:16.767+0000 7f2ede4c5640 1 -- 172.21.15.104:0/4099191150 shutdown_connections 2024-03-28T14:08:16.772 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:16.767+0000 7f2ede4c5640 1 -- 172.21.15.104:0/4099191150 wait complete. 2024-03-28T14:08:16.787 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-28T14:08:16.902 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:16.899+0000 7f72af477640 1 Processor -- start 2024-03-28T14:08:16.903 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:16.899+0000 7f72af477640 1 -- start start 2024-03-28T14:08:16.905 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:16.899+0000 7f72af477640 1 --2- >> v2:172.21.15.171:3300/0 conn(0x7f72a80b63a0 0x7f72a80af9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:16.905 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:16.899+0000 7f72af477640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f72a80b6780 con 0x7f72a81445e0 2024-03-28T14:08:16.905 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:16.899+0000 7f72af477640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f72a806dde0 con 0x7f72a80b0010 2024-03-28T14:08:16.905 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:16.899+0000 7f72af477640 1 -- --> v2:172.21.15.171:3300/0 -- mon_getmap magic: 0 -- 0x7f72a80b04c0 con 0x7f72a80b63a0 2024-03-28T14:08:16.905 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:16.903+0000 7f72ada14640 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f72a81445e0 0x7f72a81449f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:46102/0 (socket says 172.21.15.104:46102) 2024-03-28T14:08:16.906 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:16.903+0000 7f72ada14640 1 -- 172.21.15.104:0/1118423259 learned_addr learned my addr 172.21.15.104:0/1118423259 (peer_addr_for_me v1:172.21.15.104:0/0) 2024-03-28T14:08:16.906 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:16.903+0000 7f72ad213640 1 --2- 172.21.15.104:0/1118423259 >> v2:172.21.15.171:3300/0 conn(0x7f72a80b63a0 0x7f72a80af9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:16.906 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:16.903+0000 7f72a7fff640 1 -- 172.21.15.104:0/1118423259 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3672929984 0 0) 0x7f72a80b6780 con 0x7f72a81445e0 2024-03-28T14:08:16.906 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:16.903+0000 7f72a7fff640 1 -- 172.21.15.104:0/1118423259 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7290003530 con 0x7f72a81445e0 2024-03-28T14:08:16.906 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:16.903+0000 7f72a7fff640 1 -- 172.21.15.104:0/1118423259 <== mon.0 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1509303805 0 0) 0x7f7290003530 con 0x7f72a81445e0 2024-03-28T14:08:16.906 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:16.903+0000 7f72a7fff640 1 -- 172.21.15.104:0/1118423259 >> v2:172.21.15.171:3300/0 conn(0x7f72a80b63a0 msgr2=0x7f72a80af9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T14:08:16.906 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:16.903+0000 7f72a7fff640 1 --2- 172.21.15.104:0/1118423259 >> v2:172.21.15.171:3300/0 conn(0x7f72a80b63a0 0x7f72a80af9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:16.906 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:16.903+0000 7f72a7fff640 1 -- 172.21.15.104:0/1118423259 >> v1:172.21.15.171:6789/0 conn(0x7f72a80b0010 legacy=0x7f72a8141670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:16.906 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:16.903+0000 7f72a7fff640 1 -- 172.21.15.104:0/1118423259 --> v1:172.21.15.104:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f72a806e0c0 con 0x7f72a81445e0 2024-03-28T14:08:16.906 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:16.903+0000 7f72a7fff640 1 -- 172.21.15.104:0/1118423259 <== mon.0 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3745580849 0 0) 0x7f7298001680 con 0x7f72a81445e0 2024-03-28T14:08:16.906 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:16.903+0000 7f72a7fff640 1 -- 172.21.15.104:0/1118423259 >> v1:172.21.15.104:6789/0 conn(0x7f72a81445e0 legacy=0x7f72a81449f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:16.906 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:16.903+0000 7f72a7fff640 1 --2- 172.21.15.104:0/1118423259 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f72900039c0 0x7f7290003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:16.906 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:16.903+0000 7f72a7fff640 1 --2- 172.21.15.104:0/1118423259 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f7290004810 0x7f7290004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:16.907 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:16.903+0000 7f72a7fff640 1 --2- 172.21.15.104:0/1118423259 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f72900051d0 0x7f729000b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:16.907 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:16.903+0000 7f72a7fff640 1 -- 172.21.15.104:0/1118423259 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f72a80b04c0 con 0x7f72900039c0 2024-03-28T14:08:16.907 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:16.903+0000 7f72a7fff640 1 -- 172.21.15.104:0/1118423259 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f729000c170 con 0x7f72900051d0 2024-03-28T14:08:16.907 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:16.903+0000 7f72a7fff640 1 -- 172.21.15.104:0/1118423259 --> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] -- mon_getmap magic: 0 -- 0x7f729000c370 con 0x7f7290004810 2024-03-28T14:08:16.907 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:16.903+0000 7f72ad213640 1 --2- 172.21.15.104:0/1118423259 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f72900039c0 0x7f7290003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:16.907 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:16.903+0000 7f72ada14640 1 --2- 172.21.15.104:0/1118423259 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f72900051d0 0x7f729000b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:16.907 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:16.903+0000 7f72aca12640 1 --2- 172.21.15.104:0/1118423259 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f7290004810 0x7f7290004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:16.907 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:16.903+0000 7f72ad213640 1 -- 172.21.15.104:0/1118423259 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f7290004810 msgr2=0x7f7290004c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T14:08:16.907 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:16.903+0000 7f72ad213640 1 --2- 172.21.15.104:0/1118423259 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f7290004810 0x7f7290004c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:16.907 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:16.903+0000 7f72ad213640 1 -- 172.21.15.104:0/1118423259 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f72900051d0 msgr2=0x7f729000b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T14:08:16.907 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:16.903+0000 7f72ad213640 1 --2- 172.21.15.104:0/1118423259 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f72900051d0 0x7f729000b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:16.907 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:16.903+0000 7f72ad213640 1 -- 172.21.15.104:0/1118423259 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7290003530 con 0x7f72900039c0 2024-03-28T14:08:16.907 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:16.903+0000 7f72ada14640 1 --2- 172.21.15.104:0/1118423259 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f72900051d0 0x7f729000b9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T14:08:16.907 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:16.903+0000 7f72ada14640 1 -- 172.21.15.104:0/1118423259 reap_dead start 2024-03-28T14:08:16.910 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:16.903+0000 7f72ad213640 1 --2- 172.21.15.104:0/1118423259 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f72900039c0 0x7f7290003de0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f72a0009ae0 tx=0x7f72a001ca70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=75fa56d9b4d06a47 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:08:16.910 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:16.903+0000 7f72a7fff640 1 -- 172.21.15.104:0/1118423259 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72a00096b0 con 0x7f72900039c0 2024-03-28T14:08:16.911 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:16.903+0000 7f72a7fff640 1 -- 172.21.15.104:0/1118423259 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f72a00a6450 con 0x7f72900039c0 2024-03-28T14:08:16.911 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:16.903+0000 7f72a7fff640 1 -- 172.21.15.104:0/1118423259 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72a00a6840 con 0x7f72900039c0 2024-03-28T14:08:16.911 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:16.903+0000 7f72af477640 1 -- 172.21.15.104:0/1118423259 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f72900039c0 msgr2=0x7f7290003de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:16.911 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:16.903+0000 7f72af477640 1 --2- 172.21.15.104:0/1118423259 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f72900039c0 0x7f7290003de0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f72a0009ae0 tx=0x7f72a001ca70 comp rx=0 tx=0).stop 2024-03-28T14:08:16.911 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:16.903+0000 7f72af477640 1 -- 172.21.15.104:0/1118423259 shutdown_connections 2024-03-28T14:08:16.911 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:16.903+0000 7f72af477640 1 --2- 172.21.15.104:0/1118423259 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f72900039c0 0x7f7290003de0 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:16.911 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:16.903+0000 7f72af477640 1 -- 172.21.15.104:0/1118423259 >> 172.21.15.104:0/1118423259 conn(0x7f72a80ba400 msgr2=0x7f72a80ba810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T14:08:16.911 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:16.907+0000 7f72af477640 1 -- 172.21.15.104:0/1118423259 shutdown_connections 2024-03-28T14:08:16.911 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:16.907+0000 7f72af477640 1 -- 172.21.15.104:0/1118423259 wait complete. 2024-03-28T14:08:16.911 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:16.907+0000 7f72af477640 1 Processor -- start 2024-03-28T14:08:16.911 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:16.907+0000 7f72af477640 1 -- start start 2024-03-28T14:08:16.911 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:16.907+0000 7f72af477640 1 --2- >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f72900039c0 0x7f72a80b5350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:16.912 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:16.907+0000 7f72af477640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f72a81e48a0 0x7f72a8070cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:16.912 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:16.907+0000 7f72af477640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f72a813b3b0 0x7f72a8146ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:16.912 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:16.907+0000 7f72af477640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f72a8147bd0 con 0x7f72a813b3b0 2024-03-28T14:08:16.912 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:16.907+0000 7f72af477640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f72a8147a50 con 0x7f72a81e48a0 2024-03-28T14:08:16.912 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:16.907+0000 7f72af477640 1 -- --> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] -- mon_getmap magic: 0 -- 0x7f72a8147d50 con 0x7f72900039c0 2024-03-28T14:08:16.912 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:16.907+0000 7f72ada14640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f72a813b3b0 0x7f72a8146ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:16.912 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:16.907+0000 7f72ada14640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f72a813b3b0 0x7f72a8146ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:42312/0 (socket says 172.21.15.104:42312) 2024-03-28T14:08:16.912 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:16.907+0000 7f72ada14640 1 -- 172.21.15.104:0/3078379517 learned_addr learned my addr 172.21.15.104:0/3078379517 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-03-28T14:08:16.912 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:16.907+0000 7f72aca12640 1 --2- 172.21.15.104:0/3078379517 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f72a81e48a0 0x7f72a8070cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:16.912 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:16.907+0000 7f72ad213640 1 --2- 172.21.15.104:0/3078379517 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f72900039c0 0x7f72a80b5350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:16.912 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:16.907+0000 7f72ada14640 1 -- 172.21.15.104:0/3078379517 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f72900039c0 msgr2=0x7f72a80b5350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:16.912 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:16.907+0000 7f72ada14640 1 --2- 172.21.15.104:0/3078379517 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f72900039c0 0x7f72a80b5350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:16.912 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:16.907+0000 7f72ada14640 1 -- 172.21.15.104:0/3078379517 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f72a81e48a0 msgr2=0x7f72a8070cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:16.912 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:16.907+0000 7f72ada14640 1 --2- 172.21.15.104:0/3078379517 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f72a81e48a0 0x7f72a8070cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:16.913 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:16.907+0000 7f72ada14640 1 -- 172.21.15.104:0/3078379517 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f72a81e3dc0 con 0x7f72a813b3b0 2024-03-28T14:08:16.913 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:16.907+0000 7f72aca12640 1 --2- 172.21.15.104:0/3078379517 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f72a81e48a0 0x7f72a8070cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T14:08:16.913 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:16.907+0000 7f72ada14640 1 --2- 172.21.15.104:0/3078379517 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f72a813b3b0 0x7f72a8146ed0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f72a80b7ba0 tx=0x7f729800d8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:08:16.913 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:16.907+0000 7f72a5ffb640 1 -- 172.21.15.104:0/3078379517 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7298007440 con 0x7f72a813b3b0 2024-03-28T14:08:16.913 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:16.907+0000 7f72a5ffb640 1 -- 172.21.15.104:0/3078379517 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7298007600 con 0x7f72a813b3b0 2024-03-28T14:08:16.913 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:16.907+0000 7f72af477640 1 -- 172.21.15.104:0/3078379517 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f72a81e40a0 con 0x7f72a813b3b0 2024-03-28T14:08:16.913 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:16.907+0000 7f72a5ffb640 1 -- 172.21.15.104:0/3078379517 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7298004a30 con 0x7f72a813b3b0 2024-03-28T14:08:16.913 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:16.907+0000 7f72af477640 1 -- 172.21.15.104:0/3078379517 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f72a8142140 con 0x7f72a813b3b0 2024-03-28T14:08:16.913 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:16.907+0000 7f72af477640 1 -- 172.21.15.104:0/3078379517 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f72a80687c0 con 0x7f72a813b3b0 2024-03-28T14:08:16.914 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:16.911+0000 7f72a5ffb640 1 -- 172.21.15.104:0/3078379517 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) ==== 87803+0+0 (secure 0 0 0) 0x7f7298004bd0 con 0x7f72a813b3b0 2024-03-28T14:08:16.914 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:16.911+0000 7f72a5ffb640 1 --2- 172.21.15.104:0/3078379517 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f7294067fc0 0x7f729406a480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:16.914 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:16.911+0000 7f72a5ffb640 1 -- 172.21.15.104:0/3078379517 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f7298085f20 con 0x7f72a813b3b0 2024-03-28T14:08:16.914 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:16.911+0000 7f72ad213640 1 --2- 172.21.15.104:0/3078379517 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f7294067fc0 0x7f729406a480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:16.914 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:16.911+0000 7f72ad213640 1 --2- 172.21.15.104:0/3078379517 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f7294067fc0 0x7f729406a480 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f72a00ad000 tx=0x7f72a00046d0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:08:16.916 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:16.915+0000 7f72a5ffb640 1 -- 172.21.15.104:0/3078379517 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179117 (secure 0 0 0) 0x7f7298056110 con 0x7f72a813b3b0 2024-03-28T14:08:17.077 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:17.075+0000 7f72af477640 1 -- 172.21.15.104:0/3078379517 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f72a80b6780 con 0x7f72a813b3b0 2024-03-28T14:08:17.077 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:17.075+0000 7f72a5ffb640 1 -- 172.21.15.104:0/3078379517 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v21) ==== 104+0+11 (secure 0 0 0) 0x7f729805a410 con 0x7f72a813b3b0 2024-03-28T14:08:17.078 INFO:teuthology.orchestra.run.smithi104.stdout:pg_num: 32 2024-03-28T14:08:17.080 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:17.079+0000 7f72af477640 1 -- 172.21.15.104:0/3078379517 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f7294067fc0 msgr2=0x7f729406a480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:17.080 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:17.079+0000 7f72af477640 1 --2- 172.21.15.104:0/3078379517 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f7294067fc0 0x7f729406a480 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f72a00ad000 tx=0x7f72a00046d0 comp rx=0 tx=0).stop 2024-03-28T14:08:17.080 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:17.079+0000 7f72af477640 1 -- 172.21.15.104:0/3078379517 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f72a813b3b0 msgr2=0x7f72a8146ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:17.080 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:17.079+0000 7f72af477640 1 --2- 172.21.15.104:0/3078379517 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f72a813b3b0 0x7f72a8146ed0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f72a80b7ba0 tx=0x7f729800d8e0 comp rx=0 tx=0).stop 2024-03-28T14:08:17.080 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:17.079+0000 7f72af477640 1 -- 172.21.15.104:0/3078379517 shutdown_connections 2024-03-28T14:08:17.080 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:17.079+0000 7f72af477640 1 --2- 172.21.15.104:0/3078379517 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f7294067fc0 0x7f729406a480 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:17.080 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:17.079+0000 7f72af477640 1 --2- 172.21.15.104:0/3078379517 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f72a813b3b0 0x7f72a8146ed0 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:17.080 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:17.079+0000 7f72af477640 1 --2- 172.21.15.104:0/3078379517 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f72a81e48a0 0x7f72a8070cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:17.081 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:17.079+0000 7f72af477640 1 --2- 172.21.15.104:0/3078379517 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f72900039c0 0x7f72a80b5350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:17.081 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:17.079+0000 7f72af477640 1 -- 172.21.15.104:0/3078379517 >> 172.21.15.104:0/3078379517 conn(0x7f72a80ba400 msgr2=0x7f72a813e470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T14:08:17.081 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:17.079+0000 7f72af477640 1 -- 172.21.15.104:0/3078379517 shutdown_connections 2024-03-28T14:08:17.081 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:17.079+0000 7f72af477640 1 -- 172.21.15.104:0/3078379517 wait complete. 2024-03-28T14:08:17.095 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-28T14:08:17.211 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:17.207+0000 7f5017a96640 1 Processor -- start 2024-03-28T14:08:17.211 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:17.207+0000 7f5017a96640 1 -- start start 2024-03-28T14:08:17.212 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:17.211+0000 7f5017a96640 1 --2- >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f501002c460 0x7f501014afd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:17.212 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:17.211+0000 7f5017a96640 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f5010148110 0x7f50101485a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:17.213 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:17.211+0000 7f5017a96640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f501002c840 con 0x7f50100b5570 2024-03-28T14:08:17.213 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:17.211+0000 7f5017a96640 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 -- 0x7f50100b4850 con 0x7f5010148110 2024-03-28T14:08:17.213 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:17.211+0000 7f5017a96640 1 -- --> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] -- mon_getmap magic: 0 -- 0x7f50100b4d70 con 0x7f501002c460 2024-03-28T14:08:17.213 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:17.211+0000 7f5016033640 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f5010148110 0x7f50101485a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:17.213 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:17.211+0000 7f5015031640 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f50100b5570 0x7f50100b3ff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:46104/0 (socket says 172.21.15.104:46104) 2024-03-28T14:08:17.213 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:17.211+0000 7f5015031640 1 -- 172.21.15.104:0/4018496738 learned_addr learned my addr 172.21.15.104:0/4018496738 (peer_addr_for_me v1:172.21.15.104:0/0) 2024-03-28T14:08:17.213 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:17.211+0000 7f5015832640 1 --2- 172.21.15.104:0/4018496738 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f501002c460 0x7f501014afd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:17.213 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:17.211+0000 7f5016033640 1 -- 172.21.15.104:0/4018496738 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f501002c460 msgr2=0x7f501014afd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T14:08:17.213 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:17.211+0000 7f5016033640 1 --2- 172.21.15.104:0/4018496738 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f501002c460 0x7f501014afd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:17.213 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:17.211+0000 7f5016033640 1 -- 172.21.15.104:0/4018496738 >> v1:172.21.15.104:6789/0 conn(0x7f50100b5570 legacy=0x7f50100b3ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:17.213 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:17.211+0000 7f5016033640 1 -- 172.21.15.104:0/4018496738 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f50100b5050 con 0x7f5010148110 2024-03-28T14:08:17.213 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:17.211+0000 7f5014830640 1 -- 172.21.15.104:0/4018496738 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4220982157 0 0) 0x7f501002c840 con 0x7f50100b5570 2024-03-28T14:08:17.213 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:17.211+0000 7f5015832640 1 --2- 172.21.15.104:0/4018496738 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f501002c460 0x7f501014afd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T14:08:17.215 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:17.211+0000 7f5016033640 1 --2- 172.21.15.104:0/4018496738 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5010148110 0x7f50101485a0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f5008009d90 tx=0x7f500801c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=64365c9f491dd1ff server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:08:17.215 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:17.211+0000 7f5014830640 1 -- 172.21.15.104:0/4018496738 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f50080b2070 con 0x7f5010148110 2024-03-28T14:08:17.215 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:17.211+0000 7f5014830640 1 -- 172.21.15.104:0/4018496738 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f50080a59c0 con 0x7f5010148110 2024-03-28T14:08:17.215 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:17.211+0000 7f5014830640 1 -- 172.21.15.104:0/4018496738 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f50080a5d90 con 0x7f5010148110 2024-03-28T14:08:17.215 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:17.211+0000 7f5017a96640 1 -- 172.21.15.104:0/4018496738 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5010148110 msgr2=0x7f50101485a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:17.215 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:17.211+0000 7f5017a96640 1 --2- 172.21.15.104:0/4018496738 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5010148110 0x7f50101485a0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f5008009d90 tx=0x7f500801c8e0 comp rx=0 tx=0).stop 2024-03-28T14:08:17.215 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:17.211+0000 7f5017a96640 1 -- 172.21.15.104:0/4018496738 shutdown_connections 2024-03-28T14:08:17.215 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:17.211+0000 7f5017a96640 1 --2- 172.21.15.104:0/4018496738 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5010148110 0x7f50101485a0 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:17.215 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:17.211+0000 7f5017a96640 1 --2- 172.21.15.104:0/4018496738 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f501002c460 0x7f501014afd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:17.215 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:17.211+0000 7f5017a96640 1 -- 172.21.15.104:0/4018496738 >> 172.21.15.104:0/4018496738 conn(0x7f50100b95f0 msgr2=0x7f50100b99d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T14:08:17.215 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:17.211+0000 7f5017a96640 1 -- 172.21.15.104:0/4018496738 shutdown_connections 2024-03-28T14:08:17.215 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:17.211+0000 7f5017a96640 1 -- 172.21.15.104:0/4018496738 wait complete. 2024-03-28T14:08:17.217 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:17.215+0000 7f5017a96640 1 Processor -- start 2024-03-28T14:08:17.218 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:17.215+0000 7f5017a96640 1 -- start start 2024-03-28T14:08:17.218 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:17.215+0000 7f5017a96640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f501002c460 0x7f5010146e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:17.218 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:17.215+0000 7f5017a96640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f50100b5570 0x7f5010145100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:17.218 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:17.215+0000 7f5017a96640 1 --2- >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f5010148110 0x7f501009c490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:17.218 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:17.215+0000 7f5017a96640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f5010154ca0 con 0x7f50100b5570 2024-03-28T14:08:17.218 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:17.215+0000 7f5017a96640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f5010154b20 con 0x7f501002c460 2024-03-28T14:08:17.218 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:17.215+0000 7f5017a96640 1 -- --> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] -- mon_getmap magic: 0 -- 0x7f5010154e20 con 0x7f5010148110 2024-03-28T14:08:17.218 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:17.215+0000 7f5015031640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f50100b5570 0x7f5010145100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:17.218 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:17.215+0000 7f5015031640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f50100b5570 0x7f5010145100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:42344/0 (socket says 172.21.15.104:42344) 2024-03-28T14:08:17.218 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:17.215+0000 7f5015031640 1 -- 172.21.15.104:0/835191098 learned_addr learned my addr 172.21.15.104:0/835191098 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-03-28T14:08:17.218 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:17.215+0000 7f5016033640 1 --2- 172.21.15.104:0/835191098 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f5010148110 0x7f501009c490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:17.218 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:17.215+0000 7f5015031640 1 -- 172.21.15.104:0/835191098 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f5010148110 msgr2=0x7f501009c490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:17.218 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:17.215+0000 7f5015031640 1 --2- 172.21.15.104:0/835191098 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f5010148110 0x7f501009c490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:17.219 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:17.215+0000 7f5015031640 1 -- 172.21.15.104:0/835191098 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f501002c460 msgr2=0x7f5010146e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:17.219 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:17.215+0000 7f5015031640 1 --2- 172.21.15.104:0/835191098 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f501002c460 0x7f5010146e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:17.219 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:17.215+0000 7f5015031640 1 -- 172.21.15.104:0/835191098 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f501009e3b0 con 0x7f50100b5570 2024-03-28T14:08:17.219 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:17.215+0000 7f5015031640 1 --2- 172.21.15.104:0/835191098 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f50100b5570 0x7f5010145100 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f500c00c520 tx=0x7f500c00c9e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:08:17.219 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:17.215+0000 7f50067fc640 1 -- 172.21.15.104:0/835191098 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f500c018070 con 0x7f50100b5570 2024-03-28T14:08:17.219 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:17.215+0000 7f50067fc640 1 -- 172.21.15.104:0/835191098 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f500c0074e0 con 0x7f50100b5570 2024-03-28T14:08:17.219 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:17.215+0000 7f5017a96640 1 -- 172.21.15.104:0/835191098 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5010146900 con 0x7f50100b5570 2024-03-28T14:08:17.219 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:17.215+0000 7f50067fc640 1 -- 172.21.15.104:0/835191098 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f500c0049f0 con 0x7f50100b5570 2024-03-28T14:08:17.219 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:17.215+0000 7f5017a96640 1 -- 172.21.15.104:0/835191098 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5010145ee0 con 0x7f50100b5570 2024-03-28T14:08:17.219 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:17.215+0000 7f5017a96640 1 -- 172.21.15.104:0/835191098 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5010146170 con 0x7f50100b5570 2024-03-28T14:08:17.220 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:17.215+0000 7f50067fc640 1 -- 172.21.15.104:0/835191098 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) ==== 87803+0+0 (secure 0 0 0) 0x7f500c01b070 con 0x7f50100b5570 2024-03-28T14:08:17.220 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:17.215+0000 7f50067fc640 1 --2- 172.21.15.104:0/835191098 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f4ff4067fc0 0x7f4ff406a480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:17.220 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:17.215+0000 7f50067fc640 1 -- 172.21.15.104:0/835191098 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f500c0858f0 con 0x7f50100b5570 2024-03-28T14:08:17.220 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:17.215+0000 7f5015832640 1 --2- 172.21.15.104:0/835191098 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f4ff4067fc0 0x7f4ff406a480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:17.220 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:17.219+0000 7f5015832640 1 --2- 172.21.15.104:0/835191098 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f4ff4067fc0 0x7f4ff406a480 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f501009dc80 tx=0x7f5000008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:08:17.228 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:17.227+0000 7f50067fc640 1 -- 172.21.15.104:0/835191098 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179117 (secure 0 0 0) 0x7f5010146170 con 0x7f50100b5570 2024-03-28T14:08:17.380 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:17.379+0000 7f5017a96640 1 -- 172.21.15.104:0/835191098 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f50100687c0 con 0x7f50100b5570 2024-03-28T14:08:17.380 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:17.379+0000 7f50067fc640 1 -- 172.21.15.104:0/835191098 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v21) ==== 100+0+11 (secure 0 0 0) 0x7f500c056130 con 0x7f50100b5570 2024-03-28T14:08:17.381 INFO:teuthology.orchestra.run.smithi104.stdout:pg_num: 64 2024-03-28T14:08:17.383 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:17.379+0000 7f5017a96640 1 -- 172.21.15.104:0/835191098 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f4ff4067fc0 msgr2=0x7f4ff406a480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:17.383 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:17.379+0000 7f5017a96640 1 --2- 172.21.15.104:0/835191098 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f4ff4067fc0 0x7f4ff406a480 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f501009dc80 tx=0x7f5000008040 comp rx=0 tx=0).stop 2024-03-28T14:08:17.383 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:17.379+0000 7f5017a96640 1 -- 172.21.15.104:0/835191098 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f50100b5570 msgr2=0x7f5010145100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:17.383 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:17.379+0000 7f5017a96640 1 --2- 172.21.15.104:0/835191098 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f50100b5570 0x7f5010145100 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f500c00c520 tx=0x7f500c00c9e0 comp rx=0 tx=0).stop 2024-03-28T14:08:17.383 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:17.379+0000 7f5017a96640 1 -- 172.21.15.104:0/835191098 shutdown_connections 2024-03-28T14:08:17.383 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:17.379+0000 7f5017a96640 1 --2- 172.21.15.104:0/835191098 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f5010148110 0x7f501009c490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:17.383 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:17.379+0000 7f5017a96640 1 --2- 172.21.15.104:0/835191098 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f4ff4067fc0 0x7f4ff406a480 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:17.383 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:17.379+0000 7f5017a96640 1 --2- 172.21.15.104:0/835191098 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f50100b5570 0x7f5010145100 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:17.383 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:17.379+0000 7f5017a96640 1 --2- 172.21.15.104:0/835191098 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f501002c460 0x7f5010146e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:17.383 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:17.379+0000 7f5017a96640 1 -- 172.21.15.104:0/835191098 >> 172.21.15.104:0/835191098 conn(0x7f50100b95f0 msgr2=0x7f50100b8840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T14:08:17.384 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:17.379+0000 7f5017a96640 1 -- 172.21.15.104:0/835191098 shutdown_connections 2024-03-28T14:08:17.384 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:17.379+0000 7f5017a96640 1 -- 172.21.15.104:0/835191098 wait complete. 2024-03-28T14:08:17.398 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-28T14:08:17.519 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:17.515+0000 7fd7d3277640 1 Processor -- start 2024-03-28T14:08:17.519 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:17.515+0000 7fd7d3277640 1 -- start start 2024-03-28T14:08:17.521 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:17.519+0000 7fd7d3277640 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fd7cc072bf0 0x7fd7cc0b1e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:17.521 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:17.519+0000 7fd7d3277640 1 --2- >> v2:172.21.15.171:3300/0 conn(0x7fd7cc071be0 0x7fd7cc0b12f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:17.521 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:17.519+0000 7fd7d3277640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd7cc072fd0 con 0x7fd7cc0b1830 2024-03-28T14:08:17.521 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:17.519+0000 7fd7d3277640 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 -- 0x7fd7cc073250 con 0x7fd7cc072bf0 2024-03-28T14:08:17.521 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:17.519+0000 7fd7d3277640 1 -- --> v2:172.21.15.171:3300/0 -- mon_getmap magic: 0 -- 0x7fd7cc072050 con 0x7fd7cc071be0 2024-03-28T14:08:17.521 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:17.519+0000 7fd7d1013640 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fd7cc072bf0 0x7fd7cc0b1e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:17.521 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:17.519+0000 7fd7d1814640 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7fd7cc0b1830 0x7fd7cc145ce0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:46114/0 (socket says 172.21.15.104:46114) 2024-03-28T14:08:17.521 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:17.519+0000 7fd7d1814640 1 -- 172.21.15.104:0/1960751586 learned_addr learned my addr 172.21.15.104:0/1960751586 (peer_addr_for_me v1:172.21.15.104:0/0) 2024-03-28T14:08:17.521 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:17.519+0000 7fd7d0812640 1 --2- 172.21.15.104:0/1960751586 >> v2:172.21.15.171:3300/0 conn(0x7fd7cc071be0 0x7fd7cc0b12f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:17.522 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:17.519+0000 7fd7d1013640 1 -- 172.21.15.104:0/1960751586 >> v2:172.21.15.171:3300/0 conn(0x7fd7cc071be0 msgr2=0x7fd7cc0b12f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T14:08:17.522 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:17.519+0000 7fd7d1013640 1 --2- 172.21.15.104:0/1960751586 >> v2:172.21.15.171:3300/0 conn(0x7fd7cc071be0 0x7fd7cc0b12f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:17.522 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:17.519+0000 7fd7d1013640 1 -- 172.21.15.104:0/1960751586 >> v1:172.21.15.104:6789/0 conn(0x7fd7cc0b1830 legacy=0x7fd7cc145ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:17.522 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:17.519+0000 7fd7bffff640 1 -- 172.21.15.104:0/1960751586 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 24323236 0 0) 0x7fd7cc072fd0 con 0x7fd7cc0b1830 2024-03-28T14:08:17.522 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:17.519+0000 7fd7d1013640 1 -- 172.21.15.104:0/1960751586 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd7cc0734a0 con 0x7fd7cc072bf0 2024-03-28T14:08:17.523 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:17.523+0000 7fd7d1013640 1 --2- 172.21.15.104:0/1960751586 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd7cc072bf0 0x7fd7cc0b1e80 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fd7cc0b7770 tx=0x7fd7c401c810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=96cf9c4154fb15ac server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:08:17.524 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:17.523+0000 7fd7bffff640 1 -- 172.21.15.104:0/1960751586 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7c40b2070 con 0x7fd7cc072bf0 2024-03-28T14:08:17.524 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:17.523+0000 7fd7bffff640 1 -- 172.21.15.104:0/1960751586 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd7c40a5760 con 0x7fd7cc072bf0 2024-03-28T14:08:17.524 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:17.523+0000 7fd7bffff640 1 -- 172.21.15.104:0/1960751586 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7c40a5b30 con 0x7fd7cc072bf0 2024-03-28T14:08:17.524 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:17.523+0000 7fd7d3277640 1 -- 172.21.15.104:0/1960751586 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd7cc072bf0 msgr2=0x7fd7cc0b1e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:17.525 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:17.523+0000 7fd7d3277640 1 --2- 172.21.15.104:0/1960751586 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd7cc072bf0 0x7fd7cc0b1e80 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fd7cc0b7770 tx=0x7fd7c401c810 comp rx=0 tx=0).stop 2024-03-28T14:08:17.525 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:17.523+0000 7fd7d3277640 1 -- 172.21.15.104:0/1960751586 shutdown_connections 2024-03-28T14:08:17.525 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:17.523+0000 7fd7d3277640 1 --2- 172.21.15.104:0/1960751586 >> v2:172.21.15.171:3300/0 conn(0x7fd7cc071be0 0x7fd7cc0b12f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:17.525 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:17.523+0000 7fd7d3277640 1 --2- 172.21.15.104:0/1960751586 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd7cc072bf0 0x7fd7cc0b1e80 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:17.525 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:17.523+0000 7fd7d3277640 1 -- 172.21.15.104:0/1960751586 >> 172.21.15.104:0/1960751586 conn(0x7fd7cc0b7b10 msgr2=0x7fd7cc0b7f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T14:08:17.525 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:17.523+0000 7fd7d3277640 1 -- 172.21.15.104:0/1960751586 shutdown_connections 2024-03-28T14:08:17.525 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:17.523+0000 7fd7d3277640 1 -- 172.21.15.104:0/1960751586 wait complete. 2024-03-28T14:08:17.525 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:17.523+0000 7fd7d3277640 1 Processor -- start 2024-03-28T14:08:17.526 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:17.523+0000 7fd7d3277640 1 -- start start 2024-03-28T14:08:17.528 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:17.523+0000 7fd7d3277640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fd7cc071be0 0x7fd7cc1429a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:17.528 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:17.523+0000 7fd7d3277640 1 --2- >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fd7cc072bf0 0x7fd7cc09d030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:17.528 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:17.523+0000 7fd7d3277640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd7cc0b1830 0x7fd7cc09aa10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:17.528 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:17.523+0000 7fd7d3277640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7fd7cc152830 con 0x7fd7cc0b1830 2024-03-28T14:08:17.528 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:17.523+0000 7fd7d3277640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7fd7cc1526b0 con 0x7fd7cc071be0 2024-03-28T14:08:17.528 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:17.523+0000 7fd7d3277640 1 -- --> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] -- mon_getmap magic: 0 -- 0x7fd7cc1529b0 con 0x7fd7cc072bf0 2024-03-28T14:08:17.528 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:17.523+0000 7fd7d1814640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd7cc0b1830 0x7fd7cc09aa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:17.528 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:17.523+0000 7fd7d1814640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd7cc0b1830 0x7fd7cc09aa10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:42362/0 (socket says 172.21.15.104:42362) 2024-03-28T14:08:17.528 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:17.523+0000 7fd7d1814640 1 -- 172.21.15.104:0/2803492429 learned_addr learned my addr 172.21.15.104:0/2803492429 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-03-28T14:08:17.528 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:17.523+0000 7fd7d0812640 1 --2- 172.21.15.104:0/2803492429 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fd7cc072bf0 0x7fd7cc09d030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:17.528 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:17.523+0000 7fd7d1814640 1 -- 172.21.15.104:0/2803492429 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fd7cc072bf0 msgr2=0x7fd7cc09d030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:17.529 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:17.523+0000 7fd7d1814640 1 --2- 172.21.15.104:0/2803492429 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fd7cc072bf0 0x7fd7cc09d030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:17.529 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:17.523+0000 7fd7d1814640 1 -- 172.21.15.104:0/2803492429 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fd7cc071be0 msgr2=0x7fd7cc1429a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:17.529 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:17.523+0000 7fd7d1814640 1 --2- 172.21.15.104:0/2803492429 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fd7cc071be0 0x7fd7cc1429a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:17.529 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:17.523+0000 7fd7d1814640 1 -- 172.21.15.104:0/2803492429 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd7cc09cb50 con 0x7fd7cc0b1830 2024-03-28T14:08:17.529 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:17.523+0000 7fd7d0812640 1 --2- 172.21.15.104:0/2803492429 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fd7cc072bf0 0x7fd7cc09d030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T14:08:17.529 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:17.523+0000 7fd7d1814640 1 --2- 172.21.15.104:0/2803492429 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd7cc0b1830 0x7fd7cc09aa10 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fd7c800c390 tx=0x7fd7c800c850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:08:17.529 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:17.523+0000 7fd7bdffb640 1 -- 172.21.15.104:0/2803492429 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7c80074e0 con 0x7fd7cc0b1830 2024-03-28T14:08:17.529 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:17.523+0000 7fd7bdffb640 1 -- 172.21.15.104:0/2803492429 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd7c80047f0 con 0x7fd7cc0b1830 2024-03-28T14:08:17.529 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:17.523+0000 7fd7d3277640 1 -- 172.21.15.104:0/2803492429 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd7cc09f940 con 0x7fd7cc0b1830 2024-03-28T14:08:17.529 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:17.523+0000 7fd7bdffb640 1 -- 172.21.15.104:0/2803492429 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7c8007680 con 0x7fd7cc0b1830 2024-03-28T14:08:17.529 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:17.523+0000 7fd7d3277640 1 -- 172.21.15.104:0/2803492429 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd7cc09bed0 con 0x7fd7cc0b1830 2024-03-28T14:08:17.529 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:17.527+0000 7fd7d3277640 1 -- 172.21.15.104:0/2803492429 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd7cc0687c0 con 0x7fd7cc0b1830 2024-03-28T14:08:17.530 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:17.527+0000 7fd7bdffb640 1 -- 172.21.15.104:0/2803492429 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) ==== 87803+0+0 (secure 0 0 0) 0x7fd7c8011070 con 0x7fd7cc0b1830 2024-03-28T14:08:17.530 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:17.527+0000 7fd7bdffb640 1 --2- 172.21.15.104:0/2803492429 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7fd7b0067ff0 0x7fd7b006a4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:17.530 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:17.527+0000 7fd7bdffb640 1 -- 172.21.15.104:0/2803492429 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7fd7c8085fa0 con 0x7fd7cc0b1830 2024-03-28T14:08:17.531 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:17.527+0000 7fd7d1013640 1 --2- 172.21.15.104:0/2803492429 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7fd7b0067ff0 0x7fd7b006a4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:17.531 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:17.527+0000 7fd7d1013640 1 --2- 172.21.15.104:0/2803492429 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7fd7b0067ff0 0x7fd7b006a4b0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fd7c40af000 tx=0x7fd7c4004ac0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:08:17.537 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:17.535+0000 7fd7bdffb640 1 -- 172.21.15.104:0/2803492429 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179117 (secure 0 0 0) 0x7fd7c80567e0 con 0x7fd7cc0b1830 2024-03-28T14:08:17.687 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:17.683+0000 7fd7d3277640 1 -- 172.21.15.104:0/2803492429 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fd7cc09cd20 con 0x7fd7cc0b1830 2024-03-28T14:08:17.689 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:17.687+0000 7fd7bdffb640 1 -- 172.21.15.104:0/2803492429 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) ==== 74+0+16629 (secure 0 0 0) 0x7fd7c80560e0 con 0x7fd7cc0b1830 2024-03-28T14:08:17.689 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-03-28T14:08:17.689 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":21,"fsid":"428a90f2-3351-435f-8fcf-f7e66c78a13e","created":"2024-03-28T14:07:34.862458+0000","modified":"2024-03-28T14:08:00.671587+0000","last_up_change":"2024-03-28T14:07:46.028122+0000","last_in_change":"2024-03-28T14:07:37.157486+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-28T14:07:47.622988+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-28T14:07:57.147450+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-28T14:07:57.990975+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"1cb70540-4e42-402f-a446-aaec4ca83007","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6800","nonce":3745690918},{"type":"v1","addr":"172.21.15.104:6801","nonce":3745690918}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":3745690918},{"type":"v1","addr":"172.21.15.104:6803","nonce":3745690918}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":3745690918},{"type":"v1","addr":"172.21.15.104:6807","nonce":3745690918}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":3745690918},{"type":"v1","addr":"172.21.15.104:6805","nonce":3745690918}]},"public_addr":"172.21.15.104:6801/3745690918","cluster_addr":"172.21.15.104:6803/3745690918","heartbeat_back_addr":"172.21.15.104:6807/3745690918","heartbeat_front_addr":"172.21.15.104:6805/3745690918","state":["exists","up"]},{"osd":1,"uuid":"3ab7afdd-b980-418c-81f2-004e16de3dca","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":19438674},{"type":"v1","addr":"172.21.15.104:6810","nonce":19438674}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":19438674},{"type":"v1","addr":"172.21.15.104:6814","nonce":19438674}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":19438674},{"type":"v1","addr":"172.21.15.104:6823","nonce":19438674}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":19438674},{"type":"v1","addr":"172.21.15.104:6819","nonce":19438674}]},"public_addr":"172.21.15.104:6810/19438674","cluster_addr":"172.21.15.104:6814/19438674","heartbeat_back_addr":"172.21.15.104:6823/19438674","heartbeat_front_addr":"172.21.15.104:6819/19438674","state":["exists","up"]},{"osd":2,"uuid":"3e2651c8-ecae-4a9e-a053-27240a79eac2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":2352012626},{"type":"v1","addr":"172.21.15.104:6825","nonce":2352012626}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6826","nonce":2352012626},{"type":"v1","addr":"172.21.15.104:6827","nonce":2352012626}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":2352012626},{"type":"v1","addr":"172.21.15.104:6831","nonce":2352012626}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6828","nonce":2352012626},{"type":"v1","addr":"172.21.15.104:6829","nonce":2352012626}]},"public_addr":"172.21.15.104:6825/2352012626","cluster_addr":"172.21.15.104:6827/2352012626","heartbeat_back_addr":"172.21.15.104:6831/2352012626","heartbeat_front_addr":"172.21.15.104:6829/2352012626","state":["exists","up"]},{"osd":3,"uuid":"925c7e0a-38ef-47cd-b288-8bec8456fb17","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6809","nonce":3568347381},{"type":"v1","addr":"172.21.15.104:6811","nonce":3568347381}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6813","nonce":3568347381},{"type":"v1","addr":"172.21.15.104:6815","nonce":3568347381}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":3568347381},{"type":"v1","addr":"172.21.15.104:6822","nonce":3568347381}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6817","nonce":3568347381},{"type":"v1","addr":"172.21.15.104:6818","nonce":3568347381}]},"public_addr":"172.21.15.104:6811/3568347381","cluster_addr":"172.21.15.104:6815/3568347381","heartbeat_back_addr":"172.21.15.104:6822/3568347381","heartbeat_front_addr":"172.21.15.104:6818/3568347381","state":["exists","up"]},{"osd":4,"uuid":"4a8fac86-e595-4a0e-96e3-57edb3cc3f57","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6800","nonce":2107001077},{"type":"v1","addr":"172.21.15.171:6801","nonce":2107001077}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6802","nonce":2107001077},{"type":"v1","addr":"172.21.15.171:6803","nonce":2107001077}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6806","nonce":2107001077},{"type":"v1","addr":"172.21.15.171:6807","nonce":2107001077}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6804","nonce":2107001077},{"type":"v1","addr":"172.21.15.171:6805","nonce":2107001077}]},"public_addr":"172.21.15.171:6801/2107001077","cluster_addr":"172.21.15.171:6803/2107001077","heartbeat_back_addr":"172.21.15.171:6807/2107001077","heartbeat_front_addr":"172.21.15.171:6805/2107001077","state":["exists","up"]},{"osd":5,"uuid":"e48d50b0-beae-40ae-b15a-9a84dbfe88ab","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6808","nonce":2934826925},{"type":"v1","addr":"172.21.15.171:6810","nonce":2934826925}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6812","nonce":2934826925},{"type":"v1","addr":"172.21.15.171:6814","nonce":2934826925}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6819","nonce":2934826925},{"type":"v1","addr":"172.21.15.171:6821","nonce":2934826925}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6816","nonce":2934826925},{"type":"v1","addr":"172.21.15.171:6817","nonce":2934826925}]},"public_addr":"172.21.15.171:6810/2934826925","cluster_addr":"172.21.15.171:6814/2934826925","heartbeat_back_addr":"172.21.15.171:6821/2934826925","heartbeat_front_addr":"172.21.15.171:6817/2934826925","state":["exists","up"]},{"osd":6,"uuid":"5e7c3292-9936-4f35-b267-aed9d4e49323","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6824","nonce":436742919},{"type":"v1","addr":"172.21.15.171:6825","nonce":436742919}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6826","nonce":436742919},{"type":"v1","addr":"172.21.15.171:6827","nonce":436742919}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6830","nonce":436742919},{"type":"v1","addr":"172.21.15.171:6831","nonce":436742919}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6828","nonce":436742919},{"type":"v1","addr":"172.21.15.171:6829","nonce":436742919}]},"public_addr":"172.21.15.171:6825/436742919","cluster_addr":"172.21.15.171:6827/436742919","heartbeat_back_addr":"172.21.15.171:6831/436742919","heartbeat_front_addr":"172.21.15.171:6829/436742919","state":["exists","up"]},{"osd":7,"uuid":"339b8414-d7f0-4fb0-a446-efaf635cb7ff","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6809","nonce":2055631175},{"type":"v1","addr":"172.21.15.171:6811","nonce":2055631175}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6813","nonce":2055631175},{"type":"v1","addr":"172.21.15.171:6815","nonce":2055631175}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6822","nonce":2055631175},{"type":"v1","addr":"172.21.15.171:6823","nonce":2055631175}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6818","nonce":2055631175},{"type":"v1","addr":"172.21.15.171:6820","nonce":2055631175}]},"public_addr":"172.21.15.171:6811/2055631175","cluster_addr":"172.21.15.171:6815/2055631175","heartbeat_back_addr":"172.21.15.171:6823/2055631175","heartbeat_front_addr":"172.21.15.171:6820/2055631175","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T14:07:42.668881+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T14:07:42.708397+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T14:07:42.634191+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T14:07:42.733662+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T14:07:42.651541+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T14:07:42.783181+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T14:07:42.766310+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T14:07:42.670797+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-28T14:08:17.692 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:17.691+0000 7fd7d3277640 1 -- 172.21.15.104:0/2803492429 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7fd7b0067ff0 msgr2=0x7fd7b006a4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:17.692 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:17.691+0000 7fd7d3277640 1 --2- 172.21.15.104:0/2803492429 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7fd7b0067ff0 0x7fd7b006a4b0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fd7c40af000 tx=0x7fd7c4004ac0 comp rx=0 tx=0).stop 2024-03-28T14:08:17.693 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:17.691+0000 7fd7d3277640 1 -- 172.21.15.104:0/2803492429 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd7cc0b1830 msgr2=0x7fd7cc09aa10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:17.693 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:17.691+0000 7fd7d3277640 1 --2- 172.21.15.104:0/2803492429 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd7cc0b1830 0x7fd7cc09aa10 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fd7c800c390 tx=0x7fd7c800c850 comp rx=0 tx=0).stop 2024-03-28T14:08:17.693 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:17.691+0000 7fd7d3277640 1 -- 172.21.15.104:0/2803492429 shutdown_connections 2024-03-28T14:08:17.693 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:17.691+0000 7fd7d3277640 1 --2- 172.21.15.104:0/2803492429 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7fd7b0067ff0 0x7fd7b006a4b0 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:17.693 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:17.691+0000 7fd7d3277640 1 --2- 172.21.15.104:0/2803492429 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd7cc0b1830 0x7fd7cc09aa10 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:17.693 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:17.691+0000 7fd7d3277640 1 --2- 172.21.15.104:0/2803492429 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fd7cc072bf0 0x7fd7cc09d030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:17.693 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:17.691+0000 7fd7d3277640 1 --2- 172.21.15.104:0/2803492429 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fd7cc071be0 0x7fd7cc1429a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:17.693 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:17.691+0000 7fd7d3277640 1 -- 172.21.15.104:0/2803492429 >> 172.21.15.104:0/2803492429 conn(0x7fd7cc0b7b10 msgr2=0x7fd7cc0b9a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T14:08:17.693 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:17.691+0000 7fd7d3277640 1 -- 172.21.15.104:0/2803492429 shutdown_connections 2024-03-28T14:08:17.693 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:17.691+0000 7fd7d3277640 1 -- 172.21.15.104:0/2803492429 wait complete. 2024-03-28T14:08:17.715 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-28T14:07:47.622988+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-28T14:07:57.147450+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-28T14:07:57.990975+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-28T14:08:17.716 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-28T14:08:17.835 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:17.831+0000 7fd78d469640 1 Processor -- start 2024-03-28T14:08:17.835 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:17.831+0000 7fd78d469640 1 -- start start 2024-03-28T14:08:17.836 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:17.835+0000 7fd78d469640 1 --2- >> v2:172.21.15.171:3300/0 conn(0x7fd7880b0d90 0x7fd7880af020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:17.836 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:17.835+0000 7fd78d469640 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fd78806e020 0x7fd78814adb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:17.837 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:17.835+0000 7fd78d469640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd7880b1170 con 0x7fd7880b58b0 2024-03-28T14:08:17.837 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:17.835+0000 7fd78d469640 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 -- 0x7fd7880af560 con 0x7fd78806e020 2024-03-28T14:08:17.837 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:17.835+0000 7fd78d469640 1 -- --> v2:172.21.15.171:3300/0 -- mon_getmap magic: 0 -- 0x7fd78806db90 con 0x7fd7880b0d90 2024-03-28T14:08:17.837 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:17.835+0000 7fd7867fc640 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fd78806e020 0x7fd78814adb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:17.837 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:17.835+0000 7fd7867fc640 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fd78806e020 0x7fd78814adb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:42376/0 (socket says 172.21.15.104:42376) 2024-03-28T14:08:17.837 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:17.835+0000 7fd7867fc640 1 -- 172.21.15.104:0/1162339522 learned_addr learned my addr 172.21.15.104:0/1162339522 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-03-28T14:08:17.837 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:17.835+0000 7fd7867fc640 1 -- 172.21.15.104:0/1162339522 >> v2:172.21.15.171:3300/0 conn(0x7fd7880b0d90 msgr2=0x7fd7880af020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T14:08:17.837 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:17.835+0000 7fd786ffd640 1 --2- 172.21.15.104:0/1162339522 >> v2:172.21.15.171:3300/0 conn(0x7fd7880b0d90 0x7fd7880af020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:17.837 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:17.835+0000 7fd7867fc640 1 --2- 172.21.15.104:0/1162339522 >> v2:172.21.15.171:3300/0 conn(0x7fd7880b0d90 0x7fd7880af020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:17.837 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:17.835+0000 7fd7867fc640 1 -- 172.21.15.104:0/1162339522 >> v1:172.21.15.171:6789/0 conn(0x7fd7880b58b0 legacy=0x7fd788072d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T14:08:17.837 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:17.835+0000 7fd7867fc640 1 -- 172.21.15.104:0/1162339522 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd78814b2f0 con 0x7fd78806e020 2024-03-28T14:08:17.838 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:17.835+0000 7fd786ffd640 1 --2- 172.21.15.104:0/1162339522 >> v2:172.21.15.171:3300/0 conn(0x7fd7880b0d90 0x7fd7880af020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T14:08:17.838 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:17.835+0000 7fd7867fc640 1 --2- 172.21.15.104:0/1162339522 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd78806e020 0x7fd78814adb0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fd7880701c0 tx=0x7fd77001c3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=215d386658cf2501 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:08:17.838 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:17.835+0000 7fd785ffb640 1 -- 172.21.15.104:0/1162339522 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7700b0070 con 0x7fd78806e020 2024-03-28T14:08:17.838 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:17.835+0000 7fd785ffb640 1 -- 172.21.15.104:0/1162339522 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd7700ab600 con 0x7fd78806e020 2024-03-28T14:08:17.838 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:17.835+0000 7fd785ffb640 1 -- 172.21.15.104:0/1162339522 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7700ab9d0 con 0x7fd78806e020 2024-03-28T14:08:17.839 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:17.835+0000 7fd78d469640 1 -- 172.21.15.104:0/1162339522 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd78806e020 msgr2=0x7fd78814adb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:17.839 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:17.835+0000 7fd78d469640 1 --2- 172.21.15.104:0/1162339522 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd78806e020 0x7fd78814adb0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fd7880701c0 tx=0x7fd77001c3c0 comp rx=0 tx=0).stop 2024-03-28T14:08:17.840 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:17.835+0000 7fd78d469640 1 -- 172.21.15.104:0/1162339522 shutdown_connections 2024-03-28T14:08:17.840 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:17.835+0000 7fd78d469640 1 --2- 172.21.15.104:0/1162339522 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd78806e020 0x7fd78814adb0 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:17.840 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:17.835+0000 7fd78d469640 1 --2- 172.21.15.104:0/1162339522 >> v2:172.21.15.171:3300/0 conn(0x7fd7880b0d90 0x7fd7880af020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:17.840 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:17.835+0000 7fd78d469640 1 -- 172.21.15.104:0/1162339522 >> 172.21.15.104:0/1162339522 conn(0x7fd7880ba400 msgr2=0x7fd7880ba810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T14:08:17.840 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:17.835+0000 7fd78d469640 1 -- 172.21.15.104:0/1162339522 shutdown_connections 2024-03-28T14:08:17.840 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:17.835+0000 7fd78d469640 1 -- 172.21.15.104:0/1162339522 wait complete. 2024-03-28T14:08:17.842 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:17.839+0000 7fd78d469640 1 Processor -- start 2024-03-28T14:08:17.843 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:17.839+0000 7fd78d469640 1 -- start start 2024-03-28T14:08:17.843 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:17.839+0000 7fd78d469640 1 --2- >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fd78806e020 0x7fd788143610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:17.843 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:17.839+0000 7fd78d469640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fd7880b0d90 0x7fd7881449d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:17.843 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:17.839+0000 7fd78d469640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd7880b58b0 0x7fd78813f110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:17.843 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:17.839+0000 7fd78d469640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7fd788154900 con 0x7fd7880b58b0 2024-03-28T14:08:17.843 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:17.839+0000 7fd78d469640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7fd788073630 con 0x7fd7880b0d90 2024-03-28T14:08:17.843 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:17.839+0000 7fd78d469640 1 -- --> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] -- mon_getmap magic: 0 -- 0x7fd788154a80 con 0x7fd78806e020 2024-03-28T14:08:17.843 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:17.839+0000 7fd7877fe640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd7880b58b0 0x7fd78813f110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:17.843 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:17.839+0000 7fd7877fe640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd7880b58b0 0x7fd78813f110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:42392/0 (socket says 172.21.15.104:42392) 2024-03-28T14:08:17.843 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:17.839+0000 7fd7877fe640 1 -- 172.21.15.104:0/3145586147 learned_addr learned my addr 172.21.15.104:0/3145586147 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-03-28T14:08:17.843 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:17.839+0000 7fd7867fc640 1 --2- 172.21.15.104:0/3145586147 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fd7880b0d90 0x7fd7881449d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:17.844 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:17.839+0000 7fd786ffd640 1 --2- 172.21.15.104:0/3145586147 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fd78806e020 0x7fd788143610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:17.844 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:17.839+0000 7fd7877fe640 1 -- 172.21.15.104:0/3145586147 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fd78806e020 msgr2=0x7fd788143610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:17.844 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:17.839+0000 7fd7877fe640 1 --2- 172.21.15.104:0/3145586147 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fd78806e020 0x7fd788143610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:17.844 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:17.839+0000 7fd7877fe640 1 -- 172.21.15.104:0/3145586147 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fd7880b0d90 msgr2=0x7fd7881449d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:17.844 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:17.839+0000 7fd7877fe640 1 --2- 172.21.15.104:0/3145586147 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fd7880b0d90 0x7fd7881449d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:17.844 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:17.839+0000 7fd7877fe640 1 -- 172.21.15.104:0/3145586147 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd78813f650 con 0x7fd7880b58b0 2024-03-28T14:08:17.844 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:17.839+0000 7fd786ffd640 1 --2- 172.21.15.104:0/3145586147 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fd78806e020 0x7fd788143610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T14:08:17.844 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:17.839+0000 7fd7877fe640 1 --2- 172.21.15.104:0/3145586147 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd7880b58b0 0x7fd78813f110 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fd78000b9a0 tx=0x7fd78000be60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:08:17.844 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:17.839+0000 7fd77f7fe640 1 -- 172.21.15.104:0/3145586147 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd780019070 con 0x7fd7880b58b0 2024-03-28T14:08:17.844 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:17.839+0000 7fd77f7fe640 1 -- 172.21.15.104:0/3145586147 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd78000cba0 con 0x7fd7880b58b0 2024-03-28T14:08:17.844 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:17.839+0000 7fd78d469640 1 -- 172.21.15.104:0/3145586147 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd7881440f0 con 0x7fd7880b58b0 2024-03-28T14:08:17.844 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:17.839+0000 7fd77f7fe640 1 -- 172.21.15.104:0/3145586147 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd780014440 con 0x7fd7880b58b0 2024-03-28T14:08:17.844 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:17.839+0000 7fd78d469640 1 -- 172.21.15.104:0/3145586147 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd788144690 con 0x7fd7880b58b0 2024-03-28T14:08:17.844 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:17.839+0000 7fd78d469640 1 -- 172.21.15.104:0/3145586147 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd7880687c0 con 0x7fd7880b58b0 2024-03-28T14:08:17.845 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:17.843+0000 7fd77f7fe640 1 -- 172.21.15.104:0/3145586147 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) ==== 87803+0+0 (secure 0 0 0) 0x7fd78001c070 con 0x7fd7880b58b0 2024-03-28T14:08:17.845 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:17.843+0000 7fd77f7fe640 1 --2- 172.21.15.104:0/3145586147 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7fd768067fc0 0x7fd76806a480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:17.845 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:17.843+0000 7fd77f7fe640 1 -- 172.21.15.104:0/3145586147 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7fd780086270 con 0x7fd7880b58b0 2024-03-28T14:08:17.846 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:17.843+0000 7fd786ffd640 1 --2- 172.21.15.104:0/3145586147 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7fd768067fc0 0x7fd76806a480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:17.846 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:17.843+0000 7fd786ffd640 1 --2- 172.21.15.104:0/3145586147 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7fd768067fc0 0x7fd76806a480 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fd778008930 tx=0x7fd778008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:08:17.847 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:17.847+0000 7fd77f7fe640 1 -- 172.21.15.104:0/3145586147 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179117 (secure 0 0 0) 0x7fd780056ab0 con 0x7fd7880b58b0 2024-03-28T14:08:17.999 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:17.995+0000 7fd78d469640 1 -- 172.21.15.104:0/3145586147 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fd788067c20 con 0x7fd7880b58b0 2024-03-28T14:08:18.264 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:17.995+0000 7fd77f7fe640 1 -- 172.21.15.104:0/3145586147 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v21) ==== 93+0+10 (secure 0 0 0) 0x7fd7800563b0 con 0x7fd7880b58b0 2024-03-28T14:08:18.265 INFO:teuthology.orchestra.run.smithi104.stdout:pg_num: 1 2024-03-28T14:08:18.267 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:17.999+0000 7fd78d469640 1 -- 172.21.15.104:0/3145586147 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7fd768067fc0 msgr2=0x7fd76806a480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:18.267 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:17.999+0000 7fd78d469640 1 --2- 172.21.15.104:0/3145586147 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7fd768067fc0 0x7fd76806a480 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fd778008930 tx=0x7fd778008040 comp rx=0 tx=0).stop 2024-03-28T14:08:18.267 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:17.999+0000 7fd78d469640 1 -- 172.21.15.104:0/3145586147 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd7880b58b0 msgr2=0x7fd78813f110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:18.267 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:17.999+0000 7fd78d469640 1 --2- 172.21.15.104:0/3145586147 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd7880b58b0 0x7fd78813f110 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fd78000b9a0 tx=0x7fd78000be60 comp rx=0 tx=0).stop 2024-03-28T14:08:18.267 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:17.999+0000 7fd78d469640 1 -- 172.21.15.104:0/3145586147 shutdown_connections 2024-03-28T14:08:18.267 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:17.999+0000 7fd78d469640 1 --2- 172.21.15.104:0/3145586147 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7fd768067fc0 0x7fd76806a480 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:18.267 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:17.999+0000 7fd78d469640 1 --2- 172.21.15.104:0/3145586147 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd7880b58b0 0x7fd78813f110 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:18.267 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:17.999+0000 7fd78d469640 1 --2- 172.21.15.104:0/3145586147 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fd7880b0d90 0x7fd7881449d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:18.267 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:17.999+0000 7fd78d469640 1 --2- 172.21.15.104:0/3145586147 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fd78806e020 0x7fd788143610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:18.267 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:17.999+0000 7fd78d469640 1 -- 172.21.15.104:0/3145586147 >> 172.21.15.104:0/3145586147 conn(0x7fd7880ba400 msgr2=0x7fd78806bd20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T14:08:18.267 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:17.999+0000 7fd78d469640 1 -- 172.21.15.104:0/3145586147 shutdown_connections 2024-03-28T14:08:18.268 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:17.999+0000 7fd78d469640 1 -- 172.21.15.104:0/3145586147 wait complete. 2024-03-28T14:08:18.268 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-28T14:08:18.388 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:18.387+0000 7f5fc2fb5640 1 Processor -- start 2024-03-28T14:08:18.388 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:18.387+0000 7f5fc2fb5640 1 -- start start 2024-03-28T14:08:18.389 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:18.387+0000 7f5fc2fb5640 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f5fbc0af9b0 0x7f5fbc0b0010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:18.389 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:18.387+0000 7f5fc2fb5640 1 --2- >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f5fbc1445e0 0x7f5fbc1449c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:18.389 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:18.387+0000 7f5fc2fb5640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5fbc0b6780 con 0x7f5fbc0b63a0 2024-03-28T14:08:18.390 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:18.387+0000 7f5fc2fb5640 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 -- 0x7f5fbc0af610 con 0x7f5fbc0af9b0 2024-03-28T14:08:18.390 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:18.387+0000 7f5fc2fb5640 1 -- --> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] -- mon_getmap magic: 0 -- 0x7f5fbc06dde0 con 0x7f5fbc1445e0 2024-03-28T14:08:18.390 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:18.387+0000 7f5fbbfff640 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f5fbc0af9b0 0x7f5fbc0b0010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:18.390 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:18.387+0000 7f5fbbfff640 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f5fbc0af9b0 0x7f5fbc0b0010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:42394/0 (socket says 172.21.15.104:42394) 2024-03-28T14:08:18.390 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:18.387+0000 7f5fbbfff640 1 -- 172.21.15.104:0/3384134664 learned_addr learned my addr 172.21.15.104:0/3384134664 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-03-28T14:08:18.390 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:18.387+0000 7f5fc1552640 1 --2- 172.21.15.104:0/3384134664 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f5fbc1445e0 0x7f5fbc1449c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:18.390 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:18.387+0000 7f5fbbfff640 1 -- 172.21.15.104:0/3384134664 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f5fbc1445e0 msgr2=0x7f5fbc1449c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T14:08:18.390 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:18.387+0000 7f5fbbfff640 1 --2- 172.21.15.104:0/3384134664 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f5fbc1445e0 0x7f5fbc1449c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:18.390 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:18.387+0000 7f5fbbfff640 1 -- 172.21.15.104:0/3384134664 >> v1:172.21.15.104:6789/0 conn(0x7f5fbc0b63a0 legacy=0x7f5fbc141670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:18.391 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:18.387+0000 7f5fbb7fe640 1 -- 172.21.15.104:0/3384134664 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3071203659 0 0) 0x7f5fbc0b6780 con 0x7f5fbc0b63a0 2024-03-28T14:08:18.391 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:18.387+0000 7f5fbbfff640 1 -- 172.21.15.104:0/3384134664 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5fbc06e0c0 con 0x7f5fbc0af9b0 2024-03-28T14:08:18.391 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:18.387+0000 7f5fc1552640 1 --2- 172.21.15.104:0/3384134664 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f5fbc1445e0 0x7f5fbc1449c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T14:08:18.397 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:18.387+0000 7f5fbbfff640 1 --2- 172.21.15.104:0/3384134664 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5fbc0af9b0 0x7f5fbc0b0010 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f5fa800bb10 tx=0x7f5fa801eba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e497827e9b9630ae server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:08:18.397 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:18.391+0000 7f5fbb7fe640 1 -- 172.21.15.104:0/3384134664 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5fa80b4070 con 0x7f5fbc0af9b0 2024-03-28T14:08:18.397 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:18.391+0000 7f5fbb7fe640 1 -- 172.21.15.104:0/3384134664 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5fbc141f90 con 0x7f5fbc0af9b0 2024-03-28T14:08:18.397 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:18.391+0000 7f5fbb7fe640 1 -- 172.21.15.104:0/3384134664 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5fa80a7ca0 con 0x7f5fbc0af9b0 2024-03-28T14:08:18.398 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:18.391+0000 7f5fc2fb5640 1 -- 172.21.15.104:0/3384134664 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5fbc0af9b0 msgr2=0x7f5fbc0b0010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:18.398 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:18.391+0000 7f5fc2fb5640 1 --2- 172.21.15.104:0/3384134664 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5fbc0af9b0 0x7f5fbc0b0010 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f5fa800bb10 tx=0x7f5fa801eba0 comp rx=0 tx=0).stop 2024-03-28T14:08:18.398 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:18.391+0000 7f5fc2fb5640 1 -- 172.21.15.104:0/3384134664 shutdown_connections 2024-03-28T14:08:18.398 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:18.391+0000 7f5fc2fb5640 1 --2- 172.21.15.104:0/3384134664 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f5fbc1445e0 0x7f5fbc1449c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:18.398 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:18.391+0000 7f5fc2fb5640 1 --2- 172.21.15.104:0/3384134664 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5fbc0af9b0 0x7f5fbc0b0010 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:18.398 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:18.391+0000 7f5fc2fb5640 1 -- 172.21.15.104:0/3384134664 >> 172.21.15.104:0/3384134664 conn(0x7f5fbc0ba400 msgr2=0x7f5fbc0ba810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T14:08:18.398 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:18.391+0000 7f5fc2fb5640 1 -- 172.21.15.104:0/3384134664 shutdown_connections 2024-03-28T14:08:18.398 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:18.391+0000 7f5fc2fb5640 1 -- 172.21.15.104:0/3384134664 wait complete. 2024-03-28T14:08:18.398 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:18.391+0000 7f5fc2fb5640 1 Processor -- start 2024-03-28T14:08:18.398 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:18.391+0000 7f5fc2fb5640 1 -- start start 2024-03-28T14:08:18.399 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:18.391+0000 7f5fc2fb5640 1 --2- >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f5fbc0af9b0 0x7f5fbc09f8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:18.399 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:18.391+0000 7f5fc2fb5640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5fbc0b63a0 0x7f5fbc0b2400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:18.399 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:18.391+0000 7f5fc2fb5640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f5fbc1445e0 0x7f5fbc070aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:18.399 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:18.391+0000 7f5fc2fb5640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f5fbc06c170 con 0x7f5fbc0b63a0 2024-03-28T14:08:18.399 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:18.391+0000 7f5fc2fb5640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f5fbc06bff0 con 0x7f5fbc1445e0 2024-03-28T14:08:18.399 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:18.391+0000 7f5fc2fb5640 1 -- --> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] -- mon_getmap magic: 0 -- 0x7f5fbc06c2f0 con 0x7f5fbc0af9b0 2024-03-28T14:08:18.399 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:18.391+0000 7f5fbbfff640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5fbc0b63a0 0x7f5fbc0b2400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:18.400 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:18.391+0000 7f5fbbfff640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5fbc0b63a0 0x7f5fbc0b2400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:42410/0 (socket says 172.21.15.104:42410) 2024-03-28T14:08:18.400 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:18.391+0000 7f5fbbfff640 1 -- 172.21.15.104:0/3932991690 learned_addr learned my addr 172.21.15.104:0/3932991690 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-03-28T14:08:18.400 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:18.391+0000 7f5fbbfff640 1 -- 172.21.15.104:0/3932991690 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f5fbc0af9b0 msgr2=0x7f5fbc09f8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:18.400 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:18.391+0000 7f5fbbfff640 1 --2- 172.21.15.104:0/3932991690 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f5fbc0af9b0 0x7f5fbc09f8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:18.400 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:18.391+0000 7f5fbbfff640 1 -- 172.21.15.104:0/3932991690 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f5fbc1445e0 msgr2=0x7f5fbc070aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:18.400 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:18.391+0000 7f5fbbfff640 1 --2- 172.21.15.104:0/3932991690 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f5fbc1445e0 0x7f5fbc070aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:18.400 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:18.391+0000 7f5fbbfff640 1 -- 172.21.15.104:0/3932991690 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5fbc1e3220 con 0x7f5fbc0b63a0 2024-03-28T14:08:18.400 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:18.391+0000 7f5fbbfff640 1 --2- 172.21.15.104:0/3932991690 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5fbc0b63a0 0x7f5fbc0b2400 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f5fa800b2c0 tx=0x7f5fa801d3e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:08:18.400 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:18.391+0000 7f5fb97fa640 1 -- 172.21.15.104:0/3932991690 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5fa80bf070 con 0x7f5fbc0b63a0 2024-03-28T14:08:18.401 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:18.391+0000 7f5fb97fa640 1 -- 172.21.15.104:0/3932991690 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5fa8004230 con 0x7f5fbc0b63a0 2024-03-28T14:08:18.401 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:18.391+0000 7f5fc2fb5640 1 -- 172.21.15.104:0/3932991690 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5fbc0b3c80 con 0x7f5fbc0b63a0 2024-03-28T14:08:18.401 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:18.391+0000 7f5fb97fa640 1 -- 172.21.15.104:0/3932991690 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5fa80b4070 con 0x7f5fbc0b63a0 2024-03-28T14:08:18.401 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:18.391+0000 7f5fc2fb5640 1 -- 172.21.15.104:0/3932991690 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5fbc14eae0 con 0x7f5fbc0b63a0 2024-03-28T14:08:18.401 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:18.395+0000 7f5fc2fb5640 1 -- 172.21.15.104:0/3932991690 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5fbc0b1d80 con 0x7f5fbc0b63a0 2024-03-28T14:08:18.401 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:18.395+0000 7f5fb97fa640 1 -- 172.21.15.104:0/3932991690 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) ==== 87803+0+0 (secure 0 0 0) 0x7f5fa80c2070 con 0x7f5fbc0b63a0 2024-03-28T14:08:18.401 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:18.395+0000 7f5fb97fa640 1 --2- 172.21.15.104:0/3932991690 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f5fa0067fc0 0x7f5fa006a480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:18.402 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:18.395+0000 7f5fb97fa640 1 -- 172.21.15.104:0/3932991690 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f5fa8120a10 con 0x7f5fbc0b63a0 2024-03-28T14:08:18.402 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:18.395+0000 7f5fc0d51640 1 --2- 172.21.15.104:0/3932991690 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f5fa0067fc0 0x7f5fa006a480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:18.402 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:18.395+0000 7f5fc0d51640 1 --2- 172.21.15.104:0/3932991690 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f5fa0067fc0 0x7f5fa006a480 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f5fb0007fd0 tx=0x7f5fb0007920 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:08:18.407 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:18.403+0000 7f5fb97fa640 1 -- 172.21.15.104:0/3932991690 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179117 (secure 0 0 0) 0x7f5fbc0b1d80 con 0x7f5fbc0b63a0 2024-03-28T14:08:18.555 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:18.551+0000 7f5fc2fb5640 1 -- 172.21.15.104:0/3932991690 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f5fbc0687c0 con 0x7f5fbc0b63a0 2024-03-28T14:08:18.556 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:18.555+0000 7f5fb97fa640 1 -- 172.21.15.104:0/3932991690 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v21) ==== 104+0+11 (secure 0 0 0) 0x7f5fa8018820 con 0x7f5fbc0b63a0 2024-03-28T14:08:18.556 INFO:teuthology.orchestra.run.smithi104.stdout:pg_num: 32 2024-03-28T14:08:18.559 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:18.555+0000 7f5fc2fb5640 1 -- 172.21.15.104:0/3932991690 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f5fa0067fc0 msgr2=0x7f5fa006a480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:18.559 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:18.555+0000 7f5fc2fb5640 1 --2- 172.21.15.104:0/3932991690 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f5fa0067fc0 0x7f5fa006a480 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f5fb0007fd0 tx=0x7f5fb0007920 comp rx=0 tx=0).stop 2024-03-28T14:08:18.559 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:18.555+0000 7f5fc2fb5640 1 -- 172.21.15.104:0/3932991690 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5fbc0b63a0 msgr2=0x7f5fbc0b2400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:18.559 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:18.555+0000 7f5fc2fb5640 1 --2- 172.21.15.104:0/3932991690 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5fbc0b63a0 0x7f5fbc0b2400 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f5fa800b2c0 tx=0x7f5fa801d3e0 comp rx=0 tx=0).stop 2024-03-28T14:08:18.560 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:18.555+0000 7f5fc2fb5640 1 -- 172.21.15.104:0/3932991690 shutdown_connections 2024-03-28T14:08:18.560 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:18.555+0000 7f5fc2fb5640 1 --2- 172.21.15.104:0/3932991690 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f5fbc1445e0 0x7f5fbc070aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:18.560 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:18.555+0000 7f5fc2fb5640 1 --2- 172.21.15.104:0/3932991690 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f5fa0067fc0 0x7f5fa006a480 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:18.560 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:18.555+0000 7f5fc2fb5640 1 --2- 172.21.15.104:0/3932991690 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5fbc0b63a0 0x7f5fbc0b2400 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:18.560 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:18.555+0000 7f5fc2fb5640 1 --2- 172.21.15.104:0/3932991690 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f5fbc0af9b0 0x7f5fbc09f8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:18.561 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:18.555+0000 7f5fc2fb5640 1 -- 172.21.15.104:0/3932991690 >> 172.21.15.104:0/3932991690 conn(0x7f5fbc0ba400 msgr2=0x7f5fbc136ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T14:08:18.561 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:18.555+0000 7f5fc2fb5640 1 -- 172.21.15.104:0/3932991690 shutdown_connections 2024-03-28T14:08:18.561 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:18.555+0000 7f5fc2fb5640 1 -- 172.21.15.104:0/3932991690 wait complete. 2024-03-28T14:08:18.580 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-28T14:08:18.692 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:18.691+0000 7f5a47657640 1 Processor -- start 2024-03-28T14:08:18.692 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:18.691+0000 7f5a47657640 1 -- start start 2024-03-28T14:08:18.693 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:18.691+0000 7f5a47657640 1 --2- >> v2:172.21.15.171:3300/0 conn(0x7f5a4002c460 0x7f5a400b3ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:18.693 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:18.691+0000 7f5a47657640 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f5a400b4530 0x7f5a4013f540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:18.693 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:18.691+0000 7f5a47657640 1 --2- >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f5a4013fa80 0x7f5a401456c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:18.693 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:18.691+0000 7f5a47657640 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 -- 0x7f5a4002c840 con 0x7f5a400b4530 2024-03-28T14:08:18.693 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:18.691+0000 7f5a47657640 1 -- --> v2:172.21.15.171:3300/0 -- mon_getmap magic: 0 -- 0x7f5a400b4f20 con 0x7f5a4002c460 2024-03-28T14:08:18.693 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:18.691+0000 7f5a47657640 1 -- --> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] -- mon_getmap magic: 0 -- 0x7f5a400b5200 con 0x7f5a4013fa80 2024-03-28T14:08:18.693 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:18.691+0000 7f5a44bf2640 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f5a400b4530 0x7f5a4013f540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:18.693 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:18.691+0000 7f5a44bf2640 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f5a400b4530 0x7f5a4013f540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:42418/0 (socket says 172.21.15.104:42418) 2024-03-28T14:08:18.694 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:18.691+0000 7f5a44bf2640 1 -- 172.21.15.104:0/121815651 learned_addr learned my addr 172.21.15.104:0/121815651 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-03-28T14:08:18.694 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:18.691+0000 7f5a45bf4640 1 --2- 172.21.15.104:0/121815651 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f5a4013fa80 0x7f5a401456c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:18.694 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:18.691+0000 7f5a453f3640 1 --2- 172.21.15.104:0/121815651 >> v2:172.21.15.171:3300/0 conn(0x7f5a4002c460 0x7f5a400b3ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:18.694 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:18.691+0000 7f5a44bf2640 1 -- 172.21.15.104:0/121815651 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f5a4013fa80 msgr2=0x7f5a401456c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T14:08:18.694 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:18.691+0000 7f5a44bf2640 1 --2- 172.21.15.104:0/121815651 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f5a4013fa80 0x7f5a401456c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:18.694 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:18.691+0000 7f5a44bf2640 1 -- 172.21.15.104:0/121815651 >> v2:172.21.15.171:3300/0 conn(0x7f5a4002c460 msgr2=0x7f5a400b3ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T14:08:18.694 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:18.691+0000 7f5a44bf2640 1 --2- 172.21.15.104:0/121815651 >> v2:172.21.15.171:3300/0 conn(0x7f5a4002c460 0x7f5a400b3ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:18.694 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:18.691+0000 7f5a44bf2640 1 -- 172.21.15.104:0/121815651 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5a400b54e0 con 0x7f5a400b4530 2024-03-28T14:08:18.695 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:18.695+0000 7f5a44bf2640 1 --2- 172.21.15.104:0/121815651 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5a400b4530 0x7f5a4013f540 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f5a400b7190 tx=0x7f5a3001c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e0034dec71d80222 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:08:18.696 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:18.695+0000 7f5a37fff640 1 -- 172.21.15.104:0/121815651 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a300a7070 con 0x7f5a400b4530 2024-03-28T14:08:18.696 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:18.695+0000 7f5a37fff640 1 -- 172.21.15.104:0/121815651 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5a300b0070 con 0x7f5a400b4530 2024-03-28T14:08:18.696 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:18.695+0000 7f5a37fff640 1 -- 172.21.15.104:0/121815651 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a300ab890 con 0x7f5a400b4530 2024-03-28T14:08:18.696 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:18.695+0000 7f5a47657640 1 -- 172.21.15.104:0/121815651 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5a400b4530 msgr2=0x7f5a4013f540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:18.696 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:18.695+0000 7f5a47657640 1 --2- 172.21.15.104:0/121815651 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5a400b4530 0x7f5a4013f540 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f5a400b7190 tx=0x7f5a3001c490 comp rx=0 tx=0).stop 2024-03-28T14:08:18.696 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:18.695+0000 7f5a47657640 1 -- 172.21.15.104:0/121815651 shutdown_connections 2024-03-28T14:08:18.696 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:18.695+0000 7f5a47657640 1 --2- 172.21.15.104:0/121815651 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f5a4013fa80 0x7f5a401456c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:18.697 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:18.695+0000 7f5a47657640 1 --2- 172.21.15.104:0/121815651 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5a400b4530 0x7f5a4013f540 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:18.697 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:18.695+0000 7f5a47657640 1 --2- 172.21.15.104:0/121815651 >> v2:172.21.15.171:3300/0 conn(0x7f5a4002c460 0x7f5a400b3ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:18.697 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:18.695+0000 7f5a47657640 1 -- 172.21.15.104:0/121815651 >> 172.21.15.104:0/121815651 conn(0x7f5a400b95f0 msgr2=0x7f5a400b99d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T14:08:18.697 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:18.695+0000 7f5a47657640 1 -- 172.21.15.104:0/121815651 shutdown_connections 2024-03-28T14:08:18.697 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:18.695+0000 7f5a47657640 1 -- 172.21.15.104:0/121815651 wait complete. 2024-03-28T14:08:18.699 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:18.695+0000 7f5a47657640 1 Processor -- start 2024-03-28T14:08:18.699 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:18.695+0000 7f5a47657640 1 -- start start 2024-03-28T14:08:18.699 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:18.695+0000 7f5a47657640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f5a4002c460 0x7f5a401e15e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:18.701 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:18.695+0000 7f5a47657640 1 --2- >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f5a400b4530 0x7f5a401dde10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:18.701 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:18.695+0000 7f5a47657640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5a4013fa80 0x7f5a401de350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:18.702 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:18.695+0000 7f5a47657640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f5a40149810 con 0x7f5a4013fa80 2024-03-28T14:08:18.702 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:18.695+0000 7f5a47657640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f5a40149690 con 0x7f5a4002c460 2024-03-28T14:08:18.702 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:18.695+0000 7f5a47657640 1 -- --> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] -- mon_getmap magic: 0 -- 0x7f5a40149990 con 0x7f5a400b4530 2024-03-28T14:08:18.702 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:18.695+0000 7f5a45bf4640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5a4013fa80 0x7f5a401de350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:18.702 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:18.695+0000 7f5a45bf4640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5a4013fa80 0x7f5a401de350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:42426/0 (socket says 172.21.15.104:42426) 2024-03-28T14:08:18.702 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:18.695+0000 7f5a45bf4640 1 -- 172.21.15.104:0/3002383199 learned_addr learned my addr 172.21.15.104:0/3002383199 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-03-28T14:08:18.702 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:18.695+0000 7f5a453f3640 1 --2- 172.21.15.104:0/3002383199 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f5a4002c460 0x7f5a401e15e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:18.703 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:18.695+0000 7f5a44bf2640 1 --2- 172.21.15.104:0/3002383199 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f5a400b4530 0x7f5a401dde10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:18.703 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:18.695+0000 7f5a45bf4640 1 -- 172.21.15.104:0/3002383199 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f5a400b4530 msgr2=0x7f5a401dde10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:18.703 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:18.695+0000 7f5a45bf4640 1 --2- 172.21.15.104:0/3002383199 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f5a400b4530 0x7f5a401dde10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:18.703 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:18.695+0000 7f5a45bf4640 1 -- 172.21.15.104:0/3002383199 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f5a4002c460 msgr2=0x7f5a401e15e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:18.703 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:18.695+0000 7f5a45bf4640 1 --2- 172.21.15.104:0/3002383199 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f5a4002c460 0x7f5a401e15e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:18.703 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:18.695+0000 7f5a45bf4640 1 -- 172.21.15.104:0/3002383199 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5a40144d70 con 0x7f5a4013fa80 2024-03-28T14:08:18.703 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:18.695+0000 7f5a453f3640 1 --2- 172.21.15.104:0/3002383199 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f5a4002c460 0x7f5a401e15e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T14:08:18.703 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:18.695+0000 7f5a45bf4640 1 --2- 172.21.15.104:0/3002383199 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5a4013fa80 0x7f5a401de350 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f5a38007e60 tx=0x7f5a3800e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:08:18.703 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:18.695+0000 7f5a35ffb640 1 -- 172.21.15.104:0/3002383199 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a38014420 con 0x7f5a4013fa80 2024-03-28T14:08:18.703 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:18.695+0000 7f5a35ffb640 1 -- 172.21.15.104:0/3002383199 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5a38019070 con 0x7f5a4013fa80 2024-03-28T14:08:18.703 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:18.695+0000 7f5a47657640 1 -- 172.21.15.104:0/3002383199 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5a40073140 con 0x7f5a4013fa80 2024-03-28T14:08:18.703 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:18.695+0000 7f5a35ffb640 1 -- 172.21.15.104:0/3002383199 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a38014820 con 0x7f5a4013fa80 2024-03-28T14:08:18.703 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:18.695+0000 7f5a47657640 1 -- 172.21.15.104:0/3002383199 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5a40145230 con 0x7f5a4013fa80 2024-03-28T14:08:18.704 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:18.699+0000 7f5a47657640 1 -- 172.21.15.104:0/3002383199 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5a400687c0 con 0x7f5a4013fa80 2024-03-28T14:08:18.704 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:18.699+0000 7f5a35ffb640 1 -- 172.21.15.104:0/3002383199 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) ==== 87803+0+0 (secure 0 0 0) 0x7f5a380149c0 con 0x7f5a4013fa80 2024-03-28T14:08:18.705 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:18.699+0000 7f5a35ffb640 1 --2- 172.21.15.104:0/3002383199 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f5a24067fc0 0x7f5a2406a480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:18.705 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:18.699+0000 7f5a35ffb640 1 -- 172.21.15.104:0/3002383199 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f5a38086cb0 con 0x7f5a4013fa80 2024-03-28T14:08:18.705 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:18.699+0000 7f5a453f3640 1 --2- 172.21.15.104:0/3002383199 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f5a24067fc0 0x7f5a2406a480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:18.705 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:18.699+0000 7f5a453f3640 1 --2- 172.21.15.104:0/3002383199 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f5a24067fc0 0x7f5a2406a480 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f5a40148920 tx=0x7f5a3c008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:08:18.705 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:18.703+0000 7f5a35ffb640 1 -- 172.21.15.104:0/3002383199 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179117 (secure 0 0 0) 0x7f5a40073140 con 0x7f5a4013fa80 2024-03-28T14:08:18.860 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:18.855+0000 7f5a47657640 1 -- 172.21.15.104:0/3002383199 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f5a4006c930 con 0x7f5a4013fa80 2024-03-28T14:08:18.861 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:18.855+0000 7f5a35ffb640 1 -- 172.21.15.104:0/3002383199 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v21) ==== 100+0+11 (secure 0 0 0) 0x7f5a40144d70 con 0x7f5a4013fa80 2024-03-28T14:08:18.861 INFO:teuthology.orchestra.run.smithi104.stdout:pg_num: 64 2024-03-28T14:08:18.863 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:18.859+0000 7f5a47657640 1 -- 172.21.15.104:0/3002383199 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f5a24067fc0 msgr2=0x7f5a2406a480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:18.866 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:18.859+0000 7f5a47657640 1 --2- 172.21.15.104:0/3002383199 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f5a24067fc0 0x7f5a2406a480 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f5a40148920 tx=0x7f5a3c008040 comp rx=0 tx=0).stop 2024-03-28T14:08:18.866 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:18.859+0000 7f5a47657640 1 -- 172.21.15.104:0/3002383199 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5a4013fa80 msgr2=0x7f5a401de350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:18.866 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:18.859+0000 7f5a47657640 1 --2- 172.21.15.104:0/3002383199 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5a4013fa80 0x7f5a401de350 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f5a38007e60 tx=0x7f5a3800e4b0 comp rx=0 tx=0).stop 2024-03-28T14:08:18.866 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:18.859+0000 7f5a47657640 1 -- 172.21.15.104:0/3002383199 shutdown_connections 2024-03-28T14:08:18.866 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:18.859+0000 7f5a47657640 1 --2- 172.21.15.104:0/3002383199 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f5a24067fc0 0x7f5a2406a480 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:18.866 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:18.859+0000 7f5a47657640 1 --2- 172.21.15.104:0/3002383199 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5a4013fa80 0x7f5a401de350 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:18.866 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:18.859+0000 7f5a47657640 1 --2- 172.21.15.104:0/3002383199 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f5a400b4530 0x7f5a401dde10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:18.866 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:18.859+0000 7f5a47657640 1 --2- 172.21.15.104:0/3002383199 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f5a4002c460 0x7f5a401e15e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:18.867 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:18.859+0000 7f5a47657640 1 -- 172.21.15.104:0/3002383199 >> 172.21.15.104:0/3002383199 conn(0x7f5a400b95f0 msgr2=0x7f5a4006d000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T14:08:18.867 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:18.859+0000 7f5a47657640 1 -- 172.21.15.104:0/3002383199 shutdown_connections 2024-03-28T14:08:18.867 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:18.859+0000 7f5a47657640 1 -- 172.21.15.104:0/3002383199 wait complete. 2024-03-28T14:08:18.876 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-28T14:08:19.023 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:19.019+0000 7f96d1b02640 1 Processor -- start 2024-03-28T14:08:19.024 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:19.019+0000 7f96d1b02640 1 -- start start 2024-03-28T14:08:19.025 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:19.023+0000 7f96d1b02640 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f96cc02c460 0x7f96cc0b54e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:19.025 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:19.023+0000 7f96d1b02640 1 --2- >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f96cc1487e0 0x7f96cc0b5a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:19.025 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:19.023+0000 7f96d1b02640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f96cc02c840 con 0x7f96cc09e8e0 2024-03-28T14:08:19.025 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:19.023+0000 7f96d1b02640 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 -- 0x7f96cc147c40 con 0x7f96cc02c460 2024-03-28T14:08:19.025 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:19.023+0000 7f96d1b02640 1 -- --> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] -- mon_getmap magic: 0 -- 0x7f96cc0aec70 con 0x7f96cc1487e0 2024-03-28T14:08:19.025 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:19.023+0000 7f96cb7fe640 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f96cc02c460 0x7f96cc0b54e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:19.025 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:19.023+0000 7f96caffd640 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f96cc09e8e0 0x7f96cc1480d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:46128/0 (socket says 172.21.15.104:46128) 2024-03-28T14:08:19.025 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:19.023+0000 7f96caffd640 1 -- 172.21.15.104:0/378506162 learned_addr learned my addr 172.21.15.104:0/378506162 (peer_addr_for_me v1:172.21.15.104:0/0) 2024-03-28T14:08:19.025 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:19.023+0000 7f96cb7fe640 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f96cc02c460 0x7f96cc0b54e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:42430/0 (socket says 172.21.15.104:42430) 2024-03-28T14:08:19.025 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:19.023+0000 7f96cbfff640 1 --2- 172.21.15.104:0/378506162 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f96cc1487e0 0x7f96cc0b5a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:19.026 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:19.023+0000 7f96cb7fe640 1 -- 172.21.15.104:0/378506162 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f96cc1487e0 msgr2=0x7f96cc0b5a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T14:08:19.026 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:19.023+0000 7f96cb7fe640 1 --2- 172.21.15.104:0/378506162 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f96cc1487e0 0x7f96cc0b5a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:19.026 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:19.023+0000 7f96cb7fe640 1 -- 172.21.15.104:0/378506162 >> v1:172.21.15.104:6789/0 conn(0x7f96cc09e8e0 legacy=0x7f96cc1480d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:19.026 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:19.023+0000 7f96cb7fe640 1 -- 172.21.15.104:0/378506162 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f96cc0aef50 con 0x7f96cc02c460 2024-03-28T14:08:19.026 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:19.023+0000 7f96ca7fc640 1 -- 172.21.15.104:0/378506162 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 903446532 0 0) 0x7f96cc02c840 con 0x7f96cc09e8e0 2024-03-28T14:08:19.028 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:19.027+0000 7f96cb7fe640 1 --2- 172.21.15.104:0/378506162 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f96cc02c460 0x7f96cc0b54e0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f96b40097c0 tx=0x7f96b401c4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6a529574fe5bc820 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:08:19.028 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:19.027+0000 7f96ca7fc640 1 -- 172.21.15.104:0/378506162 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96b40b0070 con 0x7f96cc02c460 2024-03-28T14:08:19.028 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:19.027+0000 7f96ca7fc640 1 -- 172.21.15.104:0/378506162 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f96b40ab600 con 0x7f96cc02c460 2024-03-28T14:08:19.029 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:19.027+0000 7f96ca7fc640 1 -- 172.21.15.104:0/378506162 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96b40ab9d0 con 0x7f96cc02c460 2024-03-28T14:08:19.029 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:19.027+0000 7f96d1b02640 1 -- 172.21.15.104:0/378506162 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f96cc02c460 msgr2=0x7f96cc0b54e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:19.029 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:19.027+0000 7f96d1b02640 1 --2- 172.21.15.104:0/378506162 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f96cc02c460 0x7f96cc0b54e0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f96b40097c0 tx=0x7f96b401c4a0 comp rx=0 tx=0).stop 2024-03-28T14:08:19.029 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:19.027+0000 7f96d1b02640 1 -- 172.21.15.104:0/378506162 shutdown_connections 2024-03-28T14:08:19.030 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:19.027+0000 7f96d1b02640 1 --2- 172.21.15.104:0/378506162 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f96cc1487e0 0x7f96cc0b5a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:19.030 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:19.027+0000 7f96d1b02640 1 --2- 172.21.15.104:0/378506162 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f96cc02c460 0x7f96cc0b54e0 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:19.030 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:19.027+0000 7f96d1b02640 1 -- 172.21.15.104:0/378506162 >> 172.21.15.104:0/378506162 conn(0x7f96cc0b95f0 msgr2=0x7f96cc0b99d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T14:08:19.030 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:19.027+0000 7f96d1b02640 1 -- 172.21.15.104:0/378506162 shutdown_connections 2024-03-28T14:08:19.030 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:19.027+0000 7f96d1b02640 1 -- 172.21.15.104:0/378506162 wait complete. 2024-03-28T14:08:19.032 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:19.027+0000 7f96d1b02640 1 Processor -- start 2024-03-28T14:08:19.032 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:19.027+0000 7f96d1b02640 1 -- start start 2024-03-28T14:08:19.032 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:19.027+0000 7f96d1b02640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f96cc02c460 0x7f96cc1399c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:19.032 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:19.027+0000 7f96d1b02640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f96cc09e8e0 0x7f96cc09b560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:19.032 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:19.027+0000 7f96d1b02640 1 --2- >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f96cc1487e0 0x7f96cc136ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:19.032 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:19.027+0000 7f96d1b02640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f96cc14fb40 con 0x7f96cc09e8e0 2024-03-28T14:08:19.032 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:19.027+0000 7f96d1b02640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f96cc14f9c0 con 0x7f96cc02c460 2024-03-28T14:08:19.032 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:19.027+0000 7f96d1b02640 1 -- --> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] -- mon_getmap magic: 0 -- 0x7f96cc14fcc0 con 0x7f96cc1487e0 2024-03-28T14:08:19.032 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:19.027+0000 7f96caffd640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f96cc09e8e0 0x7f96cc09b560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:19.033 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:19.027+0000 7f96caffd640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f96cc09e8e0 0x7f96cc09b560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:42432/0 (socket says 172.21.15.104:42432) 2024-03-28T14:08:19.033 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:19.027+0000 7f96caffd640 1 -- 172.21.15.104:0/1530702365 learned_addr learned my addr 172.21.15.104:0/1530702365 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-03-28T14:08:19.033 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:19.031+0000 7f96cb7fe640 1 --2- 172.21.15.104:0/1530702365 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f96cc02c460 0x7f96cc1399c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:19.033 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:19.031+0000 7f96caffd640 1 -- 172.21.15.104:0/1530702365 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f96cc1487e0 msgr2=0x7f96cc136ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:19.033 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:19.031+0000 7f96caffd640 1 --2- 172.21.15.104:0/1530702365 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f96cc1487e0 0x7f96cc136ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:19.033 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:19.031+0000 7f96caffd640 1 -- 172.21.15.104:0/1530702365 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f96cc02c460 msgr2=0x7f96cc1399c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:19.033 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:19.031+0000 7f96caffd640 1 --2- 172.21.15.104:0/1530702365 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f96cc02c460 0x7f96cc1399c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:19.033 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:19.031+0000 7f96caffd640 1 -- 172.21.15.104:0/1530702365 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f96cc138900 con 0x7f96cc09e8e0 2024-03-28T14:08:19.033 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:19.031+0000 7f96caffd640 1 --2- 172.21.15.104:0/1530702365 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f96cc09e8e0 0x7f96cc09b560 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f96bc00c890 tx=0x7f96bc00cd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:08:19.033 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:19.031+0000 7f96c3fff640 1 -- 172.21.15.104:0/1530702365 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96bc018070 con 0x7f96cc09e8e0 2024-03-28T14:08:19.033 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:19.031+0000 7f96c3fff640 1 -- 172.21.15.104:0/1530702365 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f96bc00a640 con 0x7f96cc09e8e0 2024-03-28T14:08:19.033 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:19.031+0000 7f96d1b02640 1 -- 172.21.15.104:0/1530702365 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f96cc09c070 con 0x7f96cc09e8e0 2024-03-28T14:08:19.033 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:19.031+0000 7f96c3fff640 1 -- 172.21.15.104:0/1530702365 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96bc00aa30 con 0x7f96cc09e8e0 2024-03-28T14:08:19.033 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:19.031+0000 7f96d1b02640 1 -- 172.21.15.104:0/1530702365 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f96cc137420 con 0x7f96cc09e8e0 2024-03-28T14:08:19.034 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:19.031+0000 7f96d1b02640 1 -- 172.21.15.104:0/1530702365 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f96cc139f00 con 0x7f96cc09e8e0 2024-03-28T14:08:19.034 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:19.031+0000 7f96c3fff640 1 -- 172.21.15.104:0/1530702365 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) ==== 87803+0+0 (secure 0 0 0) 0x7f96bc01b070 con 0x7f96cc09e8e0 2024-03-28T14:08:19.034 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:19.031+0000 7f96c3fff640 1 --2- 172.21.15.104:0/1530702365 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f96b0067ff0 0x7f96b006a4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:19.035 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:19.031+0000 7f96c3fff640 1 -- 172.21.15.104:0/1530702365 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f96bc085f60 con 0x7f96cc09e8e0 2024-03-28T14:08:19.035 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:19.031+0000 7f96cb7fe640 1 --2- 172.21.15.104:0/1530702365 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f96b0067ff0 0x7f96b006a4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:19.035 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:19.031+0000 7f96cb7fe640 1 --2- 172.21.15.104:0/1530702365 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f96b0067ff0 0x7f96b006a4b0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f96b40ad000 tx=0x7f96b4008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:08:19.038 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:19.035+0000 7f96c3fff640 1 -- 172.21.15.104:0/1530702365 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179117 (secure 0 0 0) 0x7f96cc139f00 con 0x7f96cc09e8e0 2024-03-28T14:08:19.187 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:19.187+0000 7f96d1b02640 1 -- 172.21.15.104:0/1530702365 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f96cc0687c0 con 0x7f96cc09e8e0 2024-03-28T14:08:19.189 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:19.187+0000 7f96c3fff640 1 -- 172.21.15.104:0/1530702365 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) ==== 74+0+16629 (secure 0 0 0) 0x7f96bc0566f0 con 0x7f96cc09e8e0 2024-03-28T14:08:19.189 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-03-28T14:08:19.189 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":21,"fsid":"428a90f2-3351-435f-8fcf-f7e66c78a13e","created":"2024-03-28T14:07:34.862458+0000","modified":"2024-03-28T14:08:00.671587+0000","last_up_change":"2024-03-28T14:07:46.028122+0000","last_in_change":"2024-03-28T14:07:37.157486+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-28T14:07:47.622988+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-28T14:07:57.147450+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-28T14:07:57.990975+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"1cb70540-4e42-402f-a446-aaec4ca83007","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6800","nonce":3745690918},{"type":"v1","addr":"172.21.15.104:6801","nonce":3745690918}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":3745690918},{"type":"v1","addr":"172.21.15.104:6803","nonce":3745690918}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":3745690918},{"type":"v1","addr":"172.21.15.104:6807","nonce":3745690918}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":3745690918},{"type":"v1","addr":"172.21.15.104:6805","nonce":3745690918}]},"public_addr":"172.21.15.104:6801/3745690918","cluster_addr":"172.21.15.104:6803/3745690918","heartbeat_back_addr":"172.21.15.104:6807/3745690918","heartbeat_front_addr":"172.21.15.104:6805/3745690918","state":["exists","up"]},{"osd":1,"uuid":"3ab7afdd-b980-418c-81f2-004e16de3dca","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":19438674},{"type":"v1","addr":"172.21.15.104:6810","nonce":19438674}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":19438674},{"type":"v1","addr":"172.21.15.104:6814","nonce":19438674}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":19438674},{"type":"v1","addr":"172.21.15.104:6823","nonce":19438674}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":19438674},{"type":"v1","addr":"172.21.15.104:6819","nonce":19438674}]},"public_addr":"172.21.15.104:6810/19438674","cluster_addr":"172.21.15.104:6814/19438674","heartbeat_back_addr":"172.21.15.104:6823/19438674","heartbeat_front_addr":"172.21.15.104:6819/19438674","state":["exists","up"]},{"osd":2,"uuid":"3e2651c8-ecae-4a9e-a053-27240a79eac2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":2352012626},{"type":"v1","addr":"172.21.15.104:6825","nonce":2352012626}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6826","nonce":2352012626},{"type":"v1","addr":"172.21.15.104:6827","nonce":2352012626}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":2352012626},{"type":"v1","addr":"172.21.15.104:6831","nonce":2352012626}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6828","nonce":2352012626},{"type":"v1","addr":"172.21.15.104:6829","nonce":2352012626}]},"public_addr":"172.21.15.104:6825/2352012626","cluster_addr":"172.21.15.104:6827/2352012626","heartbeat_back_addr":"172.21.15.104:6831/2352012626","heartbeat_front_addr":"172.21.15.104:6829/2352012626","state":["exists","up"]},{"osd":3,"uuid":"925c7e0a-38ef-47cd-b288-8bec8456fb17","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6809","nonce":3568347381},{"type":"v1","addr":"172.21.15.104:6811","nonce":3568347381}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6813","nonce":3568347381},{"type":"v1","addr":"172.21.15.104:6815","nonce":3568347381}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":3568347381},{"type":"v1","addr":"172.21.15.104:6822","nonce":3568347381}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6817","nonce":3568347381},{"type":"v1","addr":"172.21.15.104:6818","nonce":3568347381}]},"public_addr":"172.21.15.104:6811/3568347381","cluster_addr":"172.21.15.104:6815/3568347381","heartbeat_back_addr":"172.21.15.104:6822/3568347381","heartbeat_front_addr":"172.21.15.104:6818/3568347381","state":["exists","up"]},{"osd":4,"uuid":"4a8fac86-e595-4a0e-96e3-57edb3cc3f57","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6800","nonce":2107001077},{"type":"v1","addr":"172.21.15.171:6801","nonce":2107001077}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6802","nonce":2107001077},{"type":"v1","addr":"172.21.15.171:6803","nonce":2107001077}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6806","nonce":2107001077},{"type":"v1","addr":"172.21.15.171:6807","nonce":2107001077}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6804","nonce":2107001077},{"type":"v1","addr":"172.21.15.171:6805","nonce":2107001077}]},"public_addr":"172.21.15.171:6801/2107001077","cluster_addr":"172.21.15.171:6803/2107001077","heartbeat_back_addr":"172.21.15.171:6807/2107001077","heartbeat_front_addr":"172.21.15.171:6805/2107001077","state":["exists","up"]},{"osd":5,"uuid":"e48d50b0-beae-40ae-b15a-9a84dbfe88ab","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6808","nonce":2934826925},{"type":"v1","addr":"172.21.15.171:6810","nonce":2934826925}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6812","nonce":2934826925},{"type":"v1","addr":"172.21.15.171:6814","nonce":2934826925}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6819","nonce":2934826925},{"type":"v1","addr":"172.21.15.171:6821","nonce":2934826925}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6816","nonce":2934826925},{"type":"v1","addr":"172.21.15.171:6817","nonce":2934826925}]},"public_addr":"172.21.15.171:6810/2934826925","cluster_addr":"172.21.15.171:6814/2934826925","heartbeat_back_addr":"172.21.15.171:6821/2934826925","heartbeat_front_addr":"172.21.15.171:6817/2934826925","state":["exists","up"]},{"osd":6,"uuid":"5e7c3292-9936-4f35-b267-aed9d4e49323","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6824","nonce":436742919},{"type":"v1","addr":"172.21.15.171:6825","nonce":436742919}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6826","nonce":436742919},{"type":"v1","addr":"172.21.15.171:6827","nonce":436742919}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6830","nonce":436742919},{"type":"v1","addr":"172.21.15.171:6831","nonce":436742919}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6828","nonce":436742919},{"type":"v1","addr":"172.21.15.171:6829","nonce":436742919}]},"public_addr":"172.21.15.171:6825/436742919","cluster_addr":"172.21.15.171:6827/436742919","heartbeat_back_addr":"172.21.15.171:6831/436742919","heartbeat_front_addr":"172.21.15.171:6829/436742919","state":["exists","up"]},{"osd":7,"uuid":"339b8414-d7f0-4fb0-a446-efaf635cb7ff","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6809","nonce":2055631175},{"type":"v1","addr":"172.21.15.171:6811","nonce":2055631175}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6813","nonce":2055631175},{"type":"v1","addr":"172.21.15.171:6815","nonce":2055631175}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6822","nonce":2055631175},{"type":"v1","addr":"172.21.15.171:6823","nonce":2055631175}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6818","nonce":2055631175},{"type":"v1","addr":"172.21.15.171:6820","nonce":2055631175}]},"public_addr":"172.21.15.171:6811/2055631175","cluster_addr":"172.21.15.171:6815/2055631175","heartbeat_back_addr":"172.21.15.171:6823/2055631175","heartbeat_front_addr":"172.21.15.171:6820/2055631175","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T14:07:42.668881+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T14:07:42.708397+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T14:07:42.634191+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T14:07:42.733662+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T14:07:42.651541+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T14:07:42.783181+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T14:07:42.766310+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T14:07:42.670797+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-28T14:08:19.191 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:19.187+0000 7f96d1b02640 1 -- 172.21.15.104:0/1530702365 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f96b0067ff0 msgr2=0x7f96b006a4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:19.191 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:19.187+0000 7f96d1b02640 1 --2- 172.21.15.104:0/1530702365 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f96b0067ff0 0x7f96b006a4b0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f96b40ad000 tx=0x7f96b4008040 comp rx=0 tx=0).stop 2024-03-28T14:08:19.191 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:19.187+0000 7f96d1b02640 1 -- 172.21.15.104:0/1530702365 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f96cc09e8e0 msgr2=0x7f96cc09b560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:19.191 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:19.187+0000 7f96d1b02640 1 --2- 172.21.15.104:0/1530702365 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f96cc09e8e0 0x7f96cc09b560 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f96bc00c890 tx=0x7f96bc00cd50 comp rx=0 tx=0).stop 2024-03-28T14:08:19.191 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:19.187+0000 7f96d1b02640 1 -- 172.21.15.104:0/1530702365 shutdown_connections 2024-03-28T14:08:19.191 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:19.187+0000 7f96d1b02640 1 --2- 172.21.15.104:0/1530702365 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f96cc1487e0 0x7f96cc136ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:19.191 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:19.187+0000 7f96d1b02640 1 --2- 172.21.15.104:0/1530702365 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f96b0067ff0 0x7f96b006a4b0 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:19.191 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:19.187+0000 7f96d1b02640 1 --2- 172.21.15.104:0/1530702365 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f96cc09e8e0 0x7f96cc09b560 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:19.192 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:19.187+0000 7f96d1b02640 1 --2- 172.21.15.104:0/1530702365 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f96cc02c460 0x7f96cc1399c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:19.192 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:19.187+0000 7f96d1b02640 1 -- 172.21.15.104:0/1530702365 >> 172.21.15.104:0/1530702365 conn(0x7f96cc0b95f0 msgr2=0x7f96cc067330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T14:08:19.192 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:19.187+0000 7f96d1b02640 1 -- 172.21.15.104:0/1530702365 shutdown_connections 2024-03-28T14:08:19.192 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:19.187+0000 7f96d1b02640 1 -- 172.21.15.104:0/1530702365 wait complete. 2024-03-28T14:08:19.206 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-28T14:07:47.622988+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-28T14:07:57.147450+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-28T14:07:57.990975+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-28T14:08:19.207 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-28T14:08:19.335 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:19.331+0000 7f79e2242640 1 Processor -- start 2024-03-28T14:08:19.335 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:19.331+0000 7f79e2242640 1 -- start start 2024-03-28T14:08:19.338 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:19.335+0000 7f79e2242640 1 --2- >> v2:172.21.15.171:3300/0 conn(0x7f79dc0b63a0 0x7f79dc0af9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:19.338 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:19.335+0000 7f79e2242640 1 --2- >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f79dc1445e0 0x7f79dc1449c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:19.338 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:19.335+0000 7f79e2242640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f79dc0b6780 con 0x7f79dc0b0010 2024-03-28T14:08:19.338 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:19.335+0000 7f79e2242640 1 -- --> v2:172.21.15.171:3300/0 -- mon_getmap magic: 0 -- 0x7f79dc0b04c0 con 0x7f79dc0b63a0 2024-03-28T14:08:19.338 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:19.335+0000 7f79e2242640 1 -- --> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] -- mon_getmap magic: 0 -- 0x7f79dc0af610 con 0x7f79dc1445e0 2024-03-28T14:08:19.338 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:19.335+0000 7f79daffd640 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f79dc0b0010 0x7f79dc141670 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:46144/0 (socket says 172.21.15.104:46144) 2024-03-28T14:08:19.339 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:19.335+0000 7f79daffd640 1 -- 172.21.15.104:0/2251961606 learned_addr learned my addr 172.21.15.104:0/2251961606 (peer_addr_for_me v1:172.21.15.104:0/0) 2024-03-28T14:08:19.339 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:19.335+0000 7f79dbfff640 1 --2- 172.21.15.104:0/2251961606 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f79dc1445e0 0x7f79dc1449c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:19.339 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:19.335+0000 7f79db7fe640 1 --2- 172.21.15.104:0/2251961606 >> v2:172.21.15.171:3300/0 conn(0x7f79dc0b63a0 0x7f79dc0af9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:19.339 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:19.335+0000 7f79da7fc640 1 -- 172.21.15.104:0/2251961606 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1342700302 0 0) 0x7f79dc0b6780 con 0x7f79dc0b0010 2024-03-28T14:08:19.339 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:19.335+0000 7f79da7fc640 1 -- 172.21.15.104:0/2251961606 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f79c8003530 con 0x7f79dc0b0010 2024-03-28T14:08:19.339 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:19.335+0000 7f79da7fc640 1 -- 172.21.15.104:0/2251961606 <== mon.0 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 912468265 0 0) 0x7f79c8003530 con 0x7f79dc0b0010 2024-03-28T14:08:19.339 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:19.335+0000 7f79da7fc640 1 -- 172.21.15.104:0/2251961606 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f79dc1445e0 msgr2=0x7f79dc1449c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T14:08:19.339 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:19.335+0000 7f79da7fc640 1 --2- 172.21.15.104:0/2251961606 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f79dc1445e0 0x7f79dc1449c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:19.339 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:19.335+0000 7f79da7fc640 1 -- 172.21.15.104:0/2251961606 >> v2:172.21.15.171:3300/0 conn(0x7f79dc0b63a0 msgr2=0x7f79dc0af9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T14:08:19.339 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:19.335+0000 7f79da7fc640 1 --2- 172.21.15.104:0/2251961606 >> v2:172.21.15.171:3300/0 conn(0x7f79dc0b63a0 0x7f79dc0af9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:19.339 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:19.335+0000 7f79da7fc640 1 -- 172.21.15.104:0/2251961606 --> v1:172.21.15.104:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f79dc06dde0 con 0x7f79dc0b0010 2024-03-28T14:08:19.339 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:19.335+0000 7f79da7fc640 1 -- 172.21.15.104:0/2251961606 <== mon.0 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3745580849 0 0) 0x7f79c4001680 con 0x7f79dc0b0010 2024-03-28T14:08:19.339 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:19.335+0000 7f79da7fc640 1 -- 172.21.15.104:0/2251961606 >> v1:172.21.15.104:6789/0 conn(0x7f79dc0b0010 legacy=0x7f79dc141670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:19.340 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:19.335+0000 7f79da7fc640 1 --2- 172.21.15.104:0/2251961606 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f79c80039c0 0x7f79c8003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:19.340 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:19.335+0000 7f79da7fc640 1 --2- 172.21.15.104:0/2251961606 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f79dc1445e0 0x7f79c8004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:19.340 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:19.335+0000 7f79da7fc640 1 --2- 172.21.15.104:0/2251961606 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f79c8004d50 0x7f79c80051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:19.340 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:19.335+0000 7f79da7fc640 1 -- 172.21.15.104:0/2251961606 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f79dc0b04c0 con 0x7f79c8004d50 2024-03-28T14:08:19.340 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:19.335+0000 7f79da7fc640 1 -- 172.21.15.104:0/2251961606 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f79c800b9d0 con 0x7f79c80039c0 2024-03-28T14:08:19.340 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:19.335+0000 7f79da7fc640 1 -- 172.21.15.104:0/2251961606 --> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] -- mon_getmap magic: 0 -- 0x7f79c800bb50 con 0x7f79dc1445e0 2024-03-28T14:08:19.340 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:19.335+0000 7f79dbfff640 1 --2- 172.21.15.104:0/2251961606 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f79c8004d50 0x7f79c80051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:19.340 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:19.335+0000 7f79db7fe640 1 --2- 172.21.15.104:0/2251961606 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f79c80039c0 0x7f79c8003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:19.340 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:19.335+0000 7f79daffd640 1 --2- 172.21.15.104:0/2251961606 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f79dc1445e0 0x7f79c8004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:19.340 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:19.335+0000 7f79dbfff640 1 -- 172.21.15.104:0/2251961606 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f79dc1445e0 msgr2=0x7f79c8004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T14:08:19.340 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:19.335+0000 7f79dbfff640 1 --2- 172.21.15.104:0/2251961606 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f79dc1445e0 0x7f79c8004810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:19.340 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:19.335+0000 7f79dbfff640 1 -- 172.21.15.104:0/2251961606 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f79c80039c0 msgr2=0x7f79c8003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T14:08:19.340 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:19.335+0000 7f79dbfff640 1 --2- 172.21.15.104:0/2251961606 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f79c80039c0 0x7f79c8003de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:19.341 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:19.335+0000 7f79dbfff640 1 -- 172.21.15.104:0/2251961606 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f79c8003530 con 0x7f79c8004d50 2024-03-28T14:08:19.341 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:19.335+0000 7f79daffd640 1 --2- 172.21.15.104:0/2251961606 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f79dc1445e0 0x7f79c8004810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T14:08:19.344 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:19.335+0000 7f79dbfff640 1 --2- 172.21.15.104:0/2251961606 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f79c8004d50 0x7f79c80051d0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f79d4009ae0 tx=0x7f79d401ca70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b22c46fc29223ccb server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:08:19.344 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:19.335+0000 7f79da7fc640 1 -- 172.21.15.104:0/2251961606 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79d40b2070 con 0x7f79c8004d50 2024-03-28T14:08:19.344 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:19.335+0000 7f79da7fc640 1 -- 172.21.15.104:0/2251961606 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f79d40a5660 con 0x7f79c8004d50 2024-03-28T14:08:19.344 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:19.335+0000 7f79da7fc640 1 -- 172.21.15.104:0/2251961606 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79d40096b0 con 0x7f79c8004d50 2024-03-28T14:08:19.344 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:19.339+0000 7f79e2242640 1 -- 172.21.15.104:0/2251961606 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f79c8004d50 msgr2=0x7f79c80051d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:19.345 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:19.339+0000 7f79e2242640 1 --2- 172.21.15.104:0/2251961606 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f79c8004d50 0x7f79c80051d0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f79d4009ae0 tx=0x7f79d401ca70 comp rx=0 tx=0).stop 2024-03-28T14:08:19.345 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:19.339+0000 7f79dbfff640 1 -- 172.21.15.104:0/2251961606 reap_dead start 2024-03-28T14:08:19.345 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:19.339+0000 7f79e2242640 1 -- 172.21.15.104:0/2251961606 shutdown_connections 2024-03-28T14:08:19.345 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:19.339+0000 7f79e2242640 1 -- 172.21.15.104:0/2251961606 >> 172.21.15.104:0/2251961606 conn(0x7f79dc0ba400 msgr2=0x7f79dc0ba810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T14:08:19.345 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:19.339+0000 7f79e2242640 1 -- 172.21.15.104:0/2251961606 shutdown_connections 2024-03-28T14:08:19.345 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:19.339+0000 7f79e2242640 1 -- 172.21.15.104:0/2251961606 wait complete. 2024-03-28T14:08:19.345 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:19.339+0000 7f79e2242640 1 Processor -- start 2024-03-28T14:08:19.345 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:19.339+0000 7f79e2242640 1 -- start start 2024-03-28T14:08:19.345 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:19.339+0000 7f79e2242640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f79c8004d50 0x7f79dc06b830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:19.345 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:19.339+0000 7f79e2242640 1 --2- >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f79dc0b1900 0x7f79dc0b1d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:19.345 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:19.339+0000 7f79e2242640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f79dc1ddee0 0x7f79dc1de360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:19.345 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:19.339+0000 7f79e2242640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f79dc141f00 con 0x7f79dc1ddee0 2024-03-28T14:08:19.345 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:19.339+0000 7f79e2242640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f79dc141d80 con 0x7f79c8004d50 2024-03-28T14:08:19.346 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:19.339+0000 7f79e2242640 1 -- --> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] -- mon_getmap magic: 0 -- 0x7f79dc1449c0 con 0x7f79dc0b1900 2024-03-28T14:08:19.346 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:19.339+0000 7f79dbfff640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f79dc1ddee0 0x7f79dc1de360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:19.346 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:19.339+0000 7f79dbfff640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f79dc1ddee0 0x7f79dc1de360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:42450/0 (socket says 172.21.15.104:42450) 2024-03-28T14:08:19.346 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:19.339+0000 7f79dbfff640 1 -- 172.21.15.104:0/1204343048 learned_addr learned my addr 172.21.15.104:0/1204343048 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-03-28T14:08:19.346 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:19.339+0000 7f79daffd640 1 --2- 172.21.15.104:0/1204343048 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f79dc0b1900 0x7f79dc0b1d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:19.346 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:19.339+0000 7f79db7fe640 1 --2- 172.21.15.104:0/1204343048 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f79c8004d50 0x7f79dc06b830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:19.346 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:19.339+0000 7f79dbfff640 1 -- 172.21.15.104:0/1204343048 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f79dc0b1900 msgr2=0x7f79dc0b1d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:19.346 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:19.339+0000 7f79dbfff640 1 --2- 172.21.15.104:0/1204343048 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f79dc0b1900 0x7f79dc0b1d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:19.346 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:19.339+0000 7f79dbfff640 1 -- 172.21.15.104:0/1204343048 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f79c8004d50 msgr2=0x7f79dc06b830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:19.347 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:19.339+0000 7f79dbfff640 1 --2- 172.21.15.104:0/1204343048 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f79c8004d50 0x7f79dc06b830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:19.347 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:19.339+0000 7f79dbfff640 1 -- 172.21.15.104:0/1204343048 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f79dc0b1400 con 0x7f79dc1ddee0 2024-03-28T14:08:19.347 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:19.339+0000 7f79dbfff640 1 --2- 172.21.15.104:0/1204343048 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f79dc1ddee0 0x7f79dc1de360 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f79d4009ae0 tx=0x7f79c8005840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:08:19.347 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:19.339+0000 7f79d3fff640 1 -- 172.21.15.104:0/1204343048 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79d40bb070 con 0x7f79dc1ddee0 2024-03-28T14:08:19.347 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:19.339+0000 7f79d3fff640 1 -- 172.21.15.104:0/1204343048 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f79d4000e50 con 0x7f79dc1ddee0 2024-03-28T14:08:19.347 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:19.339+0000 7f79d3fff640 1 -- 172.21.15.104:0/1204343048 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79d40b2040 con 0x7f79dc1ddee0 2024-03-28T14:08:19.347 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:19.339+0000 7f79e2242640 1 -- 172.21.15.104:0/1204343048 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f79dc09c1f0 con 0x7f79dc1ddee0 2024-03-28T14:08:19.347 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:19.339+0000 7f79e2242640 1 -- 172.21.15.104:0/1204343048 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f79dc0b2260 con 0x7f79dc1ddee0 2024-03-28T14:08:19.347 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:19.339+0000 7f79e2242640 1 -- 172.21.15.104:0/1204343048 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f79dc0b15d0 con 0x7f79dc1ddee0 2024-03-28T14:08:19.347 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:19.343+0000 7f79d3fff640 1 -- 172.21.15.104:0/1204343048 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) ==== 87803+0+0 (secure 0 0 0) 0x7f79d40be070 con 0x7f79dc1ddee0 2024-03-28T14:08:19.347 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:19.343+0000 7f79d3fff640 1 --2- 172.21.15.104:0/1204343048 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f79bc067ff0 0x7f79bc06a4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:19.347 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:19.343+0000 7f79d3fff640 1 -- 172.21.15.104:0/1204343048 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f79d411f910 con 0x7f79dc1ddee0 2024-03-28T14:08:19.347 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:19.343+0000 7f79db7fe640 1 --2- 172.21.15.104:0/1204343048 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f79bc067ff0 0x7f79bc06a4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:19.349 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:19.343+0000 7f79db7fe640 1 --2- 172.21.15.104:0/1204343048 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f79bc067ff0 0x7f79bc06a4b0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f79cc007520 tx=0x7f79cc00b040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:08:19.349 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:19.347+0000 7f79d3fff640 1 -- 172.21.15.104:0/1204343048 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179117 (secure 0 0 0) 0x7f79d4016780 con 0x7f79dc1ddee0 2024-03-28T14:08:19.500 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:19.499+0000 7f79e2242640 1 -- 172.21.15.104:0/1204343048 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f79dc140a80 con 0x7f79dc1ddee0 2024-03-28T14:08:19.501 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:19.499+0000 7f79d3fff640 1 -- 172.21.15.104:0/1204343048 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v21) ==== 93+0+10 (secure 0 0 0) 0x7f79d40efbe0 con 0x7f79dc1ddee0 2024-03-28T14:08:19.501 INFO:teuthology.orchestra.run.smithi104.stdout:pg_num: 1 2024-03-28T14:08:19.503 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:19.499+0000 7f79e2242640 1 -- 172.21.15.104:0/1204343048 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f79bc067ff0 msgr2=0x7f79bc06a4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:19.503 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:19.499+0000 7f79e2242640 1 --2- 172.21.15.104:0/1204343048 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f79bc067ff0 0x7f79bc06a4b0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f79cc007520 tx=0x7f79cc00b040 comp rx=0 tx=0).stop 2024-03-28T14:08:19.503 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:19.499+0000 7f79e2242640 1 -- 172.21.15.104:0/1204343048 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f79dc1ddee0 msgr2=0x7f79dc1de360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:19.503 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:19.499+0000 7f79e2242640 1 --2- 172.21.15.104:0/1204343048 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f79dc1ddee0 0x7f79dc1de360 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f79d4009ae0 tx=0x7f79c8005840 comp rx=0 tx=0).stop 2024-03-28T14:08:19.503 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:19.499+0000 7f79e2242640 1 -- 172.21.15.104:0/1204343048 shutdown_connections 2024-03-28T14:08:19.503 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:19.499+0000 7f79e2242640 1 --2- 172.21.15.104:0/1204343048 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f79bc067ff0 0x7f79bc06a4b0 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:19.503 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:19.499+0000 7f79e2242640 1 --2- 172.21.15.104:0/1204343048 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f79dc1ddee0 0x7f79dc1de360 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:19.503 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:19.499+0000 7f79e2242640 1 --2- 172.21.15.104:0/1204343048 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f79dc0b1900 0x7f79dc0b1d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:19.503 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:19.499+0000 7f79e2242640 1 --2- 172.21.15.104:0/1204343048 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f79c8004d50 0x7f79dc06b830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:19.504 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:19.499+0000 7f79e2242640 1 -- 172.21.15.104:0/1204343048 >> 172.21.15.104:0/1204343048 conn(0x7f79dc0ba400 msgr2=0x7f79dc0ba810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T14:08:19.504 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:19.499+0000 7f79e2242640 1 -- 172.21.15.104:0/1204343048 shutdown_connections 2024-03-28T14:08:19.504 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:19.499+0000 7f79e2242640 1 -- 172.21.15.104:0/1204343048 wait complete. 2024-03-28T14:08:19.518 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-28T14:08:19.624 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:19.623+0000 7feaac9a7640 1 Processor -- start 2024-03-28T14:08:19.624 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:19.623+0000 7feaac9a7640 1 -- start start 2024-03-28T14:08:19.626 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:19.623+0000 7feaac9a7640 1 --2- >> v2:172.21.15.171:3300/0 conn(0x7feaa813fc30 0x7feaa8148d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:19.627 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:19.623+0000 7feaac9a7640 1 --2- >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7feaa806d510 0x7feaa8149270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:19.627 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:19.623+0000 7feaac9a7640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7feaa802c840 con 0x7feaa802c460 2024-03-28T14:08:19.627 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:19.623+0000 7feaac9a7640 1 -- --> v2:172.21.15.171:3300/0 -- mon_getmap magic: 0 -- 0x7feaa80b7ab0 con 0x7feaa813fc30 2024-03-28T14:08:19.627 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:19.623+0000 7feaac9a7640 1 -- --> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] -- mon_getmap magic: 0 -- 0x7feaa80b7460 con 0x7feaa806d510 2024-03-28T14:08:19.627 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:19.623+0000 7feaa659c640 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7feaa802c460 0x7feaa813f520 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:46150/0 (socket says 172.21.15.104:46150) 2024-03-28T14:08:19.627 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:19.623+0000 7feaa659c640 1 -- 172.21.15.104:0/2879118287 learned_addr learned my addr 172.21.15.104:0/2879118287 (peer_addr_for_me v1:172.21.15.104:0/0) 2024-03-28T14:08:19.627 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:19.623+0000 7feaa6d9d640 1 --2- 172.21.15.104:0/2879118287 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7feaa806d510 0x7feaa8149270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:19.627 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:19.623+0000 7feaa5d9b640 1 --2- 172.21.15.104:0/2879118287 >> v2:172.21.15.171:3300/0 conn(0x7feaa813fc30 0x7feaa8148d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:19.627 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:19.623+0000 7feaa559a640 1 -- 172.21.15.104:0/2879118287 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2781810729 0 0) 0x7feaa802c840 con 0x7feaa802c460 2024-03-28T14:08:19.627 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:19.623+0000 7feaa559a640 1 -- 172.21.15.104:0/2879118287 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fea94003530 con 0x7feaa802c460 2024-03-28T14:08:19.627 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:19.623+0000 7feaa559a640 1 -- 172.21.15.104:0/2879118287 <== mon.0 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4001247912 0 0) 0x7fea94003530 con 0x7feaa802c460 2024-03-28T14:08:19.627 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:19.623+0000 7feaa559a640 1 -- 172.21.15.104:0/2879118287 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7feaa806d510 msgr2=0x7feaa8149270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T14:08:19.627 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:19.623+0000 7feaa559a640 1 --2- 172.21.15.104:0/2879118287 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7feaa806d510 0x7feaa8149270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:19.628 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:19.623+0000 7feaa559a640 1 -- 172.21.15.104:0/2879118287 >> v2:172.21.15.171:3300/0 conn(0x7feaa813fc30 msgr2=0x7feaa8148d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T14:08:19.628 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:19.623+0000 7feaa559a640 1 --2- 172.21.15.104:0/2879118287 >> v2:172.21.15.171:3300/0 conn(0x7feaa813fc30 0x7feaa8148d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:19.628 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:19.623+0000 7feaa559a640 1 -- 172.21.15.104:0/2879118287 --> v1:172.21.15.104:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feaa80b7740 con 0x7feaa802c460 2024-03-28T14:08:19.628 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:19.623+0000 7feaa559a640 1 -- 172.21.15.104:0/2879118287 <== mon.0 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3745580849 0 0) 0x7feaa0001a00 con 0x7feaa802c460 2024-03-28T14:08:19.628 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:19.623+0000 7feaa559a640 1 -- 172.21.15.104:0/2879118287 >> v1:172.21.15.104:6789/0 conn(0x7feaa802c460 legacy=0x7feaa813f520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:19.628 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:19.623+0000 7feaa559a640 1 --2- 172.21.15.104:0/2879118287 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fea940039c0 0x7fea94003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:19.628 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:19.623+0000 7feaa559a640 1 --2- 172.21.15.104:0/2879118287 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fea94004810 0x7fea94004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:19.628 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:19.623+0000 7feaa559a640 1 --2- 172.21.15.104:0/2879118287 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7feaa806d510 0x7fea940051f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:19.628 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:19.623+0000 7feaa559a640 1 -- 172.21.15.104:0/2879118287 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7feaa80b7ab0 con 0x7fea940039c0 2024-03-28T14:08:19.628 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:19.623+0000 7feaa559a640 1 -- 172.21.15.104:0/2879118287 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7feaa80b7460 con 0x7fea94004810 2024-03-28T14:08:19.628 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:19.623+0000 7feaa559a640 1 -- 172.21.15.104:0/2879118287 --> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] -- mon_getmap magic: 0 -- 0x7fea9400bb60 con 0x7feaa806d510 2024-03-28T14:08:19.628 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:19.623+0000 7feaa659c640 1 --2- 172.21.15.104:0/2879118287 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fea940039c0 0x7fea94003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:19.628 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:19.623+0000 7feaa5d9b640 1 --2- 172.21.15.104:0/2879118287 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fea94004810 0x7fea94004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:19.628 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:19.623+0000 7feaa659c640 1 -- 172.21.15.104:0/2879118287 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7feaa806d510 msgr2=0x7fea940051f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T14:08:19.629 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:19.623+0000 7feaa659c640 1 --2- 172.21.15.104:0/2879118287 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7feaa806d510 0x7fea940051f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:19.629 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:19.623+0000 7feaa659c640 1 -- 172.21.15.104:0/2879118287 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fea94004810 msgr2=0x7fea94004c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T14:08:19.629 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:19.623+0000 7feaa659c640 1 --2- 172.21.15.104:0/2879118287 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fea94004810 0x7fea94004c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:19.629 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:19.623+0000 7feaa659c640 1 -- 172.21.15.104:0/2879118287 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fea94003530 con 0x7fea940039c0 2024-03-28T14:08:19.629 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:19.623+0000 7feaa5d9b640 1 --2- 172.21.15.104:0/2879118287 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fea94004810 0x7fea94004c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T14:08:19.632 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:19.627+0000 7feaa659c640 1 --2- 172.21.15.104:0/2879118287 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fea940039c0 0x7fea94003de0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7feaa000a7a0 tx=0x7feaa001d2e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=79818716e7939687 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:08:19.632 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:19.627+0000 7feaa559a640 1 -- 172.21.15.104:0/2879118287 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feaa000a3f0 con 0x7fea940039c0 2024-03-28T14:08:19.632 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:19.627+0000 7feaa559a640 1 -- 172.21.15.104:0/2879118287 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7feaa00ad630 con 0x7fea940039c0 2024-03-28T14:08:19.632 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:19.627+0000 7feaa559a640 1 -- 172.21.15.104:0/2879118287 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feaa00ada60 con 0x7fea940039c0 2024-03-28T14:08:19.632 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:19.627+0000 7feaac9a7640 1 -- 172.21.15.104:0/2879118287 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fea940039c0 msgr2=0x7fea94003de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:19.632 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:19.627+0000 7feaac9a7640 1 --2- 172.21.15.104:0/2879118287 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fea940039c0 0x7fea94003de0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7feaa000a7a0 tx=0x7feaa001d2e0 comp rx=0 tx=0).stop 2024-03-28T14:08:19.632 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:19.627+0000 7feaa6d9d640 1 -- 172.21.15.104:0/2879118287 reap_dead start 2024-03-28T14:08:19.632 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:19.627+0000 7feaac9a7640 1 -- 172.21.15.104:0/2879118287 shutdown_connections 2024-03-28T14:08:19.632 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:19.627+0000 7feaac9a7640 1 -- 172.21.15.104:0/2879118287 >> 172.21.15.104:0/2879118287 conn(0x7feaa80b9660 msgr2=0x7feaa80b9a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T14:08:19.633 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:19.627+0000 7feaac9a7640 1 -- 172.21.15.104:0/2879118287 shutdown_connections 2024-03-28T14:08:19.633 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:19.627+0000 7feaac9a7640 1 -- 172.21.15.104:0/2879118287 wait complete. 2024-03-28T14:08:19.633 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:19.627+0000 7feaac9a7640 1 Processor -- start 2024-03-28T14:08:19.633 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:19.627+0000 7feaac9a7640 1 -- start start 2024-03-28T14:08:19.633 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:19.627+0000 7feaac9a7640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fea940039c0 0x7feaa80af910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:19.633 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:19.627+0000 7feaac9a7640 1 --2- >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7feaa8142c90 0x7feaa809d8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:19.633 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:19.627+0000 7feaac9a7640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7feaa81400a0 0x7feaa813c130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:19.633 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:19.627+0000 7feaac9a7640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7feaa8150b70 con 0x7feaa81400a0 2024-03-28T14:08:19.633 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:19.627+0000 7feaac9a7640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7feaa81509f0 con 0x7fea940039c0 2024-03-28T14:08:19.633 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:19.627+0000 7feaac9a7640 1 -- --> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] -- mon_getmap magic: 0 -- 0x7feaa8150cf0 con 0x7feaa8142c90 2024-03-28T14:08:19.633 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:19.627+0000 7feaa6d9d640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7feaa81400a0 0x7feaa813c130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:19.633 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:19.627+0000 7feaa6d9d640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7feaa81400a0 0x7feaa813c130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:42480/0 (socket says 172.21.15.104:42480) 2024-03-28T14:08:19.633 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:19.627+0000 7feaa6d9d640 1 -- 172.21.15.104:0/805384761 learned_addr learned my addr 172.21.15.104:0/805384761 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-03-28T14:08:19.634 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:19.627+0000 7feaa5d9b640 1 --2- 172.21.15.104:0/805384761 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7feaa8142c90 0x7feaa809d8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:19.634 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:19.627+0000 7feaa659c640 1 --2- 172.21.15.104:0/805384761 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fea940039c0 0x7feaa80af910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:19.634 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:19.627+0000 7feaa6d9d640 1 -- 172.21.15.104:0/805384761 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7feaa8142c90 msgr2=0x7feaa809d8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:19.634 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:19.627+0000 7feaa6d9d640 1 --2- 172.21.15.104:0/805384761 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7feaa8142c90 0x7feaa809d8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:19.634 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:19.627+0000 7feaa6d9d640 1 -- 172.21.15.104:0/805384761 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fea940039c0 msgr2=0x7feaa80af910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:19.634 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:19.627+0000 7feaa6d9d640 1 --2- 172.21.15.104:0/805384761 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fea940039c0 0x7feaa80af910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:19.634 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:19.627+0000 7feaa6d9d640 1 -- 172.21.15.104:0/805384761 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feaa8141ab0 con 0x7feaa81400a0 2024-03-28T14:08:19.634 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:19.627+0000 7feaa5d9b640 1 --2- 172.21.15.104:0/805384761 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7feaa8142c90 0x7feaa809d8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T14:08:19.634 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:19.627+0000 7feaa6d9d640 1 --2- 172.21.15.104:0/805384761 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7feaa81400a0 0x7feaa813c130 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7feaa80ba720 tx=0x7fea9800d820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:08:19.634 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:19.627+0000 7fea9effd640 1 -- 172.21.15.104:0/805384761 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea980125f0 con 0x7feaa81400a0 2024-03-28T14:08:19.634 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:19.627+0000 7fea9effd640 1 -- 172.21.15.104:0/805384761 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fea98019070 con 0x7feaa81400a0 2024-03-28T14:08:19.634 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:19.627+0000 7feaac9a7640 1 -- 172.21.15.104:0/805384761 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7feaa8141d20 con 0x7feaa81400a0 2024-03-28T14:08:19.634 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:19.627+0000 7fea9effd640 1 -- 172.21.15.104:0/805384761 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea980129d0 con 0x7feaa81400a0 2024-03-28T14:08:19.635 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:19.627+0000 7feaac9a7640 1 -- 172.21.15.104:0/805384761 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7feaa80affa0 con 0x7feaa81400a0 2024-03-28T14:08:19.635 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:19.631+0000 7feaac9a7640 1 -- 172.21.15.104:0/805384761 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7feaa80687c0 con 0x7feaa81400a0 2024-03-28T14:08:19.635 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:19.631+0000 7fea9effd640 1 -- 172.21.15.104:0/805384761 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) ==== 87803+0+0 (secure 0 0 0) 0x7fea98012b70 con 0x7feaa81400a0 2024-03-28T14:08:19.635 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:19.631+0000 7fea9effd640 1 --2- 172.21.15.104:0/805384761 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7fea88067fc0 0x7fea8806a480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:19.636 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:19.631+0000 7fea9effd640 1 -- 172.21.15.104:0/805384761 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7fea980863d0 con 0x7feaa81400a0 2024-03-28T14:08:19.636 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:19.631+0000 7feaa659c640 1 --2- 172.21.15.104:0/805384761 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7fea88067fc0 0x7fea8806a480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:19.636 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:19.631+0000 7feaa659c640 1 --2- 172.21.15.104:0/805384761 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7fea88067fc0 0x7fea8806a480 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7feaa00ad000 tx=0x7feaa0007040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:08:19.637 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:19.635+0000 7fea9effd640 1 -- 172.21.15.104:0/805384761 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179117 (secure 0 0 0) 0x7fea9801b310 con 0x7feaa81400a0 2024-03-28T14:08:19.786 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:19.783+0000 7feaac9a7640 1 -- 172.21.15.104:0/805384761 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7feaa813d060 con 0x7feaa81400a0 2024-03-28T14:08:19.787 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:19.783+0000 7fea9effd640 1 -- 172.21.15.104:0/805384761 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v21) ==== 104+0+11 (secure 0 0 0) 0x7fea98056c10 con 0x7feaa81400a0 2024-03-28T14:08:19.787 INFO:teuthology.orchestra.run.smithi104.stdout:pg_num: 32 2024-03-28T14:08:19.789 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:19.787+0000 7feaac9a7640 1 -- 172.21.15.104:0/805384761 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7fea88067fc0 msgr2=0x7fea8806a480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:19.789 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:19.787+0000 7feaac9a7640 1 --2- 172.21.15.104:0/805384761 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7fea88067fc0 0x7fea8806a480 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7feaa00ad000 tx=0x7feaa0007040 comp rx=0 tx=0).stop 2024-03-28T14:08:19.789 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:19.787+0000 7feaac9a7640 1 -- 172.21.15.104:0/805384761 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7feaa81400a0 msgr2=0x7feaa813c130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:19.789 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:19.787+0000 7feaac9a7640 1 --2- 172.21.15.104:0/805384761 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7feaa81400a0 0x7feaa813c130 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7feaa80ba720 tx=0x7fea9800d820 comp rx=0 tx=0).stop 2024-03-28T14:08:19.789 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:19.787+0000 7feaac9a7640 1 -- 172.21.15.104:0/805384761 shutdown_connections 2024-03-28T14:08:19.789 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:19.787+0000 7feaac9a7640 1 --2- 172.21.15.104:0/805384761 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7fea88067fc0 0x7fea8806a480 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:19.790 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:19.787+0000 7feaac9a7640 1 --2- 172.21.15.104:0/805384761 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7feaa81400a0 0x7feaa813c130 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:19.790 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:19.787+0000 7feaac9a7640 1 --2- 172.21.15.104:0/805384761 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7feaa8142c90 0x7feaa809d8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:19.790 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:19.787+0000 7feaac9a7640 1 --2- 172.21.15.104:0/805384761 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fea940039c0 0x7feaa80af910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:19.790 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:19.787+0000 7feaac9a7640 1 -- 172.21.15.104:0/805384761 >> 172.21.15.104:0/805384761 conn(0x7feaa80b9660 msgr2=0x7feaa8147f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T14:08:19.790 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:19.787+0000 7feaac9a7640 1 -- 172.21.15.104:0/805384761 shutdown_connections 2024-03-28T14:08:19.790 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:19.787+0000 7feaac9a7640 1 -- 172.21.15.104:0/805384761 wait complete. 2024-03-28T14:08:19.805 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-28T14:08:19.927 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:19.923+0000 7f18a3378640 1 Processor -- start 2024-03-28T14:08:19.927 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:19.927+0000 7f18a3378640 1 -- start start 2024-03-28T14:08:19.929 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:19.927+0000 7f18a3378640 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f189c06b830 0x7f189c0b0ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:19.929 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:19.927+0000 7f18a3378640 1 --2- >> v2:172.21.15.171:3300/0 conn(0x7f189c0af020 0x7f189c072d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:19.929 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:19.927+0000 7f18a3378640 1 --2- >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f189c0b58b0 0x7f189c073250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:19.929 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:19.927+0000 7f18a3378640 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 -- 0x7f189c0b1220 con 0x7f189c06b830 2024-03-28T14:08:19.929 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:19.927+0000 7f18a3378640 1 -- --> v2:172.21.15.171:3300/0 -- mon_getmap magic: 0 -- 0x7f189c0af500 con 0x7f189c0af020 2024-03-28T14:08:19.929 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:19.927+0000 7f18a3378640 1 -- --> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] -- mon_getmap magic: 0 -- 0x7f189c06bc10 con 0x7f189c0b58b0 2024-03-28T14:08:19.929 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:19.927+0000 7f18a1114640 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f189c06b830 0x7f189c0b0ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:19.929 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:19.927+0000 7f18a1114640 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f189c06b830 0x7f189c0b0ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:42492/0 (socket says 172.21.15.104:42492) 2024-03-28T14:08:19.929 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:19.927+0000 7f18a1114640 1 -- 172.21.15.104:0/3531698330 learned_addr learned my addr 172.21.15.104:0/3531698330 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-03-28T14:08:19.930 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:19.927+0000 7f18a0913640 1 --2- 172.21.15.104:0/3531698330 >> v2:172.21.15.171:3300/0 conn(0x7f189c0af020 0x7f189c072d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:19.930 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:19.927+0000 7f18a1915640 1 --2- 172.21.15.104:0/3531698330 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f189c0b58b0 0x7f189c073250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:19.930 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:19.927+0000 7f18a1114640 1 -- 172.21.15.104:0/3531698330 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f189c0b58b0 msgr2=0x7f189c073250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T14:08:19.930 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:19.927+0000 7f18a1114640 1 --2- 172.21.15.104:0/3531698330 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f189c0b58b0 0x7f189c073250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:19.930 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:19.927+0000 7f18a1114640 1 -- 172.21.15.104:0/3531698330 >> v2:172.21.15.171:3300/0 conn(0x7f189c0af020 msgr2=0x7f189c072d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T14:08:19.930 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:19.927+0000 7f18a1114640 1 --2- 172.21.15.104:0/3531698330 >> v2:172.21.15.171:3300/0 conn(0x7f189c0af020 0x7f189c072d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:19.930 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:19.927+0000 7f18a1114640 1 -- 172.21.15.104:0/3531698330 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f189c0b51e0 con 0x7f189c06b830 2024-03-28T14:08:19.930 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:19.927+0000 7f18a1915640 1 --2- 172.21.15.104:0/3531698330 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f189c0b58b0 0x7f189c073250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T14:08:19.930 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:19.927+0000 7f18a0913640 1 --2- 172.21.15.104:0/3531698330 >> v2:172.21.15.171:3300/0 conn(0x7f189c0af020 0x7f189c072d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T14:08:19.933 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:19.931+0000 7f18a1114640 1 --2- 172.21.15.104:0/3531698330 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f189c06b830 0x7f189c0b0ce0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f189c06bef0 tx=0x7f189801c3d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f8c953986ffd04e9 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:08:19.933 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:19.931+0000 7f1893fff640 1 -- 172.21.15.104:0/3531698330 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18980b1070 con 0x7f189c06b830 2024-03-28T14:08:19.933 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:19.931+0000 7f1893fff640 1 -- 172.21.15.104:0/3531698330 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f18980ac410 con 0x7f189c06b830 2024-03-28T14:08:19.933 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:19.931+0000 7f1893fff640 1 -- 172.21.15.104:0/3531698330 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18980ac7e0 con 0x7f189c06b830 2024-03-28T14:08:19.933 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:19.931+0000 7f18a3378640 1 -- 172.21.15.104:0/3531698330 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f189c06b830 msgr2=0x7f189c0b0ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:19.933 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:19.931+0000 7f18a3378640 1 --2- 172.21.15.104:0/3531698330 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f189c06b830 0x7f189c0b0ce0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f189c06bef0 tx=0x7f189801c3d0 comp rx=0 tx=0).stop 2024-03-28T14:08:19.933 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:19.931+0000 7f18a3378640 1 -- 172.21.15.104:0/3531698330 shutdown_connections 2024-03-28T14:08:19.933 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:19.931+0000 7f18a3378640 1 --2- 172.21.15.104:0/3531698330 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f189c0b58b0 0x7f189c073250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:19.934 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:19.931+0000 7f18a3378640 1 --2- 172.21.15.104:0/3531698330 >> v2:172.21.15.171:3300/0 conn(0x7f189c0af020 0x7f189c072d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:19.934 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:19.931+0000 7f18a3378640 1 --2- 172.21.15.104:0/3531698330 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f189c06b830 0x7f189c0b0ce0 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:19.934 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:19.931+0000 7f18a3378640 1 -- 172.21.15.104:0/3531698330 >> 172.21.15.104:0/3531698330 conn(0x7f189c0ba400 msgr2=0x7f189c0ba810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T14:08:19.934 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:19.931+0000 7f18a3378640 1 -- 172.21.15.104:0/3531698330 shutdown_connections 2024-03-28T14:08:19.934 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:19.931+0000 7f18a3378640 1 -- 172.21.15.104:0/3531698330 wait complete. 2024-03-28T14:08:19.936 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:19.931+0000 7f18a3378640 1 Processor -- start 2024-03-28T14:08:19.936 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:19.931+0000 7f18a3378640 1 -- start start 2024-03-28T14:08:19.937 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:19.931+0000 7f18a3378640 1 --2- >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f189c06b830 0x7f189c1421b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:19.937 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:19.931+0000 7f18a3378640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f189c0af020 0x7f189c0b3a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:19.937 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:19.931+0000 7f18a3378640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f189c0b58b0 0x7f189c0b13a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:19.937 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:19.931+0000 7f18a3378640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f189c06e710 con 0x7f189c0b58b0 2024-03-28T14:08:19.937 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:19.931+0000 7f18a3378640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f189c06e590 con 0x7f189c0af020 2024-03-28T14:08:19.937 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:19.931+0000 7f18a3378640 1 -- --> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] -- mon_getmap magic: 0 -- 0x7f189c06e890 con 0x7f189c06b830 2024-03-28T14:08:19.937 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:19.931+0000 7f18a1915640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f189c0b58b0 0x7f189c0b13a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:19.937 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:19.931+0000 7f18a1915640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f189c0b58b0 0x7f189c0b13a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:42508/0 (socket says 172.21.15.104:42508) 2024-03-28T14:08:19.937 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:19.931+0000 7f18a1915640 1 -- 172.21.15.104:0/480289527 learned_addr learned my addr 172.21.15.104:0/480289527 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-03-28T14:08:19.938 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:19.931+0000 7f18a1114640 1 --2- 172.21.15.104:0/480289527 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f189c06b830 0x7f189c1421b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:19.938 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:19.931+0000 7f18a0913640 1 --2- 172.21.15.104:0/480289527 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f189c0af020 0x7f189c0b3a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:19.938 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:19.931+0000 7f18a1915640 1 -- 172.21.15.104:0/480289527 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f189c06b830 msgr2=0x7f189c1421b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:19.938 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:19.931+0000 7f18a1915640 1 --2- 172.21.15.104:0/480289527 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f189c06b830 0x7f189c1421b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:19.938 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:19.931+0000 7f18a1915640 1 -- 172.21.15.104:0/480289527 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f189c0af020 msgr2=0x7f189c0b3a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:19.938 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:19.931+0000 7f18a1915640 1 --2- 172.21.15.104:0/480289527 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f189c0af020 0x7f189c0b3a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:19.938 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:19.931+0000 7f18a1915640 1 -- 172.21.15.104:0/480289527 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f189c09c940 con 0x7f189c0b58b0 2024-03-28T14:08:19.938 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:19.931+0000 7f18a1915640 1 --2- 172.21.15.104:0/480289527 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f189c0b58b0 0x7f189c0b13a0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f189400cef0 tx=0x7f189400a710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:08:19.938 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:19.931+0000 7f1891ffb640 1 -- 172.21.15.104:0/480289527 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f189401a070 con 0x7f189c0b58b0 2024-03-28T14:08:19.938 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:19.931+0000 7f1891ffb640 1 -- 172.21.15.104:0/480289527 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f189400d860 con 0x7f189c0b58b0 2024-03-28T14:08:19.938 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:19.931+0000 7f18a3378640 1 -- 172.21.15.104:0/480289527 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f189c0b4420 con 0x7f189c0b58b0 2024-03-28T14:08:19.938 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:19.931+0000 7f1891ffb640 1 -- 172.21.15.104:0/480289527 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f189400dc50 con 0x7f189c0b58b0 2024-03-28T14:08:19.938 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:19.931+0000 7f18a3378640 1 -- 172.21.15.104:0/480289527 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f189c09fbd0 con 0x7f189c0b58b0 2024-03-28T14:08:19.939 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:19.935+0000 7f18a3378640 1 -- 172.21.15.104:0/480289527 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f189c0687c0 con 0x7f189c0b58b0 2024-03-28T14:08:19.939 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:19.935+0000 7f1891ffb640 1 -- 172.21.15.104:0/480289527 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) ==== 87803+0+0 (secure 0 0 0) 0x7f189401d070 con 0x7f189c0b58b0 2024-03-28T14:08:19.939 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:19.935+0000 7f1891ffb640 1 --2- 172.21.15.104:0/480289527 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f1880067fc0 0x7f188006a480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:19.939 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:19.935+0000 7f1891ffb640 1 -- 172.21.15.104:0/480289527 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f1894086d30 con 0x7f189c0b58b0 2024-03-28T14:08:19.940 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:19.935+0000 7f18a1114640 1 --2- 172.21.15.104:0/480289527 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f1880067fc0 0x7f188006a480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:19.940 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:19.935+0000 7f18a1114640 1 --2- 172.21.15.104:0/480289527 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f1880067fc0 0x7f188006a480 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f18980ae000 tx=0x7f1898008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:08:19.941 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:19.939+0000 7f1891ffb640 1 -- 172.21.15.104:0/480289527 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179117 (secure 0 0 0) 0x7f1894057630 con 0x7f189c0b58b0 2024-03-28T14:08:20.099 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:20.095+0000 7f18a3378640 1 -- 172.21.15.104:0/480289527 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f189c0b8e10 con 0x7f189c0b58b0 2024-03-28T14:08:20.099 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:20.099+0000 7f1891ffb640 1 -- 172.21.15.104:0/480289527 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v21) ==== 100+0+11 (secure 0 0 0) 0x7f1894056f30 con 0x7f189c0b58b0 2024-03-28T14:08:20.100 INFO:teuthology.orchestra.run.smithi104.stdout:pg_num: 64 2024-03-28T14:08:20.102 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:20.099+0000 7f18a3378640 1 -- 172.21.15.104:0/480289527 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f1880067fc0 msgr2=0x7f188006a480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:20.102 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:20.099+0000 7f18a3378640 1 --2- 172.21.15.104:0/480289527 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f1880067fc0 0x7f188006a480 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f18980ae000 tx=0x7f1898008040 comp rx=0 tx=0).stop 2024-03-28T14:08:20.102 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:20.099+0000 7f18a3378640 1 -- 172.21.15.104:0/480289527 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f189c0b58b0 msgr2=0x7f189c0b13a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:20.102 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:20.099+0000 7f18a3378640 1 --2- 172.21.15.104:0/480289527 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f189c0b58b0 0x7f189c0b13a0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f189400cef0 tx=0x7f189400a710 comp rx=0 tx=0).stop 2024-03-28T14:08:20.102 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:20.099+0000 7f18a3378640 1 -- 172.21.15.104:0/480289527 shutdown_connections 2024-03-28T14:08:20.102 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:20.099+0000 7f18a3378640 1 --2- 172.21.15.104:0/480289527 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f1880067fc0 0x7f188006a480 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:20.102 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:20.099+0000 7f18a3378640 1 --2- 172.21.15.104:0/480289527 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f189c0b58b0 0x7f189c0b13a0 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:20.102 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:20.099+0000 7f18a3378640 1 --2- 172.21.15.104:0/480289527 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f189c0af020 0x7f189c0b3a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:20.102 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:20.099+0000 7f18a3378640 1 --2- 172.21.15.104:0/480289527 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f189c06b830 0x7f189c1421b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:20.103 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:20.099+0000 7f18a3378640 1 -- 172.21.15.104:0/480289527 >> 172.21.15.104:0/480289527 conn(0x7f189c0ba400 msgr2=0x7f189c148000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T14:08:20.103 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:20.099+0000 7f18a3378640 1 -- 172.21.15.104:0/480289527 shutdown_connections 2024-03-28T14:08:20.103 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:20.099+0000 7f18a3378640 1 -- 172.21.15.104:0/480289527 wait complete. 2024-03-28T14:08:20.161 INFO:tasks.cephfs_test_runner:Starting test: test_cd_with_args (tasks.cephfs.test_cephfs_shell.TestCD) Test that when cd is issued with an argument, CWD is changed 2024-03-28T14:08:20.161 INFO:tasks.cephfs_test_runner:test_cd_with_args (tasks.cephfs.test_cephfs_shell.TestCD) 2024-03-28T14:08:20.161 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-28T14:08:20.271 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:20.267+0000 7f2fd081a640 1 Processor -- start 2024-03-28T14:08:20.271 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:20.267+0000 7f2fd081a640 1 -- start start 2024-03-28T14:08:20.272 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:20.271+0000 7f2fd081a640 1 --2- >> v2:172.21.15.171:3300/0 conn(0x7f2fcc13f540 0x7f2fcc13f9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:20.272 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:20.271+0000 7f2fd081a640 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f2fcc1456c0 0x7f2fcc145b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:20.272 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:20.271+0000 7f2fd081a640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2fcc02c840 con 0x7f2fcc02c460 2024-03-28T14:08:20.272 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:20.271+0000 7f2fd081a640 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 -- 0x7f2fcc0b4f20 con 0x7f2fcc1456c0 2024-03-28T14:08:20.272 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:20.271+0000 7f2fd081a640 1 -- --> v2:172.21.15.171:3300/0 -- mon_getmap magic: 0 -- 0x7f2fcc0b5200 con 0x7f2fcc13f540 2024-03-28T14:08:20.272 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:20.271+0000 7f2fcad9d640 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f2fcc1456c0 0x7f2fcc145b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:20.272 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:20.271+0000 7f2fcad9d640 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f2fcc1456c0 0x7f2fcc145b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:42520/0 (socket says 172.21.15.104:42520) 2024-03-28T14:08:20.273 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:20.271+0000 7f2fcad9d640 1 -- 172.21.15.104:0/2673734911 learned_addr learned my addr 172.21.15.104:0/2673734911 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-03-28T14:08:20.273 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:20.271+0000 7f2fc1d9b640 1 --2- 172.21.15.104:0/2673734911 >> v2:172.21.15.171:3300/0 conn(0x7f2fcc13f540 0x7f2fcc13f9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:20.273 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:20.271+0000 7f2fcad9d640 1 -- 172.21.15.104:0/2673734911 >> v2:172.21.15.171:3300/0 conn(0x7f2fcc13f540 msgr2=0x7f2fcc13f9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T14:08:20.273 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:20.271+0000 7f2fcad9d640 1 --2- 172.21.15.104:0/2673734911 >> v2:172.21.15.171:3300/0 conn(0x7f2fcc13f540 0x7f2fcc13f9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:20.273 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:20.271+0000 7f2fcad9d640 1 -- 172.21.15.104:0/2673734911 >> v1:172.21.15.171:6789/0 conn(0x7f2fcc02c460 legacy=0x7f2fcc0b3ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T14:08:20.273 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:20.271+0000 7f2fcad9d640 1 -- 172.21.15.104:0/2673734911 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2fcc0b54e0 con 0x7f2fcc1456c0 2024-03-28T14:08:20.274 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:20.271+0000 7f2fcad9d640 1 --2- 172.21.15.104:0/2673734911 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2fcc1456c0 0x7f2fcc145b10 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f2fcc070950 tx=0x7f2fc401d380 comp rx=0 tx=0).ready entity=mon.0 client_cookie=13ef53df3b19f335 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:08:20.274 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:20.271+0000 7f2fc9d9b640 1 -- 172.21.15.104:0/2673734911 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2fc40a56f0 con 0x7f2fcc1456c0 2024-03-28T14:08:20.274 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:20.271+0000 7f2fc9d9b640 1 -- 172.21.15.104:0/2673734911 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2fc40a5890 con 0x7f2fcc1456c0 2024-03-28T14:08:20.274 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:20.271+0000 7f2fc9d9b640 1 -- 172.21.15.104:0/2673734911 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2fc40a5c60 con 0x7f2fcc1456c0 2024-03-28T14:08:20.275 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:20.271+0000 7f2fd081a640 1 -- 172.21.15.104:0/2673734911 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2fcc1456c0 msgr2=0x7f2fcc145b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:20.275 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:20.271+0000 7f2fd081a640 1 --2- 172.21.15.104:0/2673734911 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2fcc1456c0 0x7f2fcc145b10 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f2fcc070950 tx=0x7f2fc401d380 comp rx=0 tx=0).stop 2024-03-28T14:08:20.275 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:20.271+0000 7f2fd081a640 1 -- 172.21.15.104:0/2673734911 shutdown_connections 2024-03-28T14:08:20.275 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:20.271+0000 7f2fd081a640 1 --2- 172.21.15.104:0/2673734911 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2fcc1456c0 0x7f2fcc145b10 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:20.275 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:20.271+0000 7f2fd081a640 1 --2- 172.21.15.104:0/2673734911 >> v2:172.21.15.171:3300/0 conn(0x7f2fcc13f540 0x7f2fcc13f9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:20.275 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:20.271+0000 7f2fd081a640 1 -- 172.21.15.104:0/2673734911 >> 172.21.15.104:0/2673734911 conn(0x7f2fcc0b95f0 msgr2=0x7f2fcc0b99d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T14:08:20.276 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:20.271+0000 7f2fd081a640 1 -- 172.21.15.104:0/2673734911 shutdown_connections 2024-03-28T14:08:20.276 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:20.271+0000 7f2fd081a640 1 -- 172.21.15.104:0/2673734911 wait complete. 2024-03-28T14:08:20.278 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:20.275+0000 7f2fd081a640 1 Processor -- start 2024-03-28T14:08:20.278 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:20.275+0000 7f2fd081a640 1 -- start start 2024-03-28T14:08:20.278 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:20.275+0000 7f2fd081a640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2fcc02c460 0x7f2fcc148a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:20.278 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:20.275+0000 7f2fd081a640 1 --2- >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f2fcc13f540 0x7f2fcc071fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:20.278 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:20.275+0000 7f2fd081a640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f2fcc1456c0 0x7f2fcc096390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:20.278 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:20.275+0000 7f2fd081a640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f2fcc152920 con 0x7f2fcc02c460 2024-03-28T14:08:20.278 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:20.275+0000 7f2fd081a640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f2fcc1527a0 con 0x7f2fcc1456c0 2024-03-28T14:08:20.278 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:20.275+0000 7f2fd081a640 1 -- --> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] -- mon_getmap magic: 0 -- 0x7f2fcc152aa0 con 0x7f2fcc13f540 2024-03-28T14:08:20.279 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:20.275+0000 7f2fca59c640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2fcc02c460 0x7f2fcc148a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:20.279 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:20.275+0000 7f2fca59c640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2fcc02c460 0x7f2fcc148a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:42528/0 (socket says 172.21.15.104:42528) 2024-03-28T14:08:20.279 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:20.275+0000 7f2fca59c640 1 -- 172.21.15.104:0/3486327011 learned_addr learned my addr 172.21.15.104:0/3486327011 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-03-28T14:08:20.279 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:20.275+0000 7f2fc1d9b640 1 --2- 172.21.15.104:0/3486327011 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f2fcc13f540 0x7f2fcc071fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:20.279 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:20.275+0000 7f2fcad9d640 1 --2- 172.21.15.104:0/3486327011 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f2fcc1456c0 0x7f2fcc096390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:20.279 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:20.275+0000 7f2fca59c640 1 -- 172.21.15.104:0/3486327011 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f2fcc13f540 msgr2=0x7f2fcc071fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:20.279 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:20.275+0000 7f2fca59c640 1 --2- 172.21.15.104:0/3486327011 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f2fcc13f540 0x7f2fcc071fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:20.279 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:20.275+0000 7f2fca59c640 1 -- 172.21.15.104:0/3486327011 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f2fcc1456c0 msgr2=0x7f2fcc096390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:20.279 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:20.275+0000 7f2fca59c640 1 --2- 172.21.15.104:0/3486327011 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f2fcc1456c0 0x7f2fcc096390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:20.279 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:20.275+0000 7f2fca59c640 1 -- 172.21.15.104:0/3486327011 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2fcc149920 con 0x7f2fcc02c460 2024-03-28T14:08:20.279 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:20.275+0000 7f2fcad9d640 1 --2- 172.21.15.104:0/3486327011 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f2fcc1456c0 0x7f2fcc096390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T14:08:20.279 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:20.275+0000 7f2fca59c640 1 --2- 172.21.15.104:0/3486327011 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2fcc02c460 0x7f2fcc148a40 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f2fb400b440 tx=0x7f2fb400b900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:08:20.279 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:20.275+0000 7f2fc37fe640 1 -- 172.21.15.104:0/3486327011 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2fb4015020 con 0x7f2fcc02c460 2024-03-28T14:08:20.279 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:20.275+0000 7f2fc37fe640 1 -- 172.21.15.104:0/3486327011 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2fb4011890 con 0x7f2fcc02c460 2024-03-28T14:08:20.280 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:20.275+0000 7f2fd081a640 1 -- 172.21.15.104:0/3486327011 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2fcc1e21f0 con 0x7f2fcc02c460 2024-03-28T14:08:20.280 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:20.275+0000 7f2fc37fe640 1 -- 172.21.15.104:0/3486327011 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2fb4011c80 con 0x7f2fcc02c460 2024-03-28T14:08:20.280 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:20.275+0000 7f2fd081a640 1 -- 172.21.15.104:0/3486327011 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2fcc146820 con 0x7f2fcc02c460 2024-03-28T14:08:20.280 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:20.275+0000 7f2fd081a640 1 -- 172.21.15.104:0/3486327011 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2fcc0687c0 con 0x7f2fcc02c460 2024-03-28T14:08:20.281 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:20.275+0000 7f2fc37fe640 1 -- 172.21.15.104:0/3486327011 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) ==== 87803+0+0 (secure 0 0 0) 0x7f2fb4018020 con 0x7f2fcc02c460 2024-03-28T14:08:20.281 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:20.275+0000 7f2fc37fe640 1 --2- 172.21.15.104:0/3486327011 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f2fac067ff0 0x7f2fac06a4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:20.281 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:20.275+0000 7f2fc37fe640 1 -- 172.21.15.104:0/3486327011 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f2fb40862c0 con 0x7f2fcc02c460 2024-03-28T14:08:20.281 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:20.275+0000 7f2fc1d9b640 1 --2- 172.21.15.104:0/3486327011 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f2fac067ff0 0x7f2fac06a4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:20.281 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:20.279+0000 7f2fc1d9b640 1 --2- 172.21.15.104:0/3486327011 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f2fac067ff0 0x7f2fac06a4b0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f2fcc147020 tx=0x7f2fbc008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:08:20.283 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:20.279+0000 7f2fc37fe640 1 -- 172.21.15.104:0/3486327011 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179117 (secure 0 0 0) 0x7f2fb401f030 con 0x7f2fcc02c460 2024-03-28T14:08:20.480 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:20.479+0000 7f2fd081a640 1 -- 172.21.15.104:0/3486327011 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f2fcc1e1e10 con 0x7f2fcc02c460 2024-03-28T14:08:20.483 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:20.479+0000 7f2fc37fe640 1 -- 172.21.15.104:0/3486327011 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) ==== 74+0+16629 (secure 0 0 0) 0x7f2fb400d040 con 0x7f2fcc02c460 2024-03-28T14:08:20.483 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-03-28T14:08:20.483 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":21,"fsid":"428a90f2-3351-435f-8fcf-f7e66c78a13e","created":"2024-03-28T14:07:34.862458+0000","modified":"2024-03-28T14:08:00.671587+0000","last_up_change":"2024-03-28T14:07:46.028122+0000","last_in_change":"2024-03-28T14:07:37.157486+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-28T14:07:47.622988+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-28T14:07:57.147450+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-28T14:07:57.990975+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"1cb70540-4e42-402f-a446-aaec4ca83007","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6800","nonce":3745690918},{"type":"v1","addr":"172.21.15.104:6801","nonce":3745690918}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":3745690918},{"type":"v1","addr":"172.21.15.104:6803","nonce":3745690918}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":3745690918},{"type":"v1","addr":"172.21.15.104:6807","nonce":3745690918}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":3745690918},{"type":"v1","addr":"172.21.15.104:6805","nonce":3745690918}]},"public_addr":"172.21.15.104:6801/3745690918","cluster_addr":"172.21.15.104:6803/3745690918","heartbeat_back_addr":"172.21.15.104:6807/3745690918","heartbeat_front_addr":"172.21.15.104:6805/3745690918","state":["exists","up"]},{"osd":1,"uuid":"3ab7afdd-b980-418c-81f2-004e16de3dca","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":19438674},{"type":"v1","addr":"172.21.15.104:6810","nonce":19438674}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":19438674},{"type":"v1","addr":"172.21.15.104:6814","nonce":19438674}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":19438674},{"type":"v1","addr":"172.21.15.104:6823","nonce":19438674}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":19438674},{"type":"v1","addr":"172.21.15.104:6819","nonce":19438674}]},"public_addr":"172.21.15.104:6810/19438674","cluster_addr":"172.21.15.104:6814/19438674","heartbeat_back_addr":"172.21.15.104:6823/19438674","heartbeat_front_addr":"172.21.15.104:6819/19438674","state":["exists","up"]},{"osd":2,"uuid":"3e2651c8-ecae-4a9e-a053-27240a79eac2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":2352012626},{"type":"v1","addr":"172.21.15.104:6825","nonce":2352012626}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6826","nonce":2352012626},{"type":"v1","addr":"172.21.15.104:6827","nonce":2352012626}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":2352012626},{"type":"v1","addr":"172.21.15.104:6831","nonce":2352012626}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6828","nonce":2352012626},{"type":"v1","addr":"172.21.15.104:6829","nonce":2352012626}]},"public_addr":"172.21.15.104:6825/2352012626","cluster_addr":"172.21.15.104:6827/2352012626","heartbeat_back_addr":"172.21.15.104:6831/2352012626","heartbeat_front_addr":"172.21.15.104:6829/2352012626","state":["exists","up"]},{"osd":3,"uuid":"925c7e0a-38ef-47cd-b288-8bec8456fb17","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6809","nonce":3568347381},{"type":"v1","addr":"172.21.15.104:6811","nonce":3568347381}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6813","nonce":3568347381},{"type":"v1","addr":"172.21.15.104:6815","nonce":3568347381}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":3568347381},{"type":"v1","addr":"172.21.15.104:6822","nonce":3568347381}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6817","nonce":3568347381},{"type":"v1","addr":"172.21.15.104:6818","nonce":3568347381}]},"public_addr":"172.21.15.104:6811/3568347381","cluster_addr":"172.21.15.104:6815/3568347381","heartbeat_back_addr":"172.21.15.104:6822/3568347381","heartbeat_front_addr":"172.21.15.104:6818/3568347381","state":["exists","up"]},{"osd":4,"uuid":"4a8fac86-e595-4a0e-96e3-57edb3cc3f57","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6800","nonce":2107001077},{"type":"v1","addr":"172.21.15.171:6801","nonce":2107001077}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6802","nonce":2107001077},{"type":"v1","addr":"172.21.15.171:6803","nonce":2107001077}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6806","nonce":2107001077},{"type":"v1","addr":"172.21.15.171:6807","nonce":2107001077}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6804","nonce":2107001077},{"type":"v1","addr":"172.21.15.171:6805","nonce":2107001077}]},"public_addr":"172.21.15.171:6801/2107001077","cluster_addr":"172.21.15.171:6803/2107001077","heartbeat_back_addr":"172.21.15.171:6807/2107001077","heartbeat_front_addr":"172.21.15.171:6805/2107001077","state":["exists","up"]},{"osd":5,"uuid":"e48d50b0-beae-40ae-b15a-9a84dbfe88ab","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6808","nonce":2934826925},{"type":"v1","addr":"172.21.15.171:6810","nonce":2934826925}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6812","nonce":2934826925},{"type":"v1","addr":"172.21.15.171:6814","nonce":2934826925}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6819","nonce":2934826925},{"type":"v1","addr":"172.21.15.171:6821","nonce":2934826925}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6816","nonce":2934826925},{"type":"v1","addr":"172.21.15.171:6817","nonce":2934826925}]},"public_addr":"172.21.15.171:6810/2934826925","cluster_addr":"172.21.15.171:6814/2934826925","heartbeat_back_addr":"172.21.15.171:6821/2934826925","heartbeat_front_addr":"172.21.15.171:6817/2934826925","state":["exists","up"]},{"osd":6,"uuid":"5e7c3292-9936-4f35-b267-aed9d4e49323","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6824","nonce":436742919},{"type":"v1","addr":"172.21.15.171:6825","nonce":436742919}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6826","nonce":436742919},{"type":"v1","addr":"172.21.15.171:6827","nonce":436742919}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6830","nonce":436742919},{"type":"v1","addr":"172.21.15.171:6831","nonce":436742919}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6828","nonce":436742919},{"type":"v1","addr":"172.21.15.171:6829","nonce":436742919}]},"public_addr":"172.21.15.171:6825/436742919","cluster_addr":"172.21.15.171:6827/436742919","heartbeat_back_addr":"172.21.15.171:6831/436742919","heartbeat_front_addr":"172.21.15.171:6829/436742919","state":["exists","up"]},{"osd":7,"uuid":"339b8414-d7f0-4fb0-a446-efaf635cb7ff","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6809","nonce":2055631175},{"type":"v1","addr":"172.21.15.171:6811","nonce":2055631175}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6813","nonce":2055631175},{"type":"v1","addr":"172.21.15.171:6815","nonce":2055631175}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6822","nonce":2055631175},{"type":"v1","addr":"172.21.15.171:6823","nonce":2055631175}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6818","nonce":2055631175},{"type":"v1","addr":"172.21.15.171:6820","nonce":2055631175}]},"public_addr":"172.21.15.171:6811/2055631175","cluster_addr":"172.21.15.171:6815/2055631175","heartbeat_back_addr":"172.21.15.171:6823/2055631175","heartbeat_front_addr":"172.21.15.171:6820/2055631175","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T14:07:42.668881+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T14:07:42.708397+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T14:07:42.634191+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T14:07:42.733662+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T14:07:42.651541+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T14:07:42.783181+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T14:07:42.766310+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T14:07:42.670797+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-28T14:08:20.485 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:20.483+0000 7f2fd081a640 1 -- 172.21.15.104:0/3486327011 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f2fac067ff0 msgr2=0x7f2fac06a4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:20.485 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:20.483+0000 7f2fd081a640 1 --2- 172.21.15.104:0/3486327011 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f2fac067ff0 0x7f2fac06a4b0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f2fcc147020 tx=0x7f2fbc008040 comp rx=0 tx=0).stop 2024-03-28T14:08:20.485 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:20.483+0000 7f2fd081a640 1 -- 172.21.15.104:0/3486327011 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2fcc02c460 msgr2=0x7f2fcc148a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:20.486 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:20.483+0000 7f2fd081a640 1 --2- 172.21.15.104:0/3486327011 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2fcc02c460 0x7f2fcc148a40 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f2fb400b440 tx=0x7f2fb400b900 comp rx=0 tx=0).stop 2024-03-28T14:08:20.486 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:20.483+0000 7f2fd081a640 1 -- 172.21.15.104:0/3486327011 shutdown_connections 2024-03-28T14:08:20.486 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:20.483+0000 7f2fd081a640 1 --2- 172.21.15.104:0/3486327011 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f2fcc1456c0 0x7f2fcc096390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:20.486 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:20.483+0000 7f2fd081a640 1 --2- 172.21.15.104:0/3486327011 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f2fcc13f540 0x7f2fcc071fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:20.486 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:20.483+0000 7f2fd081a640 1 --2- 172.21.15.104:0/3486327011 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f2fac067ff0 0x7f2fac06a4b0 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:20.486 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:20.483+0000 7f2fd081a640 1 --2- 172.21.15.104:0/3486327011 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2fcc02c460 0x7f2fcc148a40 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:20.486 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:20.483+0000 7f2fd081a640 1 -- 172.21.15.104:0/3486327011 >> 172.21.15.104:0/3486327011 conn(0x7f2fcc0b95f0 msgr2=0x7f2fcc09d590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T14:08:20.486 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:20.483+0000 7f2fd081a640 1 -- 172.21.15.104:0/3486327011 shutdown_connections 2024-03-28T14:08:20.486 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:20.483+0000 7f2fd081a640 1 -- 172.21.15.104:0/3486327011 wait complete. 2024-03-28T14:08:20.503 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-28T14:07:47.622988+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-28T14:07:57.147450+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-28T14:07:57.990975+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-28T14:08:20.503 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-28T14:08:20.631 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:20.631+0000 7f7bb28b7640 1 Processor -- start 2024-03-28T14:08:20.632 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:20.631+0000 7f7bb28b7640 1 -- start start 2024-03-28T14:08:20.633 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:20.631+0000 7f7bb28b7640 1 --2- >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f7bac141c50 0x7f7bac0b4fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:20.633 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:20.631+0000 7f7bb28b7640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7bac02c840 con 0x7f7bac145210 2024-03-28T14:08:20.633 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:20.631+0000 7f7bb28b7640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7bac0b5690 con 0x7f7bac02c460 2024-03-28T14:08:20.633 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:20.631+0000 7f7bb28b7640 1 -- --> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] -- mon_getmap magic: 0 -- 0x7f7bac1445e0 con 0x7f7bac141c50 2024-03-28T14:08:20.633 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:20.631+0000 7f7ba3fff640 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f7bac145210 0x7f7bac141540 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:46166/0 (socket says 172.21.15.104:46166) 2024-03-28T14:08:20.633 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:20.631+0000 7f7ba3fff640 1 -- 172.21.15.104:0/764031758 learned_addr learned my addr 172.21.15.104:0/764031758 (peer_addr_for_me v1:172.21.15.104:0/0) 2024-03-28T14:08:20.633 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:20.631+0000 7f7bb0e54640 1 --2- 172.21.15.104:0/764031758 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f7bac141c50 0x7f7bac0b4fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:20.634 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:20.631+0000 7f7bab7fe640 1 -- 172.21.15.104:0/764031758 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2906373077 0 0) 0x7f7bac02c840 con 0x7f7bac145210 2024-03-28T14:08:20.634 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:20.631+0000 7f7bab7fe640 1 -- 172.21.15.104:0/764031758 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7b98003530 con 0x7f7bac145210 2024-03-28T14:08:20.634 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:20.631+0000 7f7bab7fe640 1 -- 172.21.15.104:0/764031758 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 453669885 0 0) 0x7f7bac0b5690 con 0x7f7bac02c460 2024-03-28T14:08:20.634 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:20.631+0000 7f7bab7fe640 1 -- 172.21.15.104:0/764031758 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7bac02c840 con 0x7f7bac02c460 2024-03-28T14:08:20.634 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:20.631+0000 7f7bab7fe640 1 -- 172.21.15.104:0/764031758 <== mon.0 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2487616612 0 0) 0x7f7b98003530 con 0x7f7bac145210 2024-03-28T14:08:20.635 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:20.631+0000 7f7bb0e54640 1 -- 172.21.15.104:0/764031758 >> v1:172.21.15.171:6789/0 conn(0x7f7bac02c460 legacy=0x7f7bac144b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:20.635 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:20.631+0000 7f7bb0e54640 1 -- 172.21.15.104:0/764031758 >> v1:172.21.15.104:6789/0 conn(0x7f7bac145210 legacy=0x7f7bac141540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:20.635 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:20.631+0000 7f7bb0e54640 1 -- 172.21.15.104:0/764031758 --> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7bac0b5970 con 0x7f7bac141c50 2024-03-28T14:08:20.636 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:20.635+0000 7f7bb0e54640 1 --2- 172.21.15.104:0/764031758 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f7bac141c50 0x7f7bac0b4fe0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f7ba4009dc0 tx=0x7f7ba401f0a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=50f5fd51c7bd8959 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:08:20.636 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:20.635+0000 7f7bab7fe640 1 -- 172.21.15.104:0/764031758 <== mon.2 v2:172.21.15.171:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ba40b5070 con 0x7f7bac141c50 2024-03-28T14:08:20.636 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:20.635+0000 7f7bab7fe640 1 -- 172.21.15.104:0/764031758 <== mon.2 v2:172.21.15.171:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7ba40a8c30 con 0x7f7bac141c50 2024-03-28T14:08:20.636 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:20.635+0000 7f7bab7fe640 1 -- 172.21.15.104:0/764031758 <== mon.2 v2:172.21.15.171:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b94001680 con 0x7f7bac141c50 2024-03-28T14:08:20.637 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:20.635+0000 7f7bb28b7640 1 -- 172.21.15.104:0/764031758 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f7bac141c50 msgr2=0x7f7bac0b4fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:20.637 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:20.635+0000 7f7bb28b7640 1 --2- 172.21.15.104:0/764031758 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f7bac141c50 0x7f7bac0b4fe0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f7ba4009dc0 tx=0x7f7ba401f0a0 comp rx=0 tx=0).stop 2024-03-28T14:08:20.637 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:20.635+0000 7f7bb28b7640 1 -- 172.21.15.104:0/764031758 shutdown_connections 2024-03-28T14:08:20.637 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:20.635+0000 7f7bb28b7640 1 --2- 172.21.15.104:0/764031758 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f7bac141c50 0x7f7bac0b4fe0 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:20.637 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:20.635+0000 7f7bb28b7640 1 -- 172.21.15.104:0/764031758 >> 172.21.15.104:0/764031758 conn(0x7f7bac0b9660 msgr2=0x7f7bac0b9a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T14:08:20.637 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:20.635+0000 7f7bb28b7640 1 -- 172.21.15.104:0/764031758 shutdown_connections 2024-03-28T14:08:20.638 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:20.635+0000 7f7bb28b7640 1 -- 172.21.15.104:0/764031758 wait complete. 2024-03-28T14:08:20.640 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:20.635+0000 7f7bb28b7640 1 Processor -- start 2024-03-28T14:08:20.640 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:20.635+0000 7f7bb28b7640 1 -- start start 2024-03-28T14:08:20.640 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:20.635+0000 7f7bb28b7640 1 --2- >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f7bac02c460 0x7f7bac09d820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:20.640 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:20.635+0000 7f7bb28b7640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f7bac141c50 0x7f7bac13dee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:20.640 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:20.635+0000 7f7bb28b7640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7bac145210 0x7f7bac0b2f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:20.640 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:20.635+0000 7f7bb28b7640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f7bac0b7d90 con 0x7f7bac145210 2024-03-28T14:08:20.640 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:20.635+0000 7f7bb28b7640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f7bac0b7c10 con 0x7f7bac141c50 2024-03-28T14:08:20.641 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:20.635+0000 7f7bb28b7640 1 -- --> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] -- mon_getmap magic: 0 -- 0x7f7bac06d000 con 0x7f7bac02c460 2024-03-28T14:08:20.641 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:20.635+0000 7f7bb0e54640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7bac145210 0x7f7bac0b2f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:20.641 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:20.635+0000 7f7bb0e54640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7bac145210 0x7f7bac0b2f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:42532/0 (socket says 172.21.15.104:42532) 2024-03-28T14:08:20.641 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:20.635+0000 7f7bb0e54640 1 -- 172.21.15.104:0/2235016398 learned_addr learned my addr 172.21.15.104:0/2235016398 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-03-28T14:08:20.641 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:20.635+0000 7f7babfff640 1 --2- 172.21.15.104:0/2235016398 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f7bac02c460 0x7f7bac09d820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:20.641 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:20.635+0000 7f7ba3fff640 1 --2- 172.21.15.104:0/2235016398 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f7bac141c50 0x7f7bac13dee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:20.641 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:20.635+0000 7f7bb0e54640 1 -- 172.21.15.104:0/2235016398 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f7bac02c460 msgr2=0x7f7bac09d820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:20.641 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:20.635+0000 7f7bb0e54640 1 --2- 172.21.15.104:0/2235016398 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f7bac02c460 0x7f7bac09d820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:20.641 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:20.635+0000 7f7bb0e54640 1 -- 172.21.15.104:0/2235016398 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f7bac141c50 msgr2=0x7f7bac13dee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:20.641 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:20.635+0000 7f7bb0e54640 1 --2- 172.21.15.104:0/2235016398 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f7bac141c50 0x7f7bac13dee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:20.641 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:20.635+0000 7f7bb0e54640 1 -- 172.21.15.104:0/2235016398 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7bac0b0ff0 con 0x7f7bac145210 2024-03-28T14:08:20.641 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:20.635+0000 7f7ba3fff640 1 --2- 172.21.15.104:0/2235016398 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f7bac141c50 0x7f7bac13dee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T14:08:20.641 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:20.639+0000 7f7bb0e54640 1 --2- 172.21.15.104:0/2235016398 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7bac145210 0x7f7bac0b2f30 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f7ba401ccd0 tx=0x7f7ba401be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:08:20.641 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:20.639+0000 7f7ba97fa640 1 -- 172.21.15.104:0/2235016398 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ba40b5070 con 0x7f7bac145210 2024-03-28T14:08:20.642 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:20.639+0000 7f7ba97fa640 1 -- 172.21.15.104:0/2235016398 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7ba40c0070 con 0x7f7bac145210 2024-03-28T14:08:20.642 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:20.639+0000 7f7bb28b7640 1 -- 172.21.15.104:0/2235016398 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7bac0b12d0 con 0x7f7bac145210 2024-03-28T14:08:20.642 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:20.639+0000 7f7ba97fa640 1 -- 172.21.15.104:0/2235016398 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ba4004650 con 0x7f7bac145210 2024-03-28T14:08:20.642 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:20.639+0000 7f7bb28b7640 1 -- 172.21.15.104:0/2235016398 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7bac13e420 con 0x7f7bac145210 2024-03-28T14:08:20.643 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:20.639+0000 7f7bb28b7640 1 -- 172.21.15.104:0/2235016398 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7bac0687c0 con 0x7f7bac145210 2024-03-28T14:08:20.643 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:20.639+0000 7f7ba97fa640 1 -- 172.21.15.104:0/2235016398 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) ==== 87803+0+0 (secure 0 0 0) 0x7f7ba40c5460 con 0x7f7bac145210 2024-03-28T14:08:20.643 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:20.639+0000 7f7ba97fa640 1 --2- 172.21.15.104:0/2235016398 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f7b8c067fc0 0x7f7b8c06a480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:20.643 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:20.639+0000 7f7ba97fa640 1 -- 172.21.15.104:0/2235016398 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f7ba4124680 con 0x7f7bac145210 2024-03-28T14:08:20.643 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:20.639+0000 7f7babfff640 1 --2- 172.21.15.104:0/2235016398 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f7b8c067fc0 0x7f7b8c06a480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:20.643 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:20.639+0000 7f7babfff640 1 --2- 172.21.15.104:0/2235016398 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f7b8c067fc0 0x7f7b8c06a480 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f7b9c0011f0 tx=0x7f7b9c009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:08:20.646 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:20.643+0000 7f7ba97fa640 1 -- 172.21.15.104:0/2235016398 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179117 (secure 0 0 0) 0x7f7ba4018c00 con 0x7f7bac145210 2024-03-28T14:08:20.982 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:20.979+0000 7f7bb28b7640 1 -- 172.21.15.104:0/2235016398 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f7bac13f7b0 con 0x7f7bac145210 2024-03-28T14:08:20.982 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:20.979+0000 7f7ba97fa640 1 -- 172.21.15.104:0/2235016398 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v21) ==== 93+0+10 (secure 0 0 0) 0x7f7ba40f4e30 con 0x7f7bac145210 2024-03-28T14:08:20.982 INFO:teuthology.orchestra.run.smithi104.stdout:pg_num: 1 2024-03-28T14:08:20.985 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:20.983+0000 7f7bb28b7640 1 -- 172.21.15.104:0/2235016398 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f7b8c067fc0 msgr2=0x7f7b8c06a480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:20.985 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:20.983+0000 7f7bb28b7640 1 --2- 172.21.15.104:0/2235016398 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f7b8c067fc0 0x7f7b8c06a480 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f7b9c0011f0 tx=0x7f7b9c009040 comp rx=0 tx=0).stop 2024-03-28T14:08:20.985 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:20.983+0000 7f7bb28b7640 1 -- 172.21.15.104:0/2235016398 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7bac145210 msgr2=0x7f7bac0b2f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:20.985 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:20.983+0000 7f7bb28b7640 1 --2- 172.21.15.104:0/2235016398 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7bac145210 0x7f7bac0b2f30 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f7ba401ccd0 tx=0x7f7ba401be40 comp rx=0 tx=0).stop 2024-03-28T14:08:20.985 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:20.983+0000 7f7bb28b7640 1 -- 172.21.15.104:0/2235016398 shutdown_connections 2024-03-28T14:08:20.985 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:20.983+0000 7f7bb28b7640 1 --2- 172.21.15.104:0/2235016398 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f7b8c067fc0 0x7f7b8c06a480 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:20.985 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:20.983+0000 7f7bb28b7640 1 --2- 172.21.15.104:0/2235016398 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7bac145210 0x7f7bac0b2f30 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:20.985 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:20.983+0000 7f7bb28b7640 1 --2- 172.21.15.104:0/2235016398 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f7bac141c50 0x7f7bac13dee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:20.985 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:20.983+0000 7f7bb28b7640 1 --2- 172.21.15.104:0/2235016398 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f7bac02c460 0x7f7bac09d820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:20.985 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:20.983+0000 7f7bb28b7640 1 -- 172.21.15.104:0/2235016398 >> 172.21.15.104:0/2235016398 conn(0x7f7bac0b9660 msgr2=0x7f7bac1425c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T14:08:20.985 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:20.983+0000 7f7bb28b7640 1 -- 172.21.15.104:0/2235016398 shutdown_connections 2024-03-28T14:08:20.986 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:20.983+0000 7f7bb28b7640 1 -- 172.21.15.104:0/2235016398 wait complete. 2024-03-28T14:08:21.001 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-28T14:08:21.119 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:21.119+0000 7fe2f5627640 1 Processor -- start 2024-03-28T14:08:21.119 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:21.119+0000 7fe2f5627640 1 -- start start 2024-03-28T14:08:21.121 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:21.119+0000 7fe2f5627640 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fe2f0072bf0 0x7fe2f00b1e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:21.121 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:21.119+0000 7fe2f5627640 1 --2- >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fe2f0145ce0 0x7fe2f0146130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:21.121 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:21.119+0000 7fe2f5627640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe2f0072fd0 con 0x7fe2f0071be0 2024-03-28T14:08:21.121 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:21.119+0000 7fe2f5627640 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 -- 0x7fe2f0073250 con 0x7fe2f0072bf0 2024-03-28T14:08:21.121 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:21.119+0000 7fe2f5627640 1 -- --> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] -- mon_getmap magic: 0 -- 0x7fe2f0072050 con 0x7fe2f0145ce0 2024-03-28T14:08:21.121 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:21.119+0000 7fe2eeffd640 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fe2f0072bf0 0x7fe2f00b1e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:21.121 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:21.119+0000 7fe2eeffd640 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fe2f0072bf0 0x7fe2f00b1e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:42540/0 (socket says 172.21.15.104:42540) 2024-03-28T14:08:21.121 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:21.119+0000 7fe2eeffd640 1 -- 172.21.15.104:0/411730744 learned_addr learned my addr 172.21.15.104:0/411730744 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-03-28T14:08:21.121 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:21.119+0000 7fe2ef7fe640 1 --2- 172.21.15.104:0/411730744 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fe2f0145ce0 0x7fe2f0146130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:21.121 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:21.119+0000 7fe2eeffd640 1 -- 172.21.15.104:0/411730744 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fe2f0145ce0 msgr2=0x7fe2f0146130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T14:08:21.121 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:21.119+0000 7fe2eeffd640 1 --2- 172.21.15.104:0/411730744 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fe2f0145ce0 0x7fe2f0146130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:21.122 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:21.119+0000 7fe2eeffd640 1 -- 172.21.15.104:0/411730744 >> v1:172.21.15.171:6789/0 conn(0x7fe2f0071be0 legacy=0x7fe2f00b12f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T14:08:21.122 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:21.119+0000 7fe2eeffd640 1 -- 172.21.15.104:0/411730744 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe2f00734a0 con 0x7fe2f0072bf0 2024-03-28T14:08:21.122 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:21.119+0000 7fe2ef7fe640 1 --2- 172.21.15.104:0/411730744 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fe2f0145ce0 0x7fe2f0146130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T14:08:21.123 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:21.119+0000 7fe2eeffd640 1 --2- 172.21.15.104:0/411730744 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe2f0072bf0 0x7fe2f00b1e80 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fe2f00733d0 tx=0x7fe2e001c3e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9799d5828abbe645 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:08:21.123 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:21.119+0000 7fe2edffb640 1 -- 172.21.15.104:0/411730744 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2e00b0070 con 0x7fe2f0072bf0 2024-03-28T14:08:21.123 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:21.123+0000 7fe2edffb640 1 -- 172.21.15.104:0/411730744 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe2e00ab600 con 0x7fe2f0072bf0 2024-03-28T14:08:21.123 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:21.123+0000 7fe2edffb640 1 -- 172.21.15.104:0/411730744 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2e00ab9d0 con 0x7fe2f0072bf0 2024-03-28T14:08:21.124 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:21.123+0000 7fe2f5627640 1 -- 172.21.15.104:0/411730744 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe2f0072bf0 msgr2=0x7fe2f00b1e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:21.124 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:21.123+0000 7fe2f5627640 1 --2- 172.21.15.104:0/411730744 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe2f0072bf0 0x7fe2f00b1e80 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fe2f00733d0 tx=0x7fe2e001c3e0 comp rx=0 tx=0).stop 2024-03-28T14:08:21.125 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:21.123+0000 7fe2f5627640 1 -- 172.21.15.104:0/411730744 shutdown_connections 2024-03-28T14:08:21.125 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:21.123+0000 7fe2f5627640 1 --2- 172.21.15.104:0/411730744 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fe2f0145ce0 0x7fe2f0146130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:21.125 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:21.123+0000 7fe2f5627640 1 --2- 172.21.15.104:0/411730744 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe2f0072bf0 0x7fe2f00b1e80 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:21.125 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:21.123+0000 7fe2f5627640 1 -- 172.21.15.104:0/411730744 >> 172.21.15.104:0/411730744 conn(0x7fe2f00b7b10 msgr2=0x7fe2f00b7f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T14:08:21.125 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:21.123+0000 7fe2f5627640 1 -- 172.21.15.104:0/411730744 shutdown_connections 2024-03-28T14:08:21.126 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:21.123+0000 7fe2f5627640 1 -- 172.21.15.104:0/411730744 wait complete. 2024-03-28T14:08:21.128 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:21.123+0000 7fe2f5627640 1 Processor -- start 2024-03-28T14:08:21.128 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:21.123+0000 7fe2f5627640 1 -- start start 2024-03-28T14:08:21.128 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:21.123+0000 7fe2f5627640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fe2f0071be0 0x7fe2f006e290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:21.128 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:21.123+0000 7fe2f5627640 1 --2- >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fe2f0072bf0 0x7fe2f01417b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:21.128 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:21.123+0000 7fe2f5627640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe2f0145ce0 0x7fe2f013cd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:21.128 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:21.123+0000 7fe2f5627640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7fe2f0152830 con 0x7fe2f0145ce0 2024-03-28T14:08:21.128 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:21.123+0000 7fe2f5627640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7fe2f01526b0 con 0x7fe2f0071be0 2024-03-28T14:08:21.128 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:21.123+0000 7fe2f5627640 1 -- --> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] -- mon_getmap magic: 0 -- 0x7fe2f01529b0 con 0x7fe2f0072bf0 2024-03-28T14:08:21.128 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:21.123+0000 7fe2ef7fe640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe2f0145ce0 0x7fe2f013cd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:21.129 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:21.123+0000 7fe2ef7fe640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe2f0145ce0 0x7fe2f013cd90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:42554/0 (socket says 172.21.15.104:42554) 2024-03-28T14:08:21.129 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:21.123+0000 7fe2ef7fe640 1 -- 172.21.15.104:0/241727175 learned_addr learned my addr 172.21.15.104:0/241727175 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-03-28T14:08:21.129 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:21.123+0000 7fe2ee7fc640 1 --2- 172.21.15.104:0/241727175 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fe2f0072bf0 0x7fe2f01417b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:21.129 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:21.123+0000 7fe2eeffd640 1 --2- 172.21.15.104:0/241727175 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fe2f0071be0 0x7fe2f006e290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:21.129 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:21.123+0000 7fe2ef7fe640 1 -- 172.21.15.104:0/241727175 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fe2f0072bf0 msgr2=0x7fe2f01417b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:21.129 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:21.123+0000 7fe2ef7fe640 1 --2- 172.21.15.104:0/241727175 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fe2f0072bf0 0x7fe2f01417b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:21.129 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:21.123+0000 7fe2ef7fe640 1 -- 172.21.15.104:0/241727175 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fe2f0071be0 msgr2=0x7fe2f006e290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:21.129 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:21.123+0000 7fe2ef7fe640 1 --2- 172.21.15.104:0/241727175 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fe2f0071be0 0x7fe2f006e290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:21.129 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:21.123+0000 7fe2ef7fe640 1 -- 172.21.15.104:0/241727175 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe2f00a0b00 con 0x7fe2f0145ce0 2024-03-28T14:08:21.129 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:21.123+0000 7fe2ef7fe640 1 --2- 172.21.15.104:0/241727175 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe2f0145ce0 0x7fe2f013cd90 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fe2e800cd50 tx=0x7fe2e800a650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:08:21.129 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:21.123+0000 7fe2d77fe640 1 -- 172.21.15.104:0/241727175 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2e800f680 con 0x7fe2f0145ce0 2024-03-28T14:08:21.129 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:21.123+0000 7fe2d77fe640 1 -- 172.21.15.104:0/241727175 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe2e800f840 con 0x7fe2f0145ce0 2024-03-28T14:08:21.129 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:21.123+0000 7fe2f5627640 1 -- 172.21.15.104:0/241727175 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe2f01e0090 con 0x7fe2f0145ce0 2024-03-28T14:08:21.129 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:21.123+0000 7fe2d77fe640 1 -- 172.21.15.104:0/241727175 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2e800fc30 con 0x7fe2f0145ce0 2024-03-28T14:08:21.130 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:21.123+0000 7fe2f5627640 1 -- 172.21.15.104:0/241727175 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe2f006d830 con 0x7fe2f0145ce0 2024-03-28T14:08:21.130 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:21.127+0000 7fe2f5627640 1 -- 172.21.15.104:0/241727175 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe2f00687c0 con 0x7fe2f0145ce0 2024-03-28T14:08:21.131 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:21.127+0000 7fe2d77fe640 1 -- 172.21.15.104:0/241727175 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) ==== 87803+0+0 (secure 0 0 0) 0x7fe2e80213f0 con 0x7fe2f0145ce0 2024-03-28T14:08:21.131 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:21.127+0000 7fe2d77fe640 1 --2- 172.21.15.104:0/241727175 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7fe2dc067fc0 0x7fe2dc06a480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:21.131 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:21.127+0000 7fe2d77fe640 1 -- 172.21.15.104:0/241727175 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7fe2e80874b0 con 0x7fe2f0145ce0 2024-03-28T14:08:21.131 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:21.127+0000 7fe2eeffd640 1 --2- 172.21.15.104:0/241727175 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7fe2dc067fc0 0x7fe2dc06a480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:21.131 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:21.127+0000 7fe2eeffd640 1 --2- 172.21.15.104:0/241727175 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7fe2dc067fc0 0x7fe2dc06a480 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fe2e00ad000 tx=0x7fe2e00a6000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:08:21.135 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:21.135+0000 7fe2d77fe640 1 -- 172.21.15.104:0/241727175 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179117 (secure 0 0 0) 0x7fe2e8014030 con 0x7fe2f0145ce0 2024-03-28T14:08:21.296 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:21.295+0000 7fe2f5627640 1 -- 172.21.15.104:0/241727175 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fe2f00b83d0 con 0x7fe2f0145ce0 2024-03-28T14:08:21.297 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:21.295+0000 7fe2d77fe640 1 -- 172.21.15.104:0/241727175 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v21) ==== 104+0+11 (secure 0 0 0) 0x7fe2e8057dd0 con 0x7fe2f0145ce0 2024-03-28T14:08:21.297 INFO:teuthology.orchestra.run.smithi104.stdout:pg_num: 32 2024-03-28T14:08:21.299 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:21.295+0000 7fe2f5627640 1 -- 172.21.15.104:0/241727175 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7fe2dc067fc0 msgr2=0x7fe2dc06a480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:21.299 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:21.295+0000 7fe2f5627640 1 --2- 172.21.15.104:0/241727175 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7fe2dc067fc0 0x7fe2dc06a480 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fe2e00ad000 tx=0x7fe2e00a6000 comp rx=0 tx=0).stop 2024-03-28T14:08:21.299 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:21.295+0000 7fe2f5627640 1 -- 172.21.15.104:0/241727175 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe2f0145ce0 msgr2=0x7fe2f013cd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:21.299 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:21.295+0000 7fe2f5627640 1 --2- 172.21.15.104:0/241727175 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe2f0145ce0 0x7fe2f013cd90 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fe2e800cd50 tx=0x7fe2e800a650 comp rx=0 tx=0).stop 2024-03-28T14:08:21.299 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:21.295+0000 7fe2f5627640 1 -- 172.21.15.104:0/241727175 shutdown_connections 2024-03-28T14:08:21.299 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:21.295+0000 7fe2f5627640 1 --2- 172.21.15.104:0/241727175 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7fe2dc067fc0 0x7fe2dc06a480 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:21.300 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:21.295+0000 7fe2f5627640 1 --2- 172.21.15.104:0/241727175 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe2f0145ce0 0x7fe2f013cd90 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:21.300 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:21.295+0000 7fe2f5627640 1 --2- 172.21.15.104:0/241727175 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fe2f0072bf0 0x7fe2f01417b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:21.300 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:21.295+0000 7fe2f5627640 1 --2- 172.21.15.104:0/241727175 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fe2f0071be0 0x7fe2f006e290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:21.300 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:21.295+0000 7fe2f5627640 1 -- 172.21.15.104:0/241727175 >> 172.21.15.104:0/241727175 conn(0x7fe2f00b7b10 msgr2=0x7fe2f00b6160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T14:08:21.300 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:21.295+0000 7fe2f5627640 1 -- 172.21.15.104:0/241727175 shutdown_connections 2024-03-28T14:08:21.300 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:21.295+0000 7fe2f5627640 1 -- 172.21.15.104:0/241727175 wait complete. 2024-03-28T14:08:21.316 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-28T14:08:21.435 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:21.431+0000 7fb3c2efd640 1 Processor -- start 2024-03-28T14:08:21.435 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:21.431+0000 7fb3c2efd640 1 -- start start 2024-03-28T14:08:21.438 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:21.435+0000 7fb3c2efd640 1 --2- >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fb3bc144cf0 0x7fb3bc0b0010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:21.438 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:21.435+0000 7fb3c2efd640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb3bc0b6780 con 0x7fb3bc0b63a0 2024-03-28T14:08:21.438 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:21.435+0000 7fb3c2efd640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb3bc06dde0 con 0x7fb3bc0af9b0 2024-03-28T14:08:21.438 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:21.435+0000 7fb3c2efd640 1 -- --> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] -- mon_getmap magic: 0 -- 0x7fb3bc0af610 con 0x7fb3bc144cf0 2024-03-28T14:08:21.438 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:21.435+0000 7fb3c0c99640 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7fb3bc0b63a0 0x7fb3bc141670 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:46180/0 (socket says 172.21.15.104:46180) 2024-03-28T14:08:21.438 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:21.435+0000 7fb3c0c99640 1 -- 172.21.15.104:0/2705179570 learned_addr learned my addr 172.21.15.104:0/2705179570 (peer_addr_for_me v1:172.21.15.104:0/0) 2024-03-28T14:08:21.438 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:21.435+0000 7fb3bb7fe640 1 -- 172.21.15.104:0/2705179570 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3063527049 0 0) 0x7fb3bc0b6780 con 0x7fb3bc0b63a0 2024-03-28T14:08:21.438 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:21.435+0000 7fb3bb7fe640 1 -- 172.21.15.104:0/2705179570 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb3a8003530 con 0x7fb3bc0b63a0 2024-03-28T14:08:21.438 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:21.435+0000 7fb3c149a640 1 --2- 172.21.15.104:0/2705179570 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fb3bc144cf0 0x7fb3bc0b0010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:21.439 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:21.435+0000 7fb3bb7fe640 1 -- 172.21.15.104:0/2705179570 <== mon.0 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2123110659 0 0) 0x7fb3a8003530 con 0x7fb3bc0b63a0 2024-03-28T14:08:21.439 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:21.435+0000 7fb3bb7fe640 1 -- 172.21.15.104:0/2705179570 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fb3bc144cf0 msgr2=0x7fb3bc0b0010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T14:08:21.439 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:21.435+0000 7fb3bb7fe640 1 --2- 172.21.15.104:0/2705179570 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fb3bc144cf0 0x7fb3bc0b0010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:21.439 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:21.435+0000 7fb3bb7fe640 1 -- 172.21.15.104:0/2705179570 >> v1:172.21.15.171:6789/0 conn(0x7fb3bc0af9b0 legacy=0x7fb3bc1445e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:21.439 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:21.435+0000 7fb3bb7fe640 1 -- 172.21.15.104:0/2705179570 --> v1:172.21.15.104:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb3bc06e0c0 con 0x7fb3bc0b63a0 2024-03-28T14:08:21.439 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:21.435+0000 7fb3bb7fe640 1 -- 172.21.15.104:0/2705179570 <== mon.0 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3745580849 0 0) 0x7fb3a4001680 con 0x7fb3bc0b63a0 2024-03-28T14:08:21.439 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:21.435+0000 7fb3bb7fe640 1 -- 172.21.15.104:0/2705179570 >> v1:172.21.15.104:6789/0 conn(0x7fb3bc0b63a0 legacy=0x7fb3bc141670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:21.439 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:21.435+0000 7fb3bb7fe640 1 --2- 172.21.15.104:0/2705179570 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb3a80039f0 0x7fb3a8003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:21.439 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:21.435+0000 7fb3bb7fe640 1 --2- 172.21.15.104:0/2705179570 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fb3a8004810 0x7fb3a8004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:21.439 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:21.435+0000 7fb3bb7fe640 1 --2- 172.21.15.104:0/2705179570 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fb3bc144cf0 0x7fb3a800b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:21.439 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:21.435+0000 7fb3bb7fe640 1 -- 172.21.15.104:0/2705179570 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7fb3a80054d0 con 0x7fb3a80039f0 2024-03-28T14:08:21.439 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:21.435+0000 7fb3bb7fe640 1 -- 172.21.15.104:0/2705179570 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7fb3a8005680 con 0x7fb3a8004810 2024-03-28T14:08:21.440 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:21.435+0000 7fb3bb7fe640 1 -- 172.21.15.104:0/2705179570 --> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] -- mon_getmap magic: 0 -- 0x7fb3a800bf90 con 0x7fb3bc144cf0 2024-03-28T14:08:21.440 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:21.435+0000 7fb3c0c99640 1 --2- 172.21.15.104:0/2705179570 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb3a80039f0 0x7fb3a8003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:21.440 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:21.435+0000 7fb3c0c99640 1 -- 172.21.15.104:0/2705179570 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fb3bc144cf0 msgr2=0x7fb3a800b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T14:08:21.440 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:21.435+0000 7fb3c149a640 1 --2- 172.21.15.104:0/2705179570 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fb3bc144cf0 0x7fb3a800b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:21.440 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:21.435+0000 7fb3c0c99640 1 --2- 172.21.15.104:0/2705179570 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fb3bc144cf0 0x7fb3a800b9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:21.440 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:21.435+0000 7fb3c0c99640 1 -- 172.21.15.104:0/2705179570 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fb3a8004810 msgr2=0x7fb3a8004c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T14:08:21.440 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:21.435+0000 7fb3bbfff640 1 --2- 172.21.15.104:0/2705179570 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fb3a8004810 0x7fb3a8004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:21.440 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:21.435+0000 7fb3c0c99640 1 --2- 172.21.15.104:0/2705179570 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fb3a8004810 0x7fb3a8004c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:21.440 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:21.435+0000 7fb3c0c99640 1 -- 172.21.15.104:0/2705179570 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb3a8003530 con 0x7fb3a80039f0 2024-03-28T14:08:21.441 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:21.435+0000 7fb3c0c99640 1 --2- 172.21.15.104:0/2705179570 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb3a80039f0 0x7fb3a8003e10 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fb3a400c0e0 tx=0x7fb3a401cb80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=58dbded532970bf6 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:08:21.441 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:21.439+0000 7fb3bb7fe640 1 -- 172.21.15.104:0/2705179570 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3a4007c50 con 0x7fb3a80039f0 2024-03-28T14:08:21.441 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:21.439+0000 7fb3bb7fe640 1 -- 172.21.15.104:0/2705179570 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb3a40a9020 con 0x7fb3a80039f0 2024-03-28T14:08:21.442 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:21.439+0000 7fb3bb7fe640 1 -- 172.21.15.104:0/2705179570 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3a4005b00 con 0x7fb3a80039f0 2024-03-28T14:08:21.442 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:21.439+0000 7fb3c2efd640 1 -- 172.21.15.104:0/2705179570 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb3a80039f0 msgr2=0x7fb3a8003e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:21.442 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:21.439+0000 7fb3c2efd640 1 --2- 172.21.15.104:0/2705179570 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb3a80039f0 0x7fb3a8003e10 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fb3a400c0e0 tx=0x7fb3a401cb80 comp rx=0 tx=0).stop 2024-03-28T14:08:21.442 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:21.439+0000 7fb3c149a640 1 -- 172.21.15.104:0/2705179570 reap_dead start 2024-03-28T14:08:21.442 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:21.439+0000 7fb3c2efd640 1 -- 172.21.15.104:0/2705179570 shutdown_connections 2024-03-28T14:08:21.442 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:21.439+0000 7fb3c2efd640 1 -- 172.21.15.104:0/2705179570 >> 172.21.15.104:0/2705179570 conn(0x7fb3bc0ba400 msgr2=0x7fb3bc0ba810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T14:08:21.442 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:21.439+0000 7fb3c2efd640 1 -- 172.21.15.104:0/2705179570 shutdown_connections 2024-03-28T14:08:21.442 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:21.439+0000 7fb3c2efd640 1 -- 172.21.15.104:0/2705179570 wait complete. 2024-03-28T14:08:21.442 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:21.439+0000 7fb3c2efd640 1 Processor -- start 2024-03-28T14:08:21.442 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:21.439+0000 7fb3c2efd640 1 -- start start 2024-03-28T14:08:21.445 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:21.439+0000 7fb3c2efd640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fb3a80039f0 0x7fb3bc071cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:21.445 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:21.439+0000 7fb3c2efd640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb3bc0729a0 0x7fb3bc1dd550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:21.445 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:21.439+0000 7fb3c2efd640 1 --2- >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fb3bc1dda90 0x7fb3bc138200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:21.445 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:21.439+0000 7fb3c2efd640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7fb3bc1507b0 con 0x7fb3bc0729a0 2024-03-28T14:08:21.445 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:21.439+0000 7fb3c2efd640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7fb3bc141f90 con 0x7fb3a80039f0 2024-03-28T14:08:21.445 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:21.439+0000 7fb3c2efd640 1 -- --> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] -- mon_getmap magic: 0 -- 0x7fb3bc150930 con 0x7fb3bc1dda90 2024-03-28T14:08:21.445 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:21.439+0000 7fb3bbfff640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb3bc0729a0 0x7fb3bc1dd550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:21.445 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:21.439+0000 7fb3bbfff640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb3bc0729a0 0x7fb3bc1dd550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:42572/0 (socket says 172.21.15.104:42572) 2024-03-28T14:08:21.445 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:21.439+0000 7fb3bbfff640 1 -- 172.21.15.104:0/1319495849 learned_addr learned my addr 172.21.15.104:0/1319495849 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-03-28T14:08:21.445 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:21.439+0000 7fb3c0c99640 1 --2- 172.21.15.104:0/1319495849 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fb3a80039f0 0x7fb3bc071cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:21.445 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:21.439+0000 7fb3c149a640 1 --2- 172.21.15.104:0/1319495849 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fb3bc1dda90 0x7fb3bc138200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:21.445 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:21.439+0000 7fb3bbfff640 1 -- 172.21.15.104:0/1319495849 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fb3bc1dda90 msgr2=0x7fb3bc138200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:21.446 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:21.439+0000 7fb3bbfff640 1 --2- 172.21.15.104:0/1319495849 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fb3bc1dda90 0x7fb3bc138200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:21.446 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:21.439+0000 7fb3bbfff640 1 -- 172.21.15.104:0/1319495849 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fb3a80039f0 msgr2=0x7fb3bc071cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:21.446 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:21.439+0000 7fb3bbfff640 1 --2- 172.21.15.104:0/1319495849 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fb3a80039f0 0x7fb3bc071cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:21.446 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:21.439+0000 7fb3bbfff640 1 -- 172.21.15.104:0/1319495849 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb3bc0b1860 con 0x7fb3bc0729a0 2024-03-28T14:08:21.446 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:21.443+0000 7fb3bbfff640 1 --2- 172.21.15.104:0/1319495849 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb3bc0729a0 0x7fb3bc1dd550 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7fb3ac00d7d0 tx=0x7fb3ac00dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:08:21.446 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:21.443+0000 7fb3b97fa640 1 -- 172.21.15.104:0/1319495849 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3ac005270 con 0x7fb3bc0729a0 2024-03-28T14:08:21.446 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:21.443+0000 7fb3b97fa640 1 -- 172.21.15.104:0/1319495849 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb3ac017070 con 0x7fb3bc0729a0 2024-03-28T14:08:21.446 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:21.443+0000 7fb3b97fa640 1 -- 172.21.15.104:0/1319495849 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3ac00b6b0 con 0x7fb3bc0729a0 2024-03-28T14:08:21.446 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:21.443+0000 7fb3c2efd640 1 -- 172.21.15.104:0/1319495849 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb3bc0b3310 con 0x7fb3bc0729a0 2024-03-28T14:08:21.446 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:21.443+0000 7fb3c2efd640 1 -- 172.21.15.104:0/1319495849 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb3bc0b1b00 con 0x7fb3bc0729a0 2024-03-28T14:08:21.446 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:21.443+0000 7fb3c2efd640 1 -- 172.21.15.104:0/1319495849 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb3bc0687c0 con 0x7fb3bc0729a0 2024-03-28T14:08:21.447 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:21.443+0000 7fb3b97fa640 1 -- 172.21.15.104:0/1319495849 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) ==== 87803+0+0 (secure 0 0 0) 0x7fb3ac00b850 con 0x7fb3bc0729a0 2024-03-28T14:08:21.447 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:21.443+0000 7fb3b97fa640 1 --2- 172.21.15.104:0/1319495849 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7fb39c067fc0 0x7fb39c06a480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:21.447 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:21.443+0000 7fb3b97fa640 1 -- 172.21.15.104:0/1319495849 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7fb3ac086d20 con 0x7fb3bc0729a0 2024-03-28T14:08:21.447 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:21.443+0000 7fb3c0c99640 1 --2- 172.21.15.104:0/1319495849 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7fb39c067fc0 0x7fb39c06a480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:21.447 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:21.443+0000 7fb3c0c99640 1 --2- 172.21.15.104:0/1319495849 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7fb39c067fc0 0x7fb39c06a480 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fb3a40aa000 tx=0x7fb3a40b0040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:08:21.449 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:21.447+0000 7fb3b97fa640 1 -- 172.21.15.104:0/1319495849 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179117 (secure 0 0 0) 0x7fb3ac057640 con 0x7fb3bc0729a0 2024-03-28T14:08:21.607 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:21.607+0000 7fb3c2efd640 1 -- 172.21.15.104:0/1319495849 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fb3bc0b6780 con 0x7fb3bc0729a0 2024-03-28T14:08:21.608 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:21.607+0000 7fb3b97fa640 1 -- 172.21.15.104:0/1319495849 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v21) ==== 100+0+11 (secure 0 0 0) 0x7fb3ac056f40 con 0x7fb3bc0729a0 2024-03-28T14:08:21.608 INFO:teuthology.orchestra.run.smithi104.stdout:pg_num: 64 2024-03-28T14:08:21.610 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:21.607+0000 7fb3c2efd640 1 -- 172.21.15.104:0/1319495849 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7fb39c067fc0 msgr2=0x7fb39c06a480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:21.610 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:21.607+0000 7fb3c2efd640 1 --2- 172.21.15.104:0/1319495849 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7fb39c067fc0 0x7fb39c06a480 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fb3a40aa000 tx=0x7fb3a40b0040 comp rx=0 tx=0).stop 2024-03-28T14:08:21.610 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:21.607+0000 7fb3c2efd640 1 -- 172.21.15.104:0/1319495849 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb3bc0729a0 msgr2=0x7fb3bc1dd550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:21.610 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:21.607+0000 7fb3c2efd640 1 --2- 172.21.15.104:0/1319495849 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb3bc0729a0 0x7fb3bc1dd550 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7fb3ac00d7d0 tx=0x7fb3ac00dc90 comp rx=0 tx=0).stop 2024-03-28T14:08:21.610 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:21.607+0000 7fb3c2efd640 1 -- 172.21.15.104:0/1319495849 shutdown_connections 2024-03-28T14:08:21.611 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:21.607+0000 7fb3c2efd640 1 --2- 172.21.15.104:0/1319495849 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fb3bc1dda90 0x7fb3bc138200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:21.611 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:21.607+0000 7fb3c2efd640 1 --2- 172.21.15.104:0/1319495849 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7fb39c067fc0 0x7fb39c06a480 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:21.611 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:21.607+0000 7fb3c2efd640 1 --2- 172.21.15.104:0/1319495849 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb3bc0729a0 0x7fb3bc1dd550 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:21.611 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:21.607+0000 7fb3c2efd640 1 --2- 172.21.15.104:0/1319495849 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fb3a80039f0 0x7fb3bc071cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:21.611 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:21.607+0000 7fb3c2efd640 1 -- 172.21.15.104:0/1319495849 >> 172.21.15.104:0/1319495849 conn(0x7fb3bc0ba400 msgr2=0x7fb3bc143c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T14:08:21.611 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:21.607+0000 7fb3c2efd640 1 -- 172.21.15.104:0/1319495849 shutdown_connections 2024-03-28T14:08:21.611 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:21.607+0000 7fb3c2efd640 1 -- 172.21.15.104:0/1319495849 wait complete. 2024-03-28T14:08:21.626 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_cephfs_shell.TestCD.test_cd_with_args' 2024-03-28T14:08:21.735 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:21.735+0000 7fe6bf599640 1 Processor -- start 2024-03-28T14:08:21.736 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:21.735+0000 7fe6bf599640 1 -- start start 2024-03-28T14:08:21.738 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:21.735+0000 7fe6bf599640 1 --2- >> v2:172.21.15.171:3300/0 conn(0x7fe6b802c460 0x7fe6b80b3ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:21.738 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:21.735+0000 7fe6bf599640 1 --2- >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fe6b813fc50 0x7fe6b81456c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:21.738 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:21.735+0000 7fe6bf599640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe6b802c840 con 0x7fe6b80b4530 2024-03-28T14:08:21.738 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:21.735+0000 7fe6bf599640 1 -- --> v2:172.21.15.171:3300/0 -- mon_getmap magic: 0 -- 0x7fe6b80b4f20 con 0x7fe6b802c460 2024-03-28T14:08:21.738 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:21.735+0000 7fe6bf599640 1 -- --> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] -- mon_getmap magic: 0 -- 0x7fe6b80b5200 con 0x7fe6b813fc50 2024-03-28T14:08:21.738 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:21.735+0000 7fe6bcb34640 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7fe6b80b4530 0x7fe6b813f540 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:46186/0 (socket says 172.21.15.104:46186) 2024-03-28T14:08:21.739 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:21.735+0000 7fe6bcb34640 1 -- 172.21.15.104:0/2453139678 learned_addr learned my addr 172.21.15.104:0/2453139678 (peer_addr_for_me v1:172.21.15.104:0/0) 2024-03-28T14:08:21.739 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:21.735+0000 7fe6bdb36640 1 --2- 172.21.15.104:0/2453139678 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fe6b813fc50 0x7fe6b81456c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:21.739 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:21.735+0000 7fe6bd335640 1 --2- 172.21.15.104:0/2453139678 >> v2:172.21.15.171:3300/0 conn(0x7fe6b802c460 0x7fe6b80b3ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:21.739 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:21.735+0000 7fe6affff640 1 -- 172.21.15.104:0/2453139678 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2869035836 0 0) 0x7fe6b802c840 con 0x7fe6b80b4530 2024-03-28T14:08:21.739 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:21.735+0000 7fe6affff640 1 -- 172.21.15.104:0/2453139678 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe6a4003530 con 0x7fe6b80b4530 2024-03-28T14:08:21.739 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:21.735+0000 7fe6affff640 1 -- 172.21.15.104:0/2453139678 <== mon.0 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2141689265 0 0) 0x7fe6a4003530 con 0x7fe6b80b4530 2024-03-28T14:08:21.739 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:21.735+0000 7fe6affff640 1 -- 172.21.15.104:0/2453139678 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fe6b813fc50 msgr2=0x7fe6b81456c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T14:08:21.739 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:21.735+0000 7fe6affff640 1 --2- 172.21.15.104:0/2453139678 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fe6b813fc50 0x7fe6b81456c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:21.739 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:21.735+0000 7fe6affff640 1 -- 172.21.15.104:0/2453139678 >> v2:172.21.15.171:3300/0 conn(0x7fe6b802c460 msgr2=0x7fe6b80b3ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T14:08:21.739 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:21.735+0000 7fe6affff640 1 --2- 172.21.15.104:0/2453139678 >> v2:172.21.15.171:3300/0 conn(0x7fe6b802c460 0x7fe6b80b3ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:21.739 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:21.735+0000 7fe6affff640 1 -- 172.21.15.104:0/2453139678 --> v1:172.21.15.104:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe6b80b54e0 con 0x7fe6b80b4530 2024-03-28T14:08:21.739 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:21.735+0000 7fe6affff640 1 -- 172.21.15.104:0/2453139678 <== mon.0 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3745580849 0 0) 0x7fe6a8001680 con 0x7fe6b80b4530 2024-03-28T14:08:21.739 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:21.735+0000 7fe6affff640 1 -- 172.21.15.104:0/2453139678 >> v1:172.21.15.104:6789/0 conn(0x7fe6b80b4530 legacy=0x7fe6b813f540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:21.739 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:21.735+0000 7fe6affff640 1 --2- 172.21.15.104:0/2453139678 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe6a40039c0 0x7fe6a4003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:21.740 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:21.735+0000 7fe6affff640 1 --2- 172.21.15.104:0/2453139678 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fe6a4004810 0x7fe6a4004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:21.740 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:21.735+0000 7fe6affff640 1 --2- 172.21.15.104:0/2453139678 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fe6b813fc50 0x7fe6a40051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:21.740 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:21.735+0000 7fe6affff640 1 -- 172.21.15.104:0/2453139678 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7fe6b80b4f20 con 0x7fe6a40039c0 2024-03-28T14:08:21.740 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:21.735+0000 7fe6affff640 1 -- 172.21.15.104:0/2453139678 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7fe6b80b5200 con 0x7fe6a4004810 2024-03-28T14:08:21.740 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:21.735+0000 7fe6affff640 1 -- 172.21.15.104:0/2453139678 --> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] -- mon_getmap magic: 0 -- 0x7fe6a400b9d0 con 0x7fe6b813fc50 2024-03-28T14:08:21.740 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:21.735+0000 7fe6bd335640 1 --2- 172.21.15.104:0/2453139678 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe6a40039c0 0x7fe6a4003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:21.740 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:21.735+0000 7fe6bcb34640 1 --2- 172.21.15.104:0/2453139678 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fe6a4004810 0x7fe6a4004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:21.740 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:21.735+0000 7fe6bdb36640 1 --2- 172.21.15.104:0/2453139678 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fe6b813fc50 0x7fe6a40051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:21.740 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:21.735+0000 7fe6bd335640 1 -- 172.21.15.104:0/2453139678 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fe6b813fc50 msgr2=0x7fe6a40051d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T14:08:21.740 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:21.735+0000 7fe6bd335640 1 --2- 172.21.15.104:0/2453139678 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fe6b813fc50 0x7fe6a40051d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:21.740 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:21.735+0000 7fe6bd335640 1 -- 172.21.15.104:0/2453139678 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fe6a4004810 msgr2=0x7fe6a4004c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T14:08:21.740 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:21.735+0000 7fe6bd335640 1 --2- 172.21.15.104:0/2453139678 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fe6a4004810 0x7fe6a4004c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:21.740 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:21.735+0000 7fe6bd335640 1 -- 172.21.15.104:0/2453139678 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe6a4003530 con 0x7fe6a40039c0 2024-03-28T14:08:21.742 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:21.739+0000 7fe6bd335640 1 --2- 172.21.15.104:0/2453139678 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe6a40039c0 0x7fe6a4003de0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fe6b4009b10 tx=0x7fe6b401cab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ae9aaf7ba085ac7d server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:08:21.742 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:21.739+0000 7fe6affff640 1 -- 172.21.15.104:0/2453139678 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6b40096b0 con 0x7fe6a40039c0 2024-03-28T14:08:21.742 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:21.739+0000 7fe6affff640 1 -- 172.21.15.104:0/2453139678 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe6b40a64d0 con 0x7fe6a40039c0 2024-03-28T14:08:21.742 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:21.739+0000 7fe6affff640 1 -- 172.21.15.104:0/2453139678 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6b40a68c0 con 0x7fe6a40039c0 2024-03-28T14:08:21.742 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:21.739+0000 7fe6bf599640 1 -- 172.21.15.104:0/2453139678 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe6a40039c0 msgr2=0x7fe6a4003de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:21.742 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:21.739+0000 7fe6bf599640 1 --2- 172.21.15.104:0/2453139678 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe6a40039c0 0x7fe6a4003de0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fe6b4009b10 tx=0x7fe6b401cab0 comp rx=0 tx=0).stop 2024-03-28T14:08:21.742 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:21.739+0000 7fe6bdb36640 1 -- 172.21.15.104:0/2453139678 reap_dead start 2024-03-28T14:08:21.742 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:21.739+0000 7fe6bf599640 1 -- 172.21.15.104:0/2453139678 shutdown_connections 2024-03-28T14:08:21.742 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:21.739+0000 7fe6bf599640 1 -- 172.21.15.104:0/2453139678 >> 172.21.15.104:0/2453139678 conn(0x7fe6b80b95f0 msgr2=0x7fe6b80b99d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T14:08:21.743 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:21.739+0000 7fe6bf599640 1 -- 172.21.15.104:0/2453139678 shutdown_connections 2024-03-28T14:08:21.743 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:21.739+0000 7fe6bf599640 1 -- 172.21.15.104:0/2453139678 wait complete. 2024-03-28T14:08:21.745 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:21.739+0000 7fe6bf599640 1 Processor -- start 2024-03-28T14:08:21.745 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:21.739+0000 7fe6bf599640 1 -- start start 2024-03-28T14:08:21.745 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:21.739+0000 7fe6bf599640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fe6a40039c0 0x7fe6b8099280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:21.746 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:21.739+0000 7fe6bf599640 1 --2- >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fe6b8097050 0x7fe6b81e1b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:21.746 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:21.739+0000 7fe6bf599640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe6b80974d0 0x7fe6b81e2060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:21.746 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:21.739+0000 7fe6bf599640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7fe6b8150b20 con 0x7fe6b80974d0 2024-03-28T14:08:21.746 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:21.739+0000 7fe6bf599640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7fe6b81509a0 con 0x7fe6a40039c0 2024-03-28T14:08:21.746 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:21.739+0000 7fe6bf599640 1 -- --> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] -- mon_getmap magic: 0 -- 0x7fe6b8150ca0 con 0x7fe6b8097050 2024-03-28T14:08:21.746 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:21.743+0000 7fe6bdb36640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe6b80974d0 0x7fe6b81e2060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:21.746 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:21.743+0000 7fe6bdb36640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe6b80974d0 0x7fe6b81e2060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:42594/0 (socket says 172.21.15.104:42594) 2024-03-28T14:08:21.746 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:21.743+0000 7fe6bdb36640 1 -- 172.21.15.104:0/1111954363 learned_addr learned my addr 172.21.15.104:0/1111954363 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-03-28T14:08:21.746 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:21.743+0000 7fe6bcb34640 1 --2- 172.21.15.104:0/1111954363 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fe6b8097050 0x7fe6b81e1b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:21.746 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:21.743+0000 7fe6bd335640 1 --2- 172.21.15.104:0/1111954363 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fe6a40039c0 0x7fe6b8099280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:21.746 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:21.743+0000 7fe6bdb36640 1 -- 172.21.15.104:0/1111954363 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fe6b8097050 msgr2=0x7fe6b81e1b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:21.746 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:21.743+0000 7fe6bdb36640 1 --2- 172.21.15.104:0/1111954363 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fe6b8097050 0x7fe6b81e1b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:21.746 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:21.743+0000 7fe6bdb36640 1 -- 172.21.15.104:0/1111954363 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fe6a40039c0 msgr2=0x7fe6b8099280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:21.747 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:21.743+0000 7fe6bdb36640 1 --2- 172.21.15.104:0/1111954363 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fe6a40039c0 0x7fe6b8099280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:21.747 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:21.743+0000 7fe6bdb36640 1 -- 172.21.15.104:0/1111954363 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe6b80723c0 con 0x7fe6b80974d0 2024-03-28T14:08:21.747 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:21.743+0000 7fe6bdb36640 1 --2- 172.21.15.104:0/1111954363 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe6b80974d0 0x7fe6b81e2060 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7fe6b80706b0 tx=0x7fe6b000f4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:08:21.747 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:21.743+0000 7fe6adffb640 1 -- 172.21.15.104:0/1111954363 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6b001c070 con 0x7fe6b80974d0 2024-03-28T14:08:21.747 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:21.743+0000 7fe6adffb640 1 -- 172.21.15.104:0/1111954363 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe6b0017400 con 0x7fe6b80974d0 2024-03-28T14:08:21.747 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:21.743+0000 7fe6bf599640 1 -- 172.21.15.104:0/1111954363 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe6b8099950 con 0x7fe6b80974d0 2024-03-28T14:08:21.747 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:21.743+0000 7fe6adffb640 1 -- 172.21.15.104:0/1111954363 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6b00177f0 con 0x7fe6b80974d0 2024-03-28T14:08:21.747 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:21.743+0000 7fe6bf599640 1 -- 172.21.15.104:0/1111954363 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe6b8099f10 con 0x7fe6b80974d0 2024-03-28T14:08:21.747 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:21.743+0000 7fe6bf599640 1 -- 172.21.15.104:0/1111954363 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe6b80687c0 con 0x7fe6b80974d0 2024-03-28T14:08:21.747 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:21.743+0000 7fe6adffb640 1 -- 172.21.15.104:0/1111954363 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) ==== 87803+0+0 (secure 0 0 0) 0x7fe6b001f070 con 0x7fe6b80974d0 2024-03-28T14:08:21.748 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:21.743+0000 7fe6adffb640 1 --2- 172.21.15.104:0/1111954363 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7fe69c067ff0 0x7fe69c06a4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:21.748 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:21.743+0000 7fe6adffb640 1 -- 172.21.15.104:0/1111954363 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7fe6b00865b0 con 0x7fe6b80974d0 2024-03-28T14:08:21.748 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:21.743+0000 7fe6bd335640 1 --2- 172.21.15.104:0/1111954363 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7fe69c067ff0 0x7fe69c06a4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:21.748 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:21.743+0000 7fe6bd335640 1 --2- 172.21.15.104:0/1111954363 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7fe69c067ff0 0x7fe69c06a4b0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fe6b40af000 tx=0x7fe6b40046d0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:08:21.750 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:21.747+0000 7fe6adffb640 1 -- 172.21.15.104:0/1111954363 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179117 (secure 0 0 0) 0x7fe6b0012070 con 0x7fe6b80974d0 2024-03-28T14:08:21.962 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:21.959+0000 7fe6bf599640 1 -- 172.21.15.104:0/1111954363 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_cephfs_shell.TestCD.test_cd_with_args"]} v 0) -- 0x7fe6b809a0e0 con 0x7fe6b80974d0 2024-03-28T14:08:22.814 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:22.811+0000 7fe6adffb640 1 -- 172.21.15.104:0/1111954363 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_cephfs_shell.TestCD.test_cd_with_args"]}]=0 v47) ==== 137+0+0 (secure 0 0 0) 0x7fe6b0056df0 con 0x7fe6b80974d0 2024-03-28T14:08:22.818 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:22.815+0000 7fe6bf599640 1 -- 172.21.15.104:0/1111954363 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7fe69c067ff0 msgr2=0x7fe69c06a4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:22.818 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:22.815+0000 7fe6bf599640 1 --2- 172.21.15.104:0/1111954363 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7fe69c067ff0 0x7fe69c06a4b0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fe6b40af000 tx=0x7fe6b40046d0 comp rx=0 tx=0).stop 2024-03-28T14:08:22.819 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:22.815+0000 7fe6bf599640 1 -- 172.21.15.104:0/1111954363 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe6b80974d0 msgr2=0x7fe6b81e2060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:22.819 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:22.815+0000 7fe6bf599640 1 --2- 172.21.15.104:0/1111954363 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe6b80974d0 0x7fe6b81e2060 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7fe6b80706b0 tx=0x7fe6b000f4a0 comp rx=0 tx=0).stop 2024-03-28T14:08:22.819 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:22.815+0000 7fe6bf599640 1 -- 172.21.15.104:0/1111954363 shutdown_connections 2024-03-28T14:08:22.819 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:22.815+0000 7fe6bf599640 1 --2- 172.21.15.104:0/1111954363 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7fe69c067ff0 0x7fe69c06a4b0 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:22.819 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:22.815+0000 7fe6bf599640 1 --2- 172.21.15.104:0/1111954363 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe6b80974d0 0x7fe6b81e2060 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:22.819 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:22.815+0000 7fe6bf599640 1 --2- 172.21.15.104:0/1111954363 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fe6b8097050 0x7fe6b81e1b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:22.819 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:22.815+0000 7fe6bf599640 1 --2- 172.21.15.104:0/1111954363 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fe6a40039c0 0x7fe6b8099280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:22.819 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:22.815+0000 7fe6bf599640 1 -- 172.21.15.104:0/1111954363 >> 172.21.15.104:0/1111954363 conn(0x7fe6b80b95f0 msgr2=0x7fe6b809e1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T14:08:22.819 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:22.815+0000 7fe6bf599640 1 -- 172.21.15.104:0/1111954363 shutdown_connections 2024-03-28T14:08:22.819 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:22.815+0000 7fe6bf599640 1 -- 172.21.15.104:0/1111954363 wait complete. 2024-03-28T14:08:22.843 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-03-28T14:08:22.967 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:22.963+0000 7f5a0f06a640 1 Processor -- start 2024-03-28T14:08:22.967 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:22.963+0000 7f5a0f06a640 1 -- start start 2024-03-28T14:08:22.968 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:22.967+0000 7f5a0f06a640 1 --2- >> v2:172.21.15.171:3300/0 conn(0x7f5a0802c460 0x7f5a080b54e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:22.968 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:22.967+0000 7f5a0f06a640 1 --2- >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f5a081487e0 0x7f5a080b5a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:22.968 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:22.967+0000 7f5a0f06a640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5a0802c840 con 0x7f5a0809e8e0 2024-03-28T14:08:22.968 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:22.967+0000 7f5a0f06a640 1 -- --> v2:172.21.15.171:3300/0 -- mon_getmap magic: 0 -- 0x7f5a08147c40 con 0x7f5a0802c460 2024-03-28T14:08:22.968 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:22.967+0000 7f5a0f06a640 1 -- --> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] -- mon_getmap magic: 0 -- 0x7f5a080aec70 con 0x7f5a081487e0 2024-03-28T14:08:22.968 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:22.967+0000 7f5a0ce06640 1 --2- >> v2:172.21.15.171:3300/0 conn(0x7f5a0802c460 0x7f5a080b54e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:22.969 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:22.967+0000 7f5a0d607640 1 --2- >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f5a081487e0 0x7f5a080b5a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:22.969 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:22.967+0000 7f5a0d607640 1 --2- >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f5a081487e0 0x7f5a080b5a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.171:3301/0 says I am v2:172.21.15.104:39212/0 (socket says 172.21.15.104:39212) 2024-03-28T14:08:22.969 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:22.967+0000 7f5a0d607640 1 -- 172.21.15.104:0/699091487 learned_addr learned my addr 172.21.15.104:0/699091487 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-03-28T14:08:22.969 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:22.967+0000 7f5a07fff640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7f5a0809e8e0 0x7f5a081480d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.104:53562/0 (socket says 172.21.15.104:53562) 2024-03-28T14:08:22.969 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:22.967+0000 7f5a0d607640 1 -- 172.21.15.104:0/699091487 >> v2:172.21.15.171:3300/0 conn(0x7f5a0802c460 msgr2=0x7f5a080b54e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T14:08:22.969 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:22.967+0000 7f5a0d607640 1 --2- 172.21.15.104:0/699091487 >> v2:172.21.15.171:3300/0 conn(0x7f5a0802c460 0x7f5a080b54e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:22.970 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:22.967+0000 7f5a0d607640 1 -- 172.21.15.104:0/699091487 >> v1:172.21.15.171:6789/0 conn(0x7f5a0809e8e0 legacy=0x7f5a081480d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:22.970 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:22.967+0000 7f5a0d607640 1 -- 172.21.15.104:0/699091487 --> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5a080aef50 con 0x7f5a081487e0 2024-03-28T14:08:22.970 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:22.967+0000 7f5a0ce06640 1 --2- 172.21.15.104:0/699091487 >> v2:172.21.15.171:3300/0 conn(0x7f5a0802c460 0x7f5a080b54e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T14:08:22.971 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:22.967+0000 7f5a0d607640 1 --2- 172.21.15.104:0/699091487 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f5a081487e0 0x7f5a080b5a20 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f5a00009e50 tx=0x7f5a0001ca30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a02c15515ba94de6 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:08:22.972 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:22.971+0000 7f5a077fe640 1 -- 172.21.15.104:0/699091487 <== mon.2 v2:172.21.15.171:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a000a5600 con 0x7f5a081487e0 2024-03-28T14:08:22.972 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:22.971+0000 7f5a077fe640 1 -- 172.21.15.104:0/699091487 <== mon.2 v2:172.21.15.171:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5a000a57a0 con 0x7f5a081487e0 2024-03-28T14:08:22.972 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:22.971+0000 7f5a077fe640 1 -- 172.21.15.104:0/699091487 <== mon.2 v2:172.21.15.171:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a000a5b70 con 0x7f5a081487e0 2024-03-28T14:08:22.972 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:22.971+0000 7f5a0f06a640 1 -- 172.21.15.104:0/699091487 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f5a081487e0 msgr2=0x7f5a080b5a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:22.972 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:22.971+0000 7f5a0f06a640 1 --2- 172.21.15.104:0/699091487 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f5a081487e0 0x7f5a080b5a20 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f5a00009e50 tx=0x7f5a0001ca30 comp rx=0 tx=0).stop 2024-03-28T14:08:22.972 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:22.971+0000 7f5a0f06a640 1 -- 172.21.15.104:0/699091487 shutdown_connections 2024-03-28T14:08:22.972 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:22.971+0000 7f5a0f06a640 1 --2- 172.21.15.104:0/699091487 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f5a081487e0 0x7f5a080b5a20 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:22.972 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:22.971+0000 7f5a0f06a640 1 --2- 172.21.15.104:0/699091487 >> v2:172.21.15.171:3300/0 conn(0x7f5a0802c460 0x7f5a080b54e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:22.972 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:22.971+0000 7f5a0f06a640 1 -- 172.21.15.104:0/699091487 >> 172.21.15.104:0/699091487 conn(0x7f5a080b95f0 msgr2=0x7f5a080b99d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T14:08:22.973 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:22.971+0000 7f5a0f06a640 1 -- 172.21.15.104:0/699091487 shutdown_connections 2024-03-28T14:08:22.973 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:22.971+0000 7f5a0f06a640 1 -- 172.21.15.104:0/699091487 wait complete. 2024-03-28T14:08:22.973 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:22.971+0000 7f5a0f06a640 1 Processor -- start 2024-03-28T14:08:22.973 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:22.971+0000 7f5a0f06a640 1 -- start start 2024-03-28T14:08:22.976 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:22.971+0000 7f5a0f06a640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f5a0802c460 0x7f5a080b3000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:22.976 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:22.971+0000 7f5a0f06a640 1 --2- >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f5a0809e8e0 0x7f5a08099280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:22.976 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:22.971+0000 7f5a0f06a640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5a081487e0 0x7f5a0809a740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:22.976 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:22.971+0000 7f5a0f06a640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f5a0806dc20 con 0x7f5a081487e0 2024-03-28T14:08:22.976 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:22.971+0000 7f5a0f06a640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f5a0806daa0 con 0x7f5a0802c460 2024-03-28T14:08:22.976 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:22.971+0000 7f5a0f06a640 1 -- --> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] -- mon_getmap magic: 0 -- 0x7f5a0806dda0 con 0x7f5a0809e8e0 2024-03-28T14:08:22.976 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:22.971+0000 7f5a0d607640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5a081487e0 0x7f5a0809a740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:22.976 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:22.971+0000 7f5a0d607640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5a081487e0 0x7f5a0809a740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:42602/0 (socket says 172.21.15.104:42602) 2024-03-28T14:08:22.976 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:22.971+0000 7f5a0d607640 1 -- 172.21.15.104:0/66039401 learned_addr learned my addr 172.21.15.104:0/66039401 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-03-28T14:08:22.976 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:22.971+0000 7f5a0ce06640 1 --2- 172.21.15.104:0/66039401 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f5a0802c460 0x7f5a080b3000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:22.976 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:22.971+0000 7f5a07fff640 1 --2- 172.21.15.104:0/66039401 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f5a0809e8e0 0x7f5a08099280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:22.976 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:22.971+0000 7f5a0d607640 1 -- 172.21.15.104:0/66039401 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f5a0809e8e0 msgr2=0x7f5a08099280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:22.977 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:22.971+0000 7f5a0d607640 1 --2- 172.21.15.104:0/66039401 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f5a0809e8e0 0x7f5a08099280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:22.977 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:22.971+0000 7f5a0d607640 1 -- 172.21.15.104:0/66039401 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f5a0802c460 msgr2=0x7f5a080b3000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:22.977 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:22.971+0000 7f5a0d607640 1 --2- 172.21.15.104:0/66039401 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f5a0802c460 0x7f5a080b3000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:22.977 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:22.971+0000 7f5a0d607640 1 -- 172.21.15.104:0/66039401 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5a0809ae50 con 0x7f5a081487e0 2024-03-28T14:08:22.977 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:22.971+0000 7f5a07fff640 1 --2- 172.21.15.104:0/66039401 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f5a0809e8e0 0x7f5a08099280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T14:08:22.977 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:22.971+0000 7f5a0d607640 1 --2- 172.21.15.104:0/66039401 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5a081487e0 0x7f5a0809a740 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f5a000af040 tx=0x7f5a000197d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:08:22.977 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:22.971+0000 7f5a057fa640 1 -- 172.21.15.104:0/66039401 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a000bc070 con 0x7f5a081487e0 2024-03-28T14:08:22.977 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:22.971+0000 7f5a057fa640 1 -- 172.21.15.104:0/66039401 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5a000b8640 con 0x7f5a081487e0 2024-03-28T14:08:22.977 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:22.971+0000 7f5a0f06a640 1 -- 172.21.15.104:0/66039401 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5a080b37a0 con 0x7f5a081487e0 2024-03-28T14:08:22.977 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:22.971+0000 7f5a057fa640 1 -- 172.21.15.104:0/66039401 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a000b8a30 con 0x7f5a081487e0 2024-03-28T14:08:22.977 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:22.971+0000 7f5a0f06a640 1 -- 172.21.15.104:0/66039401 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5a0809b6f0 con 0x7f5a081487e0 2024-03-28T14:08:22.977 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:22.975+0000 7f5a0f06a640 1 -- 172.21.15.104:0/66039401 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5a080687c0 con 0x7f5a081487e0 2024-03-28T14:08:22.978 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:22.975+0000 7f5a057fa640 1 -- 172.21.15.104:0/66039401 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) ==== 87803+0+0 (secure 0 0 0) 0x7f5a000b8bd0 con 0x7f5a081487e0 2024-03-28T14:08:22.978 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:22.975+0000 7f5a057fa640 1 --2- 172.21.15.104:0/66039401 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f59e8067fc0 0x7f59e806a480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:22.978 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:22.975+0000 7f5a057fa640 1 -- 172.21.15.104:0/66039401 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f5a00121490 con 0x7f5a081487e0 2024-03-28T14:08:22.978 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:22.975+0000 7f5a0ce06640 1 --2- 172.21.15.104:0/66039401 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f59e8067fc0 0x7f59e806a480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:22.978 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:22.975+0000 7f5a0ce06640 1 --2- 172.21.15.104:0/66039401 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f59e8067fc0 0x7f59e806a480 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f59f8009580 tx=0x7f59f8009190 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:08:22.980 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:22.979+0000 7f5a057fa640 1 -- 172.21.15.104:0/66039401 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179117 (secure 0 0 0) 0x7f5a0809ae50 con 0x7f5a081487e0 2024-03-28T14:08:23.139 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:23.135+0000 7f5a0f06a640 1 -- 172.21.15.104:0/66039401 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) -- 0x7f5a0809e150 con 0x7f5a081487e0 2024-03-28T14:08:23.145 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:23.143+0000 7f5a057fa640 1 -- 172.21.15.104:0/66039401 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v10)=0 v10) ==== 122+0+0 (secure 0 0 0) 0x7f5a00016620 con 0x7f5a081487e0 2024-03-28T14:08:23.147 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:23.143+0000 7f5a0f06a640 1 -- 172.21.15.104:0/66039401 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f59e8067fc0 msgr2=0x7f59e806a480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:23.147 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:23.143+0000 7f5a0f06a640 1 --2- 172.21.15.104:0/66039401 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f59e8067fc0 0x7f59e806a480 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f59f8009580 tx=0x7f59f8009190 comp rx=0 tx=0).stop 2024-03-28T14:08:23.147 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:23.143+0000 7f5a0f06a640 1 -- 172.21.15.104:0/66039401 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5a081487e0 msgr2=0x7f5a0809a740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:23.148 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:23.143+0000 7f5a0f06a640 1 --2- 172.21.15.104:0/66039401 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5a081487e0 0x7f5a0809a740 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f5a000af040 tx=0x7f5a000197d0 comp rx=0 tx=0).stop 2024-03-28T14:08:23.148 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:23.143+0000 7f5a0f06a640 1 -- 172.21.15.104:0/66039401 shutdown_connections 2024-03-28T14:08:23.148 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:23.143+0000 7f5a0f06a640 1 --2- 172.21.15.104:0/66039401 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f59e8067fc0 0x7f59e806a480 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:23.148 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:23.143+0000 7f5a0f06a640 1 --2- 172.21.15.104:0/66039401 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5a081487e0 0x7f5a0809a740 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:23.148 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:23.143+0000 7f5a0f06a640 1 --2- 172.21.15.104:0/66039401 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f5a0809e8e0 0x7f5a08099280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:23.148 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:23.143+0000 7f5a0f06a640 1 --2- 172.21.15.104:0/66039401 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f5a0802c460 0x7f5a080b3000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:23.148 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:23.143+0000 7f5a0f06a640 1 -- 172.21.15.104:0/66039401 >> 172.21.15.104:0/66039401 conn(0x7f5a080b95f0 msgr2=0x7f5a080b5f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T14:08:23.148 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:23.143+0000 7f5a0f06a640 1 -- 172.21.15.104:0/66039401 shutdown_connections 2024-03-28T14:08:23.148 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:23.143+0000 7f5a0f06a640 1 -- 172.21.15.104:0/66039401 wait complete. 2024-03-28T14:08:23.164 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-03-28T14:08:23.164 DEBUG:teuthology.orchestra.run.smithi104:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-28T14:08:23.237 DEBUG:teuthology.orchestra.run.smithi171:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-28T14:08:23.298 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-03-28T14:08:23.298 DEBUG:teuthology.orchestra.run.smithi104:> set -ex 2024-03-28T14:08:23.298 DEBUG:teuthology.orchestra.run.smithi104:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-28T14:08:23.304 DEBUG:teuthology.orchestra.run.smithi104:> set -ex 2024-03-28T14:08:23.304 DEBUG:teuthology.orchestra.run.smithi104:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-28T14:08:23.353 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-28T14:08:23.515 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:23.511+0000 7f6843bc1640 1 Processor -- start 2024-03-28T14:08:23.515 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:23.511+0000 7f6843bc1640 1 -- start start 2024-03-28T14:08:23.516 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:23.515+0000 7f6843bc1640 1 --2- >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f683c06b830 0x7f683c09fa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:23.517 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:23.515+0000 7f6843bc1640 1 --2- >> v2:172.21.15.171:3300/0 conn(0x7f683c09ff50 0x7f683c072ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:23.517 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:23.515+0000 7f6843bc1640 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f683c0733e0 0x7f683c06dad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:23.517 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:23.515+0000 7f6843bc1640 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 -- 0x7f683c06bc10 con 0x7f683c0733e0 2024-03-28T14:08:23.517 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:23.515+0000 7f6843bc1640 1 -- --> v2:172.21.15.171:3300/0 -- mon_getmap magic: 0 -- 0x7f683c0b5a10 con 0x7f683c09ff50 2024-03-28T14:08:23.517 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:23.515+0000 7f6843bc1640 1 -- --> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] -- mon_getmap magic: 0 -- 0x7f683c0b5cf0 con 0x7f683c06b830 2024-03-28T14:08:23.517 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:23.515+0000 7f684215e640 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f683c0733e0 0x7f683c06dad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:23.517 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:23.515+0000 7f684215e640 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f683c0733e0 0x7f683c06dad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:42616/0 (socket says 172.21.15.104:42616) 2024-03-28T14:08:23.517 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:23.515+0000 7f684215e640 1 -- 172.21.15.104:0/3141396812 learned_addr learned my addr 172.21.15.104:0/3141396812 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-03-28T14:08:23.517 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:23.515+0000 7f684195d640 1 --2- 172.21.15.104:0/3141396812 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f683c06b830 0x7f683c09fa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:23.517 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:23.515+0000 7f684215e640 1 -- 172.21.15.104:0/3141396812 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f683c06b830 msgr2=0x7f683c09fa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T14:08:23.517 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:23.515+0000 7f684215e640 1 --2- 172.21.15.104:0/3141396812 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f683c06b830 0x7f683c09fa10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:23.517 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:23.515+0000 7f684215e640 1 -- 172.21.15.104:0/3141396812 >> v2:172.21.15.171:3300/0 conn(0x7f683c09ff50 msgr2=0x7f683c072ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T14:08:23.517 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:23.515+0000 7f684115c640 1 --2- 172.21.15.104:0/3141396812 >> v2:172.21.15.171:3300/0 conn(0x7f683c09ff50 0x7f683c072ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:23.518 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:23.515+0000 7f684215e640 1 --2- 172.21.15.104:0/3141396812 >> v2:172.21.15.171:3300/0 conn(0x7f683c09ff50 0x7f683c072ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:23.518 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:23.515+0000 7f684215e640 1 -- 172.21.15.104:0/3141396812 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f683c0b5fd0 con 0x7f683c0733e0 2024-03-28T14:08:23.519 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:23.515+0000 7f684215e640 1 --2- 172.21.15.104:0/3141396812 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f683c0733e0 0x7f683c06dad0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f683c06ebf0 tx=0x7f683801c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a8fd96e39d86c2cb server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:08:23.519 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:23.515+0000 7f684095b640 1 -- 172.21.15.104:0/3141396812 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f68380a5550 con 0x7f683c0733e0 2024-03-28T14:08:23.519 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:23.515+0000 7f684095b640 1 -- 172.21.15.104:0/3141396812 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f68380a56f0 con 0x7f683c0733e0 2024-03-28T14:08:23.519 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:23.515+0000 7f684095b640 1 -- 172.21.15.104:0/3141396812 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f68380a5ac0 con 0x7f683c0733e0 2024-03-28T14:08:23.520 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:23.519+0000 7f6843bc1640 1 -- 172.21.15.104:0/3141396812 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f683c0733e0 msgr2=0x7f683c06dad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:23.520 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:23.519+0000 7f6843bc1640 1 --2- 172.21.15.104:0/3141396812 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f683c0733e0 0x7f683c06dad0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f683c06ebf0 tx=0x7f683801c8d0 comp rx=0 tx=0).stop 2024-03-28T14:08:23.520 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:23.519+0000 7f6843bc1640 1 -- 172.21.15.104:0/3141396812 shutdown_connections 2024-03-28T14:08:23.520 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:23.519+0000 7f6843bc1640 1 --2- 172.21.15.104:0/3141396812 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f683c0733e0 0x7f683c06dad0 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:23.520 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:23.519+0000 7f6843bc1640 1 --2- 172.21.15.104:0/3141396812 >> v2:172.21.15.171:3300/0 conn(0x7f683c09ff50 0x7f683c072ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:23.520 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:23.519+0000 7f6843bc1640 1 --2- 172.21.15.104:0/3141396812 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f683c06b830 0x7f683c09fa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:23.521 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:23.519+0000 7f6843bc1640 1 -- 172.21.15.104:0/3141396812 >> 172.21.15.104:0/3141396812 conn(0x7f683c0ba400 msgr2=0x7f683c0ba810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T14:08:23.521 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:23.519+0000 7f6843bc1640 1 -- 172.21.15.104:0/3141396812 shutdown_connections 2024-03-28T14:08:23.521 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:23.519+0000 7f6843bc1640 1 -- 172.21.15.104:0/3141396812 wait complete. 2024-03-28T14:08:23.523 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:23.519+0000 7f6843bc1640 1 Processor -- start 2024-03-28T14:08:23.523 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:23.519+0000 7f6843bc1640 1 -- start start 2024-03-28T14:08:23.523 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:23.519+0000 7f6843bc1640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f683c06b830 0x7f683c0b9140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:23.524 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:23.519+0000 7f6843bc1640 1 --2- >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f683c0733e0 0x7f683c072500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:23.524 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:23.519+0000 7f6843bc1640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f683c09ff50 0x7f683c0afcc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:23.524 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:23.519+0000 7f6843bc1640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f683c152170 con 0x7f683c09ff50 2024-03-28T14:08:23.524 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:23.519+0000 7f6843bc1640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f683c151ff0 con 0x7f683c06b830 2024-03-28T14:08:23.524 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:23.519+0000 7f6843bc1640 1 -- --> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] -- mon_getmap magic: 0 -- 0x7f683c1522f0 con 0x7f683c0733e0 2024-03-28T14:08:23.524 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:23.519+0000 7f684215e640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f683c09ff50 0x7f683c0afcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:23.524 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:23.519+0000 7f684215e640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f683c09ff50 0x7f683c0afcc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:42626/0 (socket says 172.21.15.104:42626) 2024-03-28T14:08:23.524 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:23.519+0000 7f684215e640 1 -- 172.21.15.104:0/2065990249 learned_addr learned my addr 172.21.15.104:0/2065990249 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-03-28T14:08:23.524 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:23.519+0000 7f684195d640 1 --2- 172.21.15.104:0/2065990249 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f683c06b830 0x7f683c0b9140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:23.524 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:23.519+0000 7f684115c640 1 --2- 172.21.15.104:0/2065990249 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f683c0733e0 0x7f683c072500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:23.524 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:23.519+0000 7f684215e640 1 -- 172.21.15.104:0/2065990249 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f683c0733e0 msgr2=0x7f683c072500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:23.525 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:23.519+0000 7f684215e640 1 --2- 172.21.15.104:0/2065990249 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f683c0733e0 0x7f683c072500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:23.525 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:23.519+0000 7f684215e640 1 -- 172.21.15.104:0/2065990249 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f683c06b830 msgr2=0x7f683c0b9140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:23.525 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:23.519+0000 7f684215e640 1 --2- 172.21.15.104:0/2065990249 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f683c06b830 0x7f683c0b9140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:23.525 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:23.519+0000 7f684215e640 1 -- 172.21.15.104:0/2065990249 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f683c070ff0 con 0x7f683c09ff50 2024-03-28T14:08:23.525 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:23.519+0000 7f684195d640 1 --2- 172.21.15.104:0/2065990249 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f683c06b830 0x7f683c0b9140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T14:08:23.525 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:23.519+0000 7f684215e640 1 --2- 172.21.15.104:0/2065990249 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f683c09ff50 0x7f683c0afcc0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f683801d170 tx=0x7f6838019670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:08:23.525 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:23.519+0000 7f68327fc640 1 -- 172.21.15.104:0/2065990249 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f68380bc020 con 0x7f683c09ff50 2024-03-28T14:08:23.525 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:23.519+0000 7f68327fc640 1 -- 172.21.15.104:0/2065990249 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f68380b8670 con 0x7f683c09ff50 2024-03-28T14:08:23.525 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:23.519+0000 7f68327fc640 1 -- 172.21.15.104:0/2065990249 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6838009510 con 0x7f683c09ff50 2024-03-28T14:08:23.525 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:23.519+0000 7f6843bc1640 1 -- 172.21.15.104:0/2065990249 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f683c0af550 con 0x7f683c09ff50 2024-03-28T14:08:23.525 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:23.519+0000 7f6843bc1640 1 -- 172.21.15.104:0/2065990249 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f683c0af1d0 con 0x7f683c09ff50 2024-03-28T14:08:23.526 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:23.523+0000 7f6843bc1640 1 -- 172.21.15.104:0/2065990249 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f683c0687c0 con 0x7f683c09ff50 2024-03-28T14:08:23.526 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:23.523+0000 7f68327fc640 1 -- 172.21.15.104:0/2065990249 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) ==== 87803+0+0 (secure 0 0 0) 0x7f68380bf020 con 0x7f683c09ff50 2024-03-28T14:08:23.526 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:23.523+0000 7f68327fc640 1 --2- 172.21.15.104:0/2065990249 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f6820067fc0 0x7f682006a480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:23.526 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:23.523+0000 7f68327fc640 1 -- 172.21.15.104:0/2065990249 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f6838121500 con 0x7f683c09ff50 2024-03-28T14:08:23.526 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:23.523+0000 7f684195d640 1 --2- 172.21.15.104:0/2065990249 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f6820067fc0 0x7f682006a480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:23.526 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:23.523+0000 7f684195d640 1 --2- 172.21.15.104:0/2065990249 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f6820067fc0 0x7f682006a480 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f6834006fd0 tx=0x7f6834008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:08:23.529 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:23.527+0000 7f68327fc640 1 -- 172.21.15.104:0/2065990249 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179117 (secure 0 0 0) 0x7f68380c6030 con 0x7f683c09ff50 2024-03-28T14:08:23.693 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:23.691+0000 7f6843bc1640 1 -- 172.21.15.104:0/2065990249 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f683c144f90 con 0x7f683c09ff50 2024-03-28T14:08:23.694 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:23.691+0000 7f68327fc640 1 -- 172.21.15.104:0/2065990249 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) ==== 74+0+16629 (secure 0 0 0) 0x7f68380165e0 con 0x7f683c09ff50 2024-03-28T14:08:23.694 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-03-28T14:08:23.694 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":21,"fsid":"428a90f2-3351-435f-8fcf-f7e66c78a13e","created":"2024-03-28T14:07:34.862458+0000","modified":"2024-03-28T14:08:00.671587+0000","last_up_change":"2024-03-28T14:07:46.028122+0000","last_in_change":"2024-03-28T14:07:37.157486+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-28T14:07:47.622988+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-28T14:07:57.147450+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-28T14:07:57.990975+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"1cb70540-4e42-402f-a446-aaec4ca83007","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6800","nonce":3745690918},{"type":"v1","addr":"172.21.15.104:6801","nonce":3745690918}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":3745690918},{"type":"v1","addr":"172.21.15.104:6803","nonce":3745690918}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":3745690918},{"type":"v1","addr":"172.21.15.104:6807","nonce":3745690918}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":3745690918},{"type":"v1","addr":"172.21.15.104:6805","nonce":3745690918}]},"public_addr":"172.21.15.104:6801/3745690918","cluster_addr":"172.21.15.104:6803/3745690918","heartbeat_back_addr":"172.21.15.104:6807/3745690918","heartbeat_front_addr":"172.21.15.104:6805/3745690918","state":["exists","up"]},{"osd":1,"uuid":"3ab7afdd-b980-418c-81f2-004e16de3dca","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":19438674},{"type":"v1","addr":"172.21.15.104:6810","nonce":19438674}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":19438674},{"type":"v1","addr":"172.21.15.104:6814","nonce":19438674}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":19438674},{"type":"v1","addr":"172.21.15.104:6823","nonce":19438674}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":19438674},{"type":"v1","addr":"172.21.15.104:6819","nonce":19438674}]},"public_addr":"172.21.15.104:6810/19438674","cluster_addr":"172.21.15.104:6814/19438674","heartbeat_back_addr":"172.21.15.104:6823/19438674","heartbeat_front_addr":"172.21.15.104:6819/19438674","state":["exists","up"]},{"osd":2,"uuid":"3e2651c8-ecae-4a9e-a053-27240a79eac2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":2352012626},{"type":"v1","addr":"172.21.15.104:6825","nonce":2352012626}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6826","nonce":2352012626},{"type":"v1","addr":"172.21.15.104:6827","nonce":2352012626}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":2352012626},{"type":"v1","addr":"172.21.15.104:6831","nonce":2352012626}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6828","nonce":2352012626},{"type":"v1","addr":"172.21.15.104:6829","nonce":2352012626}]},"public_addr":"172.21.15.104:6825/2352012626","cluster_addr":"172.21.15.104:6827/2352012626","heartbeat_back_addr":"172.21.15.104:6831/2352012626","heartbeat_front_addr":"172.21.15.104:6829/2352012626","state":["exists","up"]},{"osd":3,"uuid":"925c7e0a-38ef-47cd-b288-8bec8456fb17","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6809","nonce":3568347381},{"type":"v1","addr":"172.21.15.104:6811","nonce":3568347381}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6813","nonce":3568347381},{"type":"v1","addr":"172.21.15.104:6815","nonce":3568347381}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":3568347381},{"type":"v1","addr":"172.21.15.104:6822","nonce":3568347381}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6817","nonce":3568347381},{"type":"v1","addr":"172.21.15.104:6818","nonce":3568347381}]},"public_addr":"172.21.15.104:6811/3568347381","cluster_addr":"172.21.15.104:6815/3568347381","heartbeat_back_addr":"172.21.15.104:6822/3568347381","heartbeat_front_addr":"172.21.15.104:6818/3568347381","state":["exists","up"]},{"osd":4,"uuid":"4a8fac86-e595-4a0e-96e3-57edb3cc3f57","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6800","nonce":2107001077},{"type":"v1","addr":"172.21.15.171:6801","nonce":2107001077}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6802","nonce":2107001077},{"type":"v1","addr":"172.21.15.171:6803","nonce":2107001077}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6806","nonce":2107001077},{"type":"v1","addr":"172.21.15.171:6807","nonce":2107001077}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6804","nonce":2107001077},{"type":"v1","addr":"172.21.15.171:6805","nonce":2107001077}]},"public_addr":"172.21.15.171:6801/2107001077","cluster_addr":"172.21.15.171:6803/2107001077","heartbeat_back_addr":"172.21.15.171:6807/2107001077","heartbeat_front_addr":"172.21.15.171:6805/2107001077","state":["exists","up"]},{"osd":5,"uuid":"e48d50b0-beae-40ae-b15a-9a84dbfe88ab","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6808","nonce":2934826925},{"type":"v1","addr":"172.21.15.171:6810","nonce":2934826925}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6812","nonce":2934826925},{"type":"v1","addr":"172.21.15.171:6814","nonce":2934826925}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6819","nonce":2934826925},{"type":"v1","addr":"172.21.15.171:6821","nonce":2934826925}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6816","nonce":2934826925},{"type":"v1","addr":"172.21.15.171:6817","nonce":2934826925}]},"public_addr":"172.21.15.171:6810/2934826925","cluster_addr":"172.21.15.171:6814/2934826925","heartbeat_back_addr":"172.21.15.171:6821/2934826925","heartbeat_front_addr":"172.21.15.171:6817/2934826925","state":["exists","up"]},{"osd":6,"uuid":"5e7c3292-9936-4f35-b267-aed9d4e49323","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6824","nonce":436742919},{"type":"v1","addr":"172.21.15.171:6825","nonce":436742919}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6826","nonce":436742919},{"type":"v1","addr":"172.21.15.171:6827","nonce":436742919}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6830","nonce":436742919},{"type":"v1","addr":"172.21.15.171:6831","nonce":436742919}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6828","nonce":436742919},{"type":"v1","addr":"172.21.15.171:6829","nonce":436742919}]},"public_addr":"172.21.15.171:6825/436742919","cluster_addr":"172.21.15.171:6827/436742919","heartbeat_back_addr":"172.21.15.171:6831/436742919","heartbeat_front_addr":"172.21.15.171:6829/436742919","state":["exists","up"]},{"osd":7,"uuid":"339b8414-d7f0-4fb0-a446-efaf635cb7ff","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6809","nonce":2055631175},{"type":"v1","addr":"172.21.15.171:6811","nonce":2055631175}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6813","nonce":2055631175},{"type":"v1","addr":"172.21.15.171:6815","nonce":2055631175}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6822","nonce":2055631175},{"type":"v1","addr":"172.21.15.171:6823","nonce":2055631175}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6818","nonce":2055631175},{"type":"v1","addr":"172.21.15.171:6820","nonce":2055631175}]},"public_addr":"172.21.15.171:6811/2055631175","cluster_addr":"172.21.15.171:6815/2055631175","heartbeat_back_addr":"172.21.15.171:6823/2055631175","heartbeat_front_addr":"172.21.15.171:6820/2055631175","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T14:07:42.668881+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T14:07:42.708397+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T14:07:42.634191+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T14:07:42.733662+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T14:07:42.651541+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T14:07:42.783181+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T14:07:42.766310+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T14:07:42.670797+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-28T14:08:23.697 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:23.695+0000 7f6843bc1640 1 -- 172.21.15.104:0/2065990249 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f6820067fc0 msgr2=0x7f682006a480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:23.697 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:23.695+0000 7f6843bc1640 1 --2- 172.21.15.104:0/2065990249 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f6820067fc0 0x7f682006a480 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f6834006fd0 tx=0x7f6834008040 comp rx=0 tx=0).stop 2024-03-28T14:08:23.697 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:23.695+0000 7f6843bc1640 1 -- 172.21.15.104:0/2065990249 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f683c09ff50 msgr2=0x7f683c0afcc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:23.697 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:23.695+0000 7f6843bc1640 1 --2- 172.21.15.104:0/2065990249 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f683c09ff50 0x7f683c0afcc0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f683801d170 tx=0x7f6838019670 comp rx=0 tx=0).stop 2024-03-28T14:08:23.697 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:23.695+0000 7f6843bc1640 1 -- 172.21.15.104:0/2065990249 shutdown_connections 2024-03-28T14:08:23.697 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:23.695+0000 7f6843bc1640 1 --2- 172.21.15.104:0/2065990249 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f6820067fc0 0x7f682006a480 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:23.698 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:23.695+0000 7f6843bc1640 1 --2- 172.21.15.104:0/2065990249 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f683c09ff50 0x7f683c0afcc0 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:23.698 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:23.695+0000 7f6843bc1640 1 --2- 172.21.15.104:0/2065990249 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f683c0733e0 0x7f683c072500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:23.698 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:23.695+0000 7f6843bc1640 1 --2- 172.21.15.104:0/2065990249 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f683c06b830 0x7f683c0b9140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:23.698 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:23.695+0000 7f6843bc1640 1 -- 172.21.15.104:0/2065990249 >> 172.21.15.104:0/2065990249 conn(0x7f683c0ba400 msgr2=0x7f683c0b6a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T14:08:23.698 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:23.695+0000 7f6843bc1640 1 -- 172.21.15.104:0/2065990249 shutdown_connections 2024-03-28T14:08:23.698 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:23.695+0000 7f6843bc1640 1 -- 172.21.15.104:0/2065990249 wait complete. 2024-03-28T14:08:23.718 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-28T14:07:47.622988+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-28T14:07:57.147450+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-28T14:07:57.990975+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-28T14:08:23.721 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-28T14:08:23.845 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:23.843+0000 7fc9c9f84640 1 Processor -- start 2024-03-28T14:08:23.845 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:23.843+0000 7fc9c9f84640 1 -- start start 2024-03-28T14:08:23.845 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:23.843+0000 7fc9c9f84640 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fc9c402c460 0x7fc9c40b3ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:23.845 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:23.843+0000 7fc9c9f84640 1 --2- >> v2:172.21.15.171:3300/0 conn(0x7fc9c40b4530 0x7fc9c413f540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:23.845 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:23.843+0000 7fc9c9f84640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc9c402c840 con 0x7fc9c413fa80 2024-03-28T14:08:23.845 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:23.843+0000 7fc9c9f84640 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 -- 0x7fc9c40b4f20 con 0x7fc9c402c460 2024-03-28T14:08:23.845 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:23.843+0000 7fc9c9f84640 1 -- --> v2:172.21.15.171:3300/0 -- mon_getmap magic: 0 -- 0x7fc9c40b5200 con 0x7fc9c40b4530 2024-03-28T14:08:23.845 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:23.843+0000 7fc9c37fe640 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fc9c402c460 0x7fc9c40b3ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:23.846 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:23.843+0000 7fc9c3fff640 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7fc9c413fa80 0x7fc9c41456c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:46194/0 (socket says 172.21.15.104:46194) 2024-03-28T14:08:23.846 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:23.843+0000 7fc9c3fff640 1 -- 172.21.15.104:0/289078890 learned_addr learned my addr 172.21.15.104:0/289078890 (peer_addr_for_me v1:172.21.15.104:0/0) 2024-03-28T14:08:23.846 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:23.843+0000 7fc9c2ffd640 1 --2- 172.21.15.104:0/289078890 >> v2:172.21.15.171:3300/0 conn(0x7fc9c40b4530 0x7fc9c413f540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:23.846 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:23.843+0000 7fc9c37fe640 1 -- 172.21.15.104:0/289078890 >> v2:172.21.15.171:3300/0 conn(0x7fc9c40b4530 msgr2=0x7fc9c413f540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T14:08:23.846 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:23.843+0000 7fc9c37fe640 1 --2- 172.21.15.104:0/289078890 >> v2:172.21.15.171:3300/0 conn(0x7fc9c40b4530 0x7fc9c413f540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:23.847 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:23.843+0000 7fc9c37fe640 1 -- 172.21.15.104:0/289078890 >> v1:172.21.15.104:6789/0 conn(0x7fc9c413fa80 legacy=0x7fc9c41456c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:23.847 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:23.843+0000 7fc9c37fe640 1 -- 172.21.15.104:0/289078890 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc9c40b54e0 con 0x7fc9c402c460 2024-03-28T14:08:23.847 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:23.847+0000 7fc9c37fe640 1 --2- 172.21.15.104:0/289078890 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc9c402c460 0x7fc9c40b3ff0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7fc9c4070950 tx=0x7fc9b801c3f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=505d2da5694c3b5f server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:08:23.848 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:23.847+0000 7fc9c27fc640 1 -- 172.21.15.104:0/289078890 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc9b80b0070 con 0x7fc9c402c460 2024-03-28T14:08:23.848 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:23.847+0000 7fc9c27fc640 1 -- 172.21.15.104:0/289078890 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc9b80ab600 con 0x7fc9c402c460 2024-03-28T14:08:23.849 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:23.847+0000 7fc9c27fc640 1 -- 172.21.15.104:0/289078890 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc9b80ab9d0 con 0x7fc9c402c460 2024-03-28T14:08:23.849 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:23.847+0000 7fc9c9f84640 1 -- 172.21.15.104:0/289078890 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc9c402c460 msgr2=0x7fc9c40b3ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:23.849 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:23.847+0000 7fc9c9f84640 1 --2- 172.21.15.104:0/289078890 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc9c402c460 0x7fc9c40b3ff0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7fc9c4070950 tx=0x7fc9b801c3f0 comp rx=0 tx=0).stop 2024-03-28T14:08:23.849 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:23.847+0000 7fc9c9f84640 1 -- 172.21.15.104:0/289078890 shutdown_connections 2024-03-28T14:08:23.849 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:23.847+0000 7fc9c9f84640 1 --2- 172.21.15.104:0/289078890 >> v2:172.21.15.171:3300/0 conn(0x7fc9c40b4530 0x7fc9c413f540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:23.850 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:23.847+0000 7fc9c9f84640 1 --2- 172.21.15.104:0/289078890 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc9c402c460 0x7fc9c40b3ff0 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:23.850 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:23.847+0000 7fc9c9f84640 1 -- 172.21.15.104:0/289078890 >> 172.21.15.104:0/289078890 conn(0x7fc9c40b95f0 msgr2=0x7fc9c40b99d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T14:08:23.850 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:23.847+0000 7fc9c9f84640 1 -- 172.21.15.104:0/289078890 shutdown_connections 2024-03-28T14:08:23.850 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:23.847+0000 7fc9c9f84640 1 -- 172.21.15.104:0/289078890 wait complete. 2024-03-28T14:08:23.852 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:23.847+0000 7fc9c9f84640 1 Processor -- start 2024-03-28T14:08:23.852 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:23.847+0000 7fc9c9f84640 1 -- start start 2024-03-28T14:08:23.852 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:23.847+0000 7fc9c9f84640 1 --2- >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fc9c402c460 0x7fc9c413cd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:23.852 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:23.847+0000 7fc9c9f84640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fc9c40b4530 0x7fc9c4143ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:23.852 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:23.847+0000 7fc9c9f84640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc9c413fa80 0x7fc9c4144530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:23.853 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:23.847+0000 7fc9c9f84640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7fc9c4152b30 con 0x7fc9c413fa80 2024-03-28T14:08:23.853 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:23.847+0000 7fc9c9f84640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7fc9c41529b0 con 0x7fc9c40b4530 2024-03-28T14:08:23.853 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:23.847+0000 7fc9c9f84640 1 -- --> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] -- mon_getmap magic: 0 -- 0x7fc9c4152cb0 con 0x7fc9c402c460 2024-03-28T14:08:23.853 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:23.847+0000 7fc9c3fff640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc9c413fa80 0x7fc9c4144530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:23.853 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:23.847+0000 7fc9c3fff640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc9c413fa80 0x7fc9c4144530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:42638/0 (socket says 172.21.15.104:42638) 2024-03-28T14:08:23.853 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:23.847+0000 7fc9c3fff640 1 -- 172.21.15.104:0/3235343091 learned_addr learned my addr 172.21.15.104:0/3235343091 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-03-28T14:08:23.853 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:23.847+0000 7fc9c2ffd640 1 --2- 172.21.15.104:0/3235343091 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fc9c40b4530 0x7fc9c4143ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:23.853 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:23.847+0000 7fc9c37fe640 1 --2- 172.21.15.104:0/3235343091 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fc9c402c460 0x7fc9c413cd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:23.853 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:23.847+0000 7fc9c3fff640 1 -- 172.21.15.104:0/3235343091 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fc9c402c460 msgr2=0x7fc9c413cd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:23.853 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:23.847+0000 7fc9c3fff640 1 --2- 172.21.15.104:0/3235343091 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fc9c402c460 0x7fc9c413cd50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:23.853 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:23.847+0000 7fc9c3fff640 1 -- 172.21.15.104:0/3235343091 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fc9c40b4530 msgr2=0x7fc9c4143ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:23.853 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:23.847+0000 7fc9c3fff640 1 --2- 172.21.15.104:0/3235343091 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fc9c40b4530 0x7fc9c4143ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:23.853 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:23.847+0000 7fc9c3fff640 1 -- 172.21.15.104:0/3235343091 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc9c4071510 con 0x7fc9c413fa80 2024-03-28T14:08:23.853 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:23.847+0000 7fc9c2ffd640 1 --2- 172.21.15.104:0/3235343091 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fc9c40b4530 0x7fc9c4143ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T14:08:23.854 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:23.847+0000 7fc9c3fff640 1 --2- 172.21.15.104:0/3235343091 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc9c413fa80 0x7fc9c4144530 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fc9bc00cbc0 tx=0x7fc9bc00a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:08:23.854 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:23.847+0000 7fc9abfff640 1 -- 172.21.15.104:0/3235343091 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc9bc017070 con 0x7fc9c413fa80 2024-03-28T14:08:23.854 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:23.847+0000 7fc9abfff640 1 -- 172.21.15.104:0/3235343091 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc9bc00ace0 con 0x7fc9c413fa80 2024-03-28T14:08:23.854 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:23.847+0000 7fc9abfff640 1 -- 172.21.15.104:0/3235343091 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc9bc012600 con 0x7fc9c413fa80 2024-03-28T14:08:23.854 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:23.847+0000 7fc9c9f84640 1 -- 172.21.15.104:0/3235343091 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc9c413d710 con 0x7fc9c413fa80 2024-03-28T14:08:23.854 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:23.851+0000 7fc9c9f84640 1 -- 172.21.15.104:0/3235343091 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc9c4136fe0 con 0x7fc9c413fa80 2024-03-28T14:08:23.854 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:23.851+0000 7fc9c9f84640 1 -- 172.21.15.104:0/3235343091 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc9c40687c0 con 0x7fc9c413fa80 2024-03-28T14:08:23.855 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:23.851+0000 7fc9abfff640 1 -- 172.21.15.104:0/3235343091 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) ==== 87803+0+0 (secure 0 0 0) 0x7fc9bc01a070 con 0x7fc9c413fa80 2024-03-28T14:08:23.855 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:23.851+0000 7fc9abfff640 1 --2- 172.21.15.104:0/3235343091 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7fc9b0067ff0 0x7fc9b006a4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:23.855 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:23.851+0000 7fc9abfff640 1 -- 172.21.15.104:0/3235343091 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7fc9bc0860e0 con 0x7fc9c413fa80 2024-03-28T14:08:23.855 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:23.851+0000 7fc9c37fe640 1 --2- 172.21.15.104:0/3235343091 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7fc9b0067ff0 0x7fc9b006a4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:23.855 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:23.851+0000 7fc9c37fe640 1 --2- 172.21.15.104:0/3235343091 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7fc9b0067ff0 0x7fc9b006a4b0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fc9b80ad000 tx=0x7fc9b8008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:08:23.859 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:23.859+0000 7fc9abfff640 1 -- 172.21.15.104:0/3235343091 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179117 (secure 0 0 0) 0x7fc9bc0562d0 con 0x7fc9c413fa80 2024-03-28T14:08:24.009 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:24.007+0000 7fc9c9f84640 1 -- 172.21.15.104:0/3235343091 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fc9c4143250 con 0x7fc9c413fa80 2024-03-28T14:08:24.011 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:24.007+0000 7fc9abfff640 1 -- 172.21.15.104:0/3235343091 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v21) ==== 93+0+10 (secure 0 0 0) 0x7fc9bc05a5d0 con 0x7fc9c413fa80 2024-03-28T14:08:24.012 INFO:teuthology.orchestra.run.smithi104.stdout:pg_num: 1 2024-03-28T14:08:24.014 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:24.011+0000 7fc9c9f84640 1 -- 172.21.15.104:0/3235343091 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7fc9b0067ff0 msgr2=0x7fc9b006a4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:24.014 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:24.011+0000 7fc9c9f84640 1 --2- 172.21.15.104:0/3235343091 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7fc9b0067ff0 0x7fc9b006a4b0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fc9b80ad000 tx=0x7fc9b8008040 comp rx=0 tx=0).stop 2024-03-28T14:08:24.014 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:24.011+0000 7fc9c9f84640 1 -- 172.21.15.104:0/3235343091 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc9c413fa80 msgr2=0x7fc9c4144530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:24.014 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:24.011+0000 7fc9c9f84640 1 --2- 172.21.15.104:0/3235343091 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc9c413fa80 0x7fc9c4144530 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fc9bc00cbc0 tx=0x7fc9bc00a430 comp rx=0 tx=0).stop 2024-03-28T14:08:24.014 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:24.011+0000 7fc9c9f84640 1 -- 172.21.15.104:0/3235343091 shutdown_connections 2024-03-28T14:08:24.014 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:24.011+0000 7fc9c9f84640 1 --2- 172.21.15.104:0/3235343091 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7fc9b0067ff0 0x7fc9b006a4b0 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:24.015 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:24.011+0000 7fc9c9f84640 1 --2- 172.21.15.104:0/3235343091 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc9c413fa80 0x7fc9c4144530 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:24.015 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:24.011+0000 7fc9c9f84640 1 --2- 172.21.15.104:0/3235343091 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fc9c40b4530 0x7fc9c4143ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:24.015 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:24.011+0000 7fc9c9f84640 1 --2- 172.21.15.104:0/3235343091 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fc9c402c460 0x7fc9c413cd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:24.015 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:24.011+0000 7fc9c9f84640 1 -- 172.21.15.104:0/3235343091 >> 172.21.15.104:0/3235343091 conn(0x7fc9c40b95f0 msgr2=0x7fc9c413dfc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T14:08:24.015 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:24.011+0000 7fc9c9f84640 1 -- 172.21.15.104:0/3235343091 shutdown_connections 2024-03-28T14:08:24.015 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:24.011+0000 7fc9c9f84640 1 -- 172.21.15.104:0/3235343091 wait complete. 2024-03-28T14:08:24.027 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-28T14:08:24.140 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:24.139+0000 7f3c608e3640 1 Processor -- start 2024-03-28T14:08:24.140 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:24.139+0000 7f3c608e3640 1 -- start start 2024-03-28T14:08:24.142 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:24.139+0000 7f3c608e3640 1 --2- >> v2:172.21.15.171:3300/0 conn(0x7f3c5c02c460 0x7f3c5c06d020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:24.142 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:24.139+0000 7f3c608e3640 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f3c5c0afce0 0x7f3c5c14b780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:24.142 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:24.139+0000 7f3c608e3640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3c5c06d560 con 0x7f3c5c06e400 2024-03-28T14:08:24.142 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:24.139+0000 7f3c608e3640 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 -- 0x7f3c5c0af590 con 0x7f3c5c0afce0 2024-03-28T14:08:24.142 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:24.139+0000 7f3c608e3640 1 -- --> v2:172.21.15.171:3300/0 -- mon_getmap magic: 0 -- 0x7f3c5c0af850 con 0x7f3c5c02c460 2024-03-28T14:08:24.142 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:24.139+0000 7f3c5ad9d640 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f3c5c0afce0 0x7f3c5c14b780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:24.142 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:24.139+0000 7f3c5ad9d640 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f3c5c0afce0 0x7f3c5c14b780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:42642/0 (socket says 172.21.15.104:42642) 2024-03-28T14:08:24.143 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:24.139+0000 7f3c5ad9d640 1 -- 172.21.15.104:0/819244207 learned_addr learned my addr 172.21.15.104:0/819244207 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-03-28T14:08:24.143 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:24.139+0000 7f3c5ad9d640 1 -- 172.21.15.104:0/819244207 >> v2:172.21.15.171:3300/0 conn(0x7f3c5c02c460 msgr2=0x7f3c5c06d020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T14:08:24.143 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:24.139+0000 7f3c5a59c640 1 --2- 172.21.15.104:0/819244207 >> v2:172.21.15.171:3300/0 conn(0x7f3c5c02c460 0x7f3c5c06d020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:24.143 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:24.139+0000 7f3c5ad9d640 1 --2- 172.21.15.104:0/819244207 >> v2:172.21.15.171:3300/0 conn(0x7f3c5c02c460 0x7f3c5c06d020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:24.143 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:24.139+0000 7f3c5ad9d640 1 -- 172.21.15.104:0/819244207 >> v1:172.21.15.171:6789/0 conn(0x7f3c5c06e400 legacy=0x7f3c5c14b070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T14:08:24.143 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:24.139+0000 7f3c5ad9d640 1 -- 172.21.15.104:0/819244207 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3c5c02c840 con 0x7f3c5c0afce0 2024-03-28T14:08:24.144 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:24.143+0000 7f3c5ad9d640 1 --2- 172.21.15.104:0/819244207 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3c5c0afce0 0x7f3c5c14b780 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f3c5c0b3cb0 tx=0x7f3c5401c890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=454e485a97a88759 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:08:24.145 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:24.143+0000 7f3c5959a640 1 -- 172.21.15.104:0/819244207 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c540a5640 con 0x7f3c5c0afce0 2024-03-28T14:08:24.145 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:24.143+0000 7f3c5959a640 1 -- 172.21.15.104:0/819244207 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3c540a57e0 con 0x7f3c5c0afce0 2024-03-28T14:08:24.145 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:24.143+0000 7f3c5959a640 1 -- 172.21.15.104:0/819244207 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c540a5bb0 con 0x7f3c5c0afce0 2024-03-28T14:08:24.146 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:24.143+0000 7f3c608e3640 1 -- 172.21.15.104:0/819244207 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3c5c0afce0 msgr2=0x7f3c5c14b780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:24.146 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:24.143+0000 7f3c608e3640 1 --2- 172.21.15.104:0/819244207 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3c5c0afce0 0x7f3c5c14b780 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f3c5c0b3cb0 tx=0x7f3c5401c890 comp rx=0 tx=0).stop 2024-03-28T14:08:24.146 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:24.143+0000 7f3c608e3640 1 -- 172.21.15.104:0/819244207 shutdown_connections 2024-03-28T14:08:24.147 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:24.143+0000 7f3c608e3640 1 --2- 172.21.15.104:0/819244207 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3c5c0afce0 0x7f3c5c14b780 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:24.147 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:24.143+0000 7f3c608e3640 1 --2- 172.21.15.104:0/819244207 >> v2:172.21.15.171:3300/0 conn(0x7f3c5c02c460 0x7f3c5c06d020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:24.147 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:24.143+0000 7f3c608e3640 1 -- 172.21.15.104:0/819244207 >> 172.21.15.104:0/819244207 conn(0x7f3c5c0b95f0 msgr2=0x7f3c5c0b99d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T14:08:24.147 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:24.143+0000 7f3c608e3640 1 -- 172.21.15.104:0/819244207 shutdown_connections 2024-03-28T14:08:24.147 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:24.143+0000 7f3c608e3640 1 -- 172.21.15.104:0/819244207 wait complete. 2024-03-28T14:08:24.148 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:24.147+0000 7f3c608e3640 1 Processor -- start 2024-03-28T14:08:24.148 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:24.147+0000 7f3c608e3640 1 -- start start 2024-03-28T14:08:24.152 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:24.147+0000 7f3c608e3640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3c5c02c460 0x7f3c5c1e4380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:24.152 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:24.147+0000 7f3c608e3640 1 --2- >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f3c5c06e400 0x7f3c5c1e0290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:24.152 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:24.147+0000 7f3c608e3640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f3c5c0afce0 0x7f3c5c140a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:24.152 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:24.147+0000 7f3c608e3640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f3c5c0aeb30 con 0x7f3c5c02c460 2024-03-28T14:08:24.152 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:24.147+0000 7f3c608e3640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f3c5c0ae9b0 con 0x7f3c5c0afce0 2024-03-28T14:08:24.152 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:24.147+0000 7f3c608e3640 1 -- --> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] -- mon_getmap magic: 0 -- 0x7f3c5c14bcc0 con 0x7f3c5c06e400 2024-03-28T14:08:24.153 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:24.147+0000 7f3c5a59c640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3c5c02c460 0x7f3c5c1e4380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:24.153 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:24.147+0000 7f3c5a59c640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3c5c02c460 0x7f3c5c1e4380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:42644/0 (socket says 172.21.15.104:42644) 2024-03-28T14:08:24.153 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:24.147+0000 7f3c5a59c640 1 -- 172.21.15.104:0/351715545 learned_addr learned my addr 172.21.15.104:0/351715545 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-03-28T14:08:24.153 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:24.147+0000 7f3c5ad9d640 1 --2- 172.21.15.104:0/351715545 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f3c5c0afce0 0x7f3c5c140a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:24.153 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:24.147+0000 7f3c5a59c640 1 -- 172.21.15.104:0/351715545 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f3c5c06e400 msgr2=0x7f3c5c1e0290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:24.153 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:24.147+0000 7f3c5a59c640 1 --2- 172.21.15.104:0/351715545 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f3c5c06e400 0x7f3c5c1e0290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:24.153 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:24.147+0000 7f3c5a59c640 1 -- 172.21.15.104:0/351715545 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f3c5c0afce0 msgr2=0x7f3c5c140a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:24.154 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:24.147+0000 7f3c5a59c640 1 --2- 172.21.15.104:0/351715545 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f3c5c0afce0 0x7f3c5c140a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:24.154 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:24.147+0000 7f3c5a59c640 1 -- 172.21.15.104:0/351715545 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3c5c1e38a0 con 0x7f3c5c02c460 2024-03-28T14:08:24.154 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:24.147+0000 7f3c5a59c640 1 --2- 172.21.15.104:0/351715545 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3c5c02c460 0x7f3c5c1e4380 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f3c4c00c7f0 tx=0x7f3c4c00ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:08:24.154 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:24.147+0000 7f3c52ffd640 1 -- 172.21.15.104:0/351715545 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c4c015070 con 0x7f3c5c02c460 2024-03-28T14:08:24.154 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:24.147+0000 7f3c52ffd640 1 -- 172.21.15.104:0/351715545 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3c4c004240 con 0x7f3c5c02c460 2024-03-28T14:08:24.154 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:24.147+0000 7f3c608e3640 1 -- 172.21.15.104:0/351715545 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3c5c1e3b80 con 0x7f3c5c02c460 2024-03-28T14:08:24.155 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:24.147+0000 7f3c52ffd640 1 -- 172.21.15.104:0/351715545 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c4c003620 con 0x7f3c5c02c460 2024-03-28T14:08:24.155 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:24.147+0000 7f3c608e3640 1 -- 172.21.15.104:0/351715545 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3c5c144eb0 con 0x7f3c5c02c460 2024-03-28T14:08:24.155 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:24.147+0000 7f3c608e3640 1 -- 172.21.15.104:0/351715545 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3c5c141060 con 0x7f3c5c02c460 2024-03-28T14:08:24.155 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:24.147+0000 7f3c52ffd640 1 -- 172.21.15.104:0/351715545 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) ==== 87803+0+0 (secure 0 0 0) 0x7f3c4c0039d0 con 0x7f3c5c02c460 2024-03-28T14:08:24.155 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:24.151+0000 7f3c52ffd640 1 --2- 172.21.15.104:0/351715545 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f3c3c067fc0 0x7f3c3c06a480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:24.155 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:24.151+0000 7f3c52ffd640 1 -- 172.21.15.104:0/351715545 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f3c4c086610 con 0x7f3c5c02c460 2024-03-28T14:08:24.156 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:24.151+0000 7f3c59d9b640 1 --2- 172.21.15.104:0/351715545 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f3c3c067fc0 0x7f3c3c06a480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:24.157 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:24.151+0000 7f3c59d9b640 1 --2- 172.21.15.104:0/351715545 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f3c3c067fc0 0x7f3c3c06a480 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f3c5c1425d0 tx=0x7f3c440079c0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:08:24.158 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:24.155+0000 7f3c52ffd640 1 -- 172.21.15.104:0/351715545 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179117 (secure 0 0 0) 0x7f3c5c141060 con 0x7f3c5c02c460 2024-03-28T14:08:24.304 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:24.303+0000 7f3c608e3640 1 -- 172.21.15.104:0/351715545 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f3c5c1e4080 con 0x7f3c5c02c460 2024-03-28T14:08:24.304 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:24.303+0000 7f3c52ffd640 1 -- 172.21.15.104:0/351715545 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v21) ==== 104+0+11 (secure 0 0 0) 0x7f3c5c1e4080 con 0x7f3c5c02c460 2024-03-28T14:08:24.305 INFO:teuthology.orchestra.run.smithi104.stdout:pg_num: 32 2024-03-28T14:08:24.306 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:24.303+0000 7f3c608e3640 1 -- 172.21.15.104:0/351715545 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f3c3c067fc0 msgr2=0x7f3c3c06a480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:24.307 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:24.303+0000 7f3c608e3640 1 --2- 172.21.15.104:0/351715545 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f3c3c067fc0 0x7f3c3c06a480 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f3c5c1425d0 tx=0x7f3c440079c0 comp rx=0 tx=0).stop 2024-03-28T14:08:24.307 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:24.303+0000 7f3c608e3640 1 -- 172.21.15.104:0/351715545 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3c5c02c460 msgr2=0x7f3c5c1e4380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:24.307 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:24.303+0000 7f3c608e3640 1 --2- 172.21.15.104:0/351715545 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3c5c02c460 0x7f3c5c1e4380 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f3c4c00c7f0 tx=0x7f3c4c00ccb0 comp rx=0 tx=0).stop 2024-03-28T14:08:24.307 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:24.303+0000 7f3c608e3640 1 -- 172.21.15.104:0/351715545 shutdown_connections 2024-03-28T14:08:24.307 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:24.303+0000 7f3c608e3640 1 --2- 172.21.15.104:0/351715545 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f3c5c0afce0 0x7f3c5c140a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:24.307 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:24.303+0000 7f3c608e3640 1 --2- 172.21.15.104:0/351715545 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f3c5c06e400 0x7f3c5c1e0290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:24.307 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:24.303+0000 7f3c608e3640 1 --2- 172.21.15.104:0/351715545 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f3c3c067fc0 0x7f3c3c06a480 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:24.307 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:24.303+0000 7f3c608e3640 1 --2- 172.21.15.104:0/351715545 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3c5c02c460 0x7f3c5c1e4380 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:24.307 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:24.303+0000 7f3c608e3640 1 -- 172.21.15.104:0/351715545 >> 172.21.15.104:0/351715545 conn(0x7f3c5c0b95f0 msgr2=0x7f3c5c09dca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T14:08:24.307 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:24.303+0000 7f3c608e3640 1 -- 172.21.15.104:0/351715545 shutdown_connections 2024-03-28T14:08:24.307 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:24.303+0000 7f3c608e3640 1 -- 172.21.15.104:0/351715545 wait complete. 2024-03-28T14:08:24.325 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-28T14:08:24.451 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:24.447+0000 7f1724603640 1 Processor -- start 2024-03-28T14:08:24.452 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:24.447+0000 7f1724603640 1 -- start start 2024-03-28T14:08:24.452 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:24.451+0000 7f1724603640 1 --2- >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f171c14afd0 0x7f171c0b3ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:24.452 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:24.451+0000 7f1724603640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f171c02c840 con 0x7f171c02c460 2024-03-28T14:08:24.452 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:24.451+0000 7f1724603640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f171c0b4850 con 0x7f171c148110 2024-03-28T14:08:24.452 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:24.451+0000 7f1724603640 1 -- --> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] -- mon_getmap magic: 0 -- 0x7f171c0b4d70 con 0x7f171c14afd0 2024-03-28T14:08:24.452 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:24.451+0000 7f172239f640 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f171c02c460 0x7f171c0b5570 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:46202/0 (socket says 172.21.15.104:46202) 2024-03-28T14:08:24.452 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:24.451+0000 7f172239f640 1 -- 172.21.15.104:0/3040660550 learned_addr learned my addr 172.21.15.104:0/3040660550 (peer_addr_for_me v1:172.21.15.104:0/0) 2024-03-28T14:08:24.454 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:24.451+0000 7f1721b9e640 1 --2- 172.21.15.104:0/3040660550 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f171c14afd0 0x7f171c0b3ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:24.455 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:24.451+0000 7f172139d640 1 -- 172.21.15.104:0/3040660550 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2501311919 0 0) 0x7f171c02c840 con 0x7f171c02c460 2024-03-28T14:08:24.455 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:24.451+0000 7f172139d640 1 -- 172.21.15.104:0/3040660550 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1708003530 con 0x7f171c02c460 2024-03-28T14:08:24.455 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:24.451+0000 7f172139d640 1 -- 172.21.15.104:0/3040660550 <== mon.0 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2415630262 0 0) 0x7f1708003530 con 0x7f171c02c460 2024-03-28T14:08:24.455 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:24.451+0000 7f172139d640 1 -- 172.21.15.104:0/3040660550 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f171c14afd0 msgr2=0x7f171c0b3ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T14:08:24.455 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:24.451+0000 7f172139d640 1 --2- 172.21.15.104:0/3040660550 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f171c14afd0 0x7f171c0b3ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:24.455 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:24.451+0000 7f172139d640 1 -- 172.21.15.104:0/3040660550 >> v1:172.21.15.171:6789/0 conn(0x7f171c148110 legacy=0x7f171c1485a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:24.455 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:24.451+0000 7f172139d640 1 -- 172.21.15.104:0/3040660550 --> v1:172.21.15.104:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f171c0b5050 con 0x7f171c02c460 2024-03-28T14:08:24.455 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:24.451+0000 7f172139d640 1 -- 172.21.15.104:0/3040660550 <== mon.0 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3745580849 0 0) 0x7f1718001680 con 0x7f171c02c460 2024-03-28T14:08:24.455 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:24.451+0000 7f172139d640 1 -- 172.21.15.104:0/3040660550 >> v1:172.21.15.104:6789/0 conn(0x7f171c02c460 legacy=0x7f171c0b5570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:24.455 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:24.451+0000 7f172139d640 1 --2- 172.21.15.104:0/3040660550 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f17080039f0 0x7f1708003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:24.455 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:24.451+0000 7f172139d640 1 --2- 172.21.15.104:0/3040660550 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f171c14afd0 0x7f1708004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:24.455 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:24.451+0000 7f172139d640 1 --2- 172.21.15.104:0/3040660550 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1708004d50 0x7f17080051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:24.456 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:24.451+0000 7f172139d640 1 -- 172.21.15.104:0/3040660550 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f171c0b4d70 con 0x7f1708004d50 2024-03-28T14:08:24.456 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:24.451+0000 7f172139d640 1 -- 172.21.15.104:0/3040660550 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f170800bb00 con 0x7f17080039f0 2024-03-28T14:08:24.456 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:24.451+0000 7f172139d640 1 -- 172.21.15.104:0/3040660550 --> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] -- mon_getmap magic: 0 -- 0x7f170800bd00 con 0x7f171c14afd0 2024-03-28T14:08:24.456 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:24.451+0000 7f1722ba0640 1 --2- 172.21.15.104:0/3040660550 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1708004d50 0x7f17080051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:24.456 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:24.451+0000 7f172239f640 1 --2- 172.21.15.104:0/3040660550 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f17080039f0 0x7f1708003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:24.456 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:24.451+0000 7f1721b9e640 1 --2- 172.21.15.104:0/3040660550 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f171c14afd0 0x7f1708004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:24.456 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:24.451+0000 7f1722ba0640 1 -- 172.21.15.104:0/3040660550 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f171c14afd0 msgr2=0x7f1708004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T14:08:24.456 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:24.451+0000 7f1722ba0640 1 --2- 172.21.15.104:0/3040660550 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f171c14afd0 0x7f1708004810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:24.456 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:24.451+0000 7f1722ba0640 1 -- 172.21.15.104:0/3040660550 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f17080039f0 msgr2=0x7f1708003e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T14:08:24.456 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:24.451+0000 7f1722ba0640 1 --2- 172.21.15.104:0/3040660550 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f17080039f0 0x7f1708003e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:24.456 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:24.451+0000 7f1722ba0640 1 -- 172.21.15.104:0/3040660550 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1708003530 con 0x7f1708004d50 2024-03-28T14:08:24.457 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:24.451+0000 7f1722ba0640 1 --2- 172.21.15.104:0/3040660550 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1708004d50 0x7f17080051d0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f1714001e70 tx=0x7f171401c780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6b11cda88f646247 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:08:24.458 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:24.455+0000 7f172139d640 1 -- 172.21.15.104:0/3040660550 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1714001a90 con 0x7f1708004d50 2024-03-28T14:08:24.458 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:24.455+0000 7f172139d640 1 -- 172.21.15.104:0/3040660550 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1714003b80 con 0x7f1708004d50 2024-03-28T14:08:24.458 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:24.455+0000 7f172139d640 1 -- 172.21.15.104:0/3040660550 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1714005080 con 0x7f1708004d50 2024-03-28T14:08:24.458 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:24.455+0000 7f1724603640 1 -- 172.21.15.104:0/3040660550 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1708004d50 msgr2=0x7f17080051d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:24.458 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:24.455+0000 7f1724603640 1 --2- 172.21.15.104:0/3040660550 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1708004d50 0x7f17080051d0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f1714001e70 tx=0x7f171401c780 comp rx=0 tx=0).stop 2024-03-28T14:08:24.458 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:24.455+0000 7f1722ba0640 1 -- 172.21.15.104:0/3040660550 reap_dead start 2024-03-28T14:08:24.458 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:24.455+0000 7f1724603640 1 -- 172.21.15.104:0/3040660550 shutdown_connections 2024-03-28T14:08:24.458 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:24.455+0000 7f1724603640 1 -- 172.21.15.104:0/3040660550 >> 172.21.15.104:0/3040660550 conn(0x7f171c0b95f0 msgr2=0x7f171c0b99d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T14:08:24.458 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:24.455+0000 7f1724603640 1 -- 172.21.15.104:0/3040660550 shutdown_connections 2024-03-28T14:08:24.458 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:24.455+0000 7f1724603640 1 -- 172.21.15.104:0/3040660550 wait complete. 2024-03-28T14:08:24.458 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:24.455+0000 7f1724603640 1 Processor -- start 2024-03-28T14:08:24.458 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:24.455+0000 7f1724603640 1 -- start start 2024-03-28T14:08:24.461 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:24.455+0000 7f1724603640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f171c0b3140 0x7f171c1dda80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:24.461 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:24.455+0000 7f1724603640 1 --2- >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f171c1e0480 0x7f171c1e0900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:24.462 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:24.455+0000 7f1724603640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f171c1e0e40 0x7f171c0afac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:24.462 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:24.455+0000 7f1724603640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f171c0b4170 con 0x7f171c0b3140 2024-03-28T14:08:24.462 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:24.455+0000 7f1724603640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f171c0b3ff0 con 0x7f171c1e0e40 2024-03-28T14:08:24.462 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:24.455+0000 7f1724603640 1 -- --> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] -- mon_getmap magic: 0 -- 0x7f171c0b42f0 con 0x7f171c1e0480 2024-03-28T14:08:24.462 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:24.455+0000 7f172239f640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f171c0b3140 0x7f171c1dda80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:24.462 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:24.455+0000 7f172239f640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f171c0b3140 0x7f171c1dda80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:42658/0 (socket says 172.21.15.104:42658) 2024-03-28T14:08:24.462 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:24.455+0000 7f172239f640 1 -- 172.21.15.104:0/647812010 learned_addr learned my addr 172.21.15.104:0/647812010 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-03-28T14:08:24.462 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:24.455+0000 7f1721b9e640 1 --2- 172.21.15.104:0/647812010 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f171c1e0480 0x7f171c1e0900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:24.462 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:24.455+0000 7f1722ba0640 1 --2- 172.21.15.104:0/647812010 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f171c1e0e40 0x7f171c0afac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:24.462 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:24.455+0000 7f172239f640 1 -- 172.21.15.104:0/647812010 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f171c1e0480 msgr2=0x7f171c1e0900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:24.462 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:24.455+0000 7f172239f640 1 --2- 172.21.15.104:0/647812010 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f171c1e0480 0x7f171c1e0900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:24.462 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:24.455+0000 7f172239f640 1 -- 172.21.15.104:0/647812010 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f171c1e0e40 msgr2=0x7f171c0afac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:24.462 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:24.455+0000 7f172239f640 1 --2- 172.21.15.104:0/647812010 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f171c1e0e40 0x7f171c0afac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:24.463 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:24.455+0000 7f172239f640 1 -- 172.21.15.104:0/647812010 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f171c1e12c0 con 0x7f171c0b3140 2024-03-28T14:08:24.463 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:24.455+0000 7f1721b9e640 1 --2- 172.21.15.104:0/647812010 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f171c1e0480 0x7f171c1e0900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T14:08:24.463 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:24.455+0000 7f1722ba0640 1 --2- 172.21.15.104:0/647812010 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f171c1e0e40 0x7f171c0afac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T14:08:24.463 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:24.455+0000 7f172239f640 1 --2- 172.21.15.104:0/647812010 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f171c0b3140 0x7f171c1dda80 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f1718012490 tx=0x7f1718012950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:08:24.463 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:24.455+0000 7f1712ffd640 1 -- 172.21.15.104:0/647812010 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f171801c070 con 0x7f171c0b3140 2024-03-28T14:08:24.463 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:24.455+0000 7f1712ffd640 1 -- 172.21.15.104:0/647812010 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1718014770 con 0x7f171c0b3140 2024-03-28T14:08:24.463 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:24.455+0000 7f1724603640 1 -- 172.21.15.104:0/647812010 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f171c0b3520 con 0x7f171c0b3140 2024-03-28T14:08:24.463 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:24.455+0000 7f1712ffd640 1 -- 172.21.15.104:0/647812010 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1718014b60 con 0x7f171c0b3140 2024-03-28T14:08:24.463 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:24.459+0000 7f1724603640 1 -- 172.21.15.104:0/647812010 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f171c1dcfa0 con 0x7f171c0b3140 2024-03-28T14:08:24.463 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:24.459+0000 7f1724603640 1 -- 172.21.15.104:0/647812010 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f171c0687c0 con 0x7f171c0b3140 2024-03-28T14:08:24.463 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:24.459+0000 7f1712ffd640 1 -- 172.21.15.104:0/647812010 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) ==== 87803+0+0 (secure 0 0 0) 0x7f171801f070 con 0x7f171c0b3140 2024-03-28T14:08:24.463 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:24.459+0000 7f1712ffd640 1 --2- 172.21.15.104:0/647812010 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f1700067ff0 0x7f170006a4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:24.464 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:24.459+0000 7f1712ffd640 1 -- 172.21.15.104:0/647812010 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f1718088de0 con 0x7f171c0b3140 2024-03-28T14:08:24.464 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:24.459+0000 7f1721b9e640 1 --2- 172.21.15.104:0/647812010 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f1700067ff0 0x7f170006a4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:24.464 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:24.459+0000 7f1721b9e640 1 --2- 172.21.15.104:0/647812010 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f1700067ff0 0x7f170006a4b0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f171c0b8e20 tx=0x7f170c008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:08:24.468 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:24.467+0000 7f1712ffd640 1 -- 172.21.15.104:0/647812010 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179117 (secure 0 0 0) 0x7f1718059620 con 0x7f171c0b3140 2024-03-28T14:08:24.627 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:24.623+0000 7f1724603640 1 -- 172.21.15.104:0/647812010 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f171c09cf40 con 0x7f171c0b3140 2024-03-28T14:08:24.628 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:24.627+0000 7f1712ffd640 1 -- 172.21.15.104:0/647812010 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v21) ==== 100+0+11 (secure 0 0 0) 0x7f1718058f20 con 0x7f171c0b3140 2024-03-28T14:08:24.628 INFO:teuthology.orchestra.run.smithi104.stdout:pg_num: 64 2024-03-28T14:08:24.630 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:24.627+0000 7f1724603640 1 -- 172.21.15.104:0/647812010 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f1700067ff0 msgr2=0x7f170006a4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:24.631 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:24.627+0000 7f1724603640 1 --2- 172.21.15.104:0/647812010 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f1700067ff0 0x7f170006a4b0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f171c0b8e20 tx=0x7f170c008040 comp rx=0 tx=0).stop 2024-03-28T14:08:24.631 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:24.627+0000 7f1724603640 1 -- 172.21.15.104:0/647812010 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f171c0b3140 msgr2=0x7f171c1dda80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:24.631 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:24.627+0000 7f1724603640 1 --2- 172.21.15.104:0/647812010 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f171c0b3140 0x7f171c1dda80 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f1718012490 tx=0x7f1718012950 comp rx=0 tx=0).stop 2024-03-28T14:08:24.631 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:24.627+0000 7f1724603640 1 -- 172.21.15.104:0/647812010 shutdown_connections 2024-03-28T14:08:24.631 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:24.627+0000 7f1724603640 1 --2- 172.21.15.104:0/647812010 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f171c1e0e40 0x7f171c0afac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:24.631 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:24.627+0000 7f1724603640 1 --2- 172.21.15.104:0/647812010 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f171c1e0480 0x7f171c1e0900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:24.631 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:24.627+0000 7f1724603640 1 --2- 172.21.15.104:0/647812010 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f1700067ff0 0x7f170006a4b0 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:24.631 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:24.627+0000 7f1724603640 1 --2- 172.21.15.104:0/647812010 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f171c0b3140 0x7f171c1dda80 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:24.632 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:24.627+0000 7f1724603640 1 -- 172.21.15.104:0/647812010 >> 172.21.15.104:0/647812010 conn(0x7f171c0b95f0 msgr2=0x7f171c09c600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T14:08:24.632 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:24.627+0000 7f1724603640 1 -- 172.21.15.104:0/647812010 shutdown_connections 2024-03-28T14:08:24.632 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:24.627+0000 7f1724603640 1 -- 172.21.15.104:0/647812010 wait complete. 2024-03-28T14:08:24.648 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-28T14:08:24.769 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:24.767+0000 7fd9cdb25640 1 Processor -- start 2024-03-28T14:08:24.770 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:24.767+0000 7fd9cdb25640 1 -- start start 2024-03-28T14:08:24.771 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:24.767+0000 7fd9cdb25640 1 --2- >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fd9c8072bf0 0x7fd9c80b1e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:24.771 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:24.767+0000 7fd9cdb25640 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fd9c8145ce0 0x7fd9c8146150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:24.771 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:24.767+0000 7fd9cdb25640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd9c8072020 con 0x7fd9c8071be0 2024-03-28T14:08:24.771 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:24.767+0000 7fd9cdb25640 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 -- 0x7fd9c8073250 con 0x7fd9c8145ce0 2024-03-28T14:08:24.771 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:24.767+0000 7fd9cdb25640 1 -- --> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] -- mon_getmap magic: 0 -- 0x7fd9c8072fd0 con 0x7fd9c8072bf0 2024-03-28T14:08:24.771 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:24.767+0000 7fd9beffd640 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7fd9c8071be0 0x7fd9c80b12f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:46208/0 (socket says 172.21.15.104:46208) 2024-03-28T14:08:24.771 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:24.767+0000 7fd9beffd640 1 -- 172.21.15.104:0/1120891974 learned_addr learned my addr 172.21.15.104:0/1120891974 (peer_addr_for_me v1:172.21.15.104:0/0) 2024-03-28T14:08:24.771 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:24.767+0000 7fd9c7fff640 1 --2- 172.21.15.104:0/1120891974 >> v2:172.21.15.104:3300/0 conn(0x7fd9c8145ce0 0x7fd9c8146150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:24.772 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:24.767+0000 7fd9c77fe640 1 --2- 172.21.15.104:0/1120891974 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fd9c8072bf0 0x7fd9c80b1e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:24.772 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:24.767+0000 7fd9c7fff640 1 -- 172.21.15.104:0/1120891974 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fd9c8072bf0 msgr2=0x7fd9c80b1e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T14:08:24.772 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:24.767+0000 7fd9c7fff640 1 --2- 172.21.15.104:0/1120891974 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fd9c8072bf0 0x7fd9c80b1e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:24.772 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:24.767+0000 7fd9c7fff640 1 -- 172.21.15.104:0/1120891974 >> v1:172.21.15.104:6789/0 conn(0x7fd9c8071be0 legacy=0x7fd9c80b12f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:24.772 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:24.767+0000 7fd9c6ffd640 1 -- 172.21.15.104:0/1120891974 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 845207433 0 0) 0x7fd9c8072020 con 0x7fd9c8071be0 2024-03-28T14:08:24.772 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:24.767+0000 7fd9c7fff640 1 -- 172.21.15.104:0/1120891974 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd9c80734a0 con 0x7fd9c8145ce0 2024-03-28T14:08:24.773 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:24.771+0000 7fd9c7fff640 1 --2- 172.21.15.104:0/1120891974 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd9c8145ce0 0x7fd9c8146150 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7fd9c0009d90 tx=0x7fd9c001c850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5cc4de618eb5d416 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:08:24.773 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:24.771+0000 7fd9c6ffd640 1 -- 172.21.15.104:0/1120891974 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9c00b2070 con 0x7fd9c8145ce0 2024-03-28T14:08:24.773 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:24.771+0000 7fd9c6ffd640 1 -- 172.21.15.104:0/1120891974 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd9c00a57c0 con 0x7fd9c8145ce0 2024-03-28T14:08:24.774 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:24.771+0000 7fd9c6ffd640 1 -- 172.21.15.104:0/1120891974 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9c00a5b90 con 0x7fd9c8145ce0 2024-03-28T14:08:24.774 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:24.771+0000 7fd9cdb25640 1 -- 172.21.15.104:0/1120891974 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd9c8145ce0 msgr2=0x7fd9c8146150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:24.775 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:24.771+0000 7fd9cdb25640 1 --2- 172.21.15.104:0/1120891974 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd9c8145ce0 0x7fd9c8146150 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7fd9c0009d90 tx=0x7fd9c001c850 comp rx=0 tx=0).stop 2024-03-28T14:08:24.775 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:24.771+0000 7fd9cdb25640 1 -- 172.21.15.104:0/1120891974 shutdown_connections 2024-03-28T14:08:24.775 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:24.771+0000 7fd9cdb25640 1 --2- 172.21.15.104:0/1120891974 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd9c8145ce0 0x7fd9c8146150 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:24.775 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:24.771+0000 7fd9cdb25640 1 --2- 172.21.15.104:0/1120891974 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fd9c8072bf0 0x7fd9c80b1e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:24.775 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:24.771+0000 7fd9cdb25640 1 -- 172.21.15.104:0/1120891974 >> 172.21.15.104:0/1120891974 conn(0x7fd9c80b7b10 msgr2=0x7fd9c80b7f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T14:08:24.775 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:24.771+0000 7fd9cdb25640 1 -- 172.21.15.104:0/1120891974 shutdown_connections 2024-03-28T14:08:24.775 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:24.771+0000 7fd9cdb25640 1 -- 172.21.15.104:0/1120891974 wait complete. 2024-03-28T14:08:24.778 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:24.771+0000 7fd9cdb25640 1 Processor -- start 2024-03-28T14:08:24.778 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:24.771+0000 7fd9cdb25640 1 -- start start 2024-03-28T14:08:24.778 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:24.775+0000 7fd9cdb25640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fd9c8071be0 0x7fd9c8148410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:24.778 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:24.775+0000 7fd9cdb25640 1 --2- >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fd9c8072bf0 0x7fd9c8149290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:24.778 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:24.775+0000 7fd9cdb25640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd9c8145ce0 0x7fd9c8146e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:24.778 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:24.775+0000 7fd9cdb25640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7fd9c8152830 con 0x7fd9c8145ce0 2024-03-28T14:08:24.779 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:24.775+0000 7fd9cdb25640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7fd9c81526b0 con 0x7fd9c8071be0 2024-03-28T14:08:24.779 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:24.775+0000 7fd9cdb25640 1 -- --> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] -- mon_getmap magic: 0 -- 0x7fd9c81529b0 con 0x7fd9c8072bf0 2024-03-28T14:08:24.779 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:24.775+0000 7fd9c7fff640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd9c8145ce0 0x7fd9c8146e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:24.779 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:24.775+0000 7fd9c7fff640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd9c8145ce0 0x7fd9c8146e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:42682/0 (socket says 172.21.15.104:42682) 2024-03-28T14:08:24.779 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:24.775+0000 7fd9c7fff640 1 -- 172.21.15.104:0/2967973152 learned_addr learned my addr 172.21.15.104:0/2967973152 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-03-28T14:08:24.779 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:24.775+0000 7fd9beffd640 1 --2- 172.21.15.104:0/2967973152 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fd9c8072bf0 0x7fd9c8149290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:24.779 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:24.775+0000 7fd9c7fff640 1 -- 172.21.15.104:0/2967973152 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fd9c8072bf0 msgr2=0x7fd9c8149290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:24.779 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:24.775+0000 7fd9c7fff640 1 --2- 172.21.15.104:0/2967973152 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fd9c8072bf0 0x7fd9c8149290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:24.779 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:24.775+0000 7fd9c77fe640 1 --2- 172.21.15.104:0/2967973152 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fd9c8071be0 0x7fd9c8148410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:24.779 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:24.775+0000 7fd9c7fff640 1 -- 172.21.15.104:0/2967973152 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fd9c8071be0 msgr2=0x7fd9c8148410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:24.779 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:24.775+0000 7fd9c7fff640 1 --2- 172.21.15.104:0/2967973152 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fd9c8071be0 0x7fd9c8148410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:24.779 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:24.775+0000 7fd9c7fff640 1 -- 172.21.15.104:0/2967973152 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd9c8148b20 con 0x7fd9c8145ce0 2024-03-28T14:08:24.779 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:24.775+0000 7fd9c7fff640 1 --2- 172.21.15.104:0/2967973152 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd9c8145ce0 0x7fd9c8146e00 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7fd9c00b0040 tx=0x7fd9c00195f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:08:24.779 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:24.775+0000 7fd9c4ff9640 1 -- 172.21.15.104:0/2967973152 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9c00bd030 con 0x7fd9c8145ce0 2024-03-28T14:08:24.779 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:24.775+0000 7fd9c4ff9640 1 -- 172.21.15.104:0/2967973152 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd9c00a55c0 con 0x7fd9c8145ce0 2024-03-28T14:08:24.780 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:24.775+0000 7fd9c4ff9640 1 -- 172.21.15.104:0/2967973152 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9c00b2040 con 0x7fd9c8145ce0 2024-03-28T14:08:24.780 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:24.775+0000 7fd9cdb25640 1 -- 172.21.15.104:0/2967973152 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd9c80a0440 con 0x7fd9c8145ce0 2024-03-28T14:08:24.780 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:24.775+0000 7fd9cdb25640 1 -- 172.21.15.104:0/2967973152 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd9c809f650 con 0x7fd9c8145ce0 2024-03-28T14:08:24.780 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:24.775+0000 7fd9cdb25640 1 -- 172.21.15.104:0/2967973152 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd9c80687c0 con 0x7fd9c8145ce0 2024-03-28T14:08:24.780 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:24.775+0000 7fd9c4ff9640 1 -- 172.21.15.104:0/2967973152 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) ==== 87803+0+0 (secure 0 0 0) 0x7fd9c00c0070 con 0x7fd9c8145ce0 2024-03-28T14:08:24.781 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:24.775+0000 7fd9c4ff9640 1 --2- 172.21.15.104:0/2967973152 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7fd9a8067fc0 0x7fd9a806a480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:24.781 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:24.775+0000 7fd9c4ff9640 1 -- 172.21.15.104:0/2967973152 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7fd9c0121280 con 0x7fd9c8145ce0 2024-03-28T14:08:24.781 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:24.775+0000 7fd9c77fe640 1 --2- 172.21.15.104:0/2967973152 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7fd9a8067fc0 0x7fd9a806a480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:24.781 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:24.779+0000 7fd9c77fe640 1 --2- 172.21.15.104:0/2967973152 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7fd9a8067fc0 0x7fd9a806a480 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fd9b8009790 tx=0x7fd9b8008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:08:24.783 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:24.779+0000 7fd9c4ff9640 1 -- 172.21.15.104:0/2967973152 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179117 (secure 0 0 0) 0x7fd9c0016560 con 0x7fd9c8145ce0 2024-03-28T14:08:24.956 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:24.955+0000 7fd9cdb25640 1 -- 172.21.15.104:0/2967973152 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7fd9c80a1150 con 0x7fd9c8145ce0 2024-03-28T14:08:24.957 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:24.955+0000 7fd9c4ff9640 1 -- 172.21.15.104:0/2967973152 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v21) ==== 80+0+0 (secure 0 0 0) 0x7fd9c00f1ac0 con 0x7fd9c8145ce0 2024-03-28T14:08:24.957 INFO:teuthology.orchestra.run.smithi104.stderr:listed 0 entries 2024-03-28T14:08:24.959 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:24.955+0000 7fd9cdb25640 1 -- 172.21.15.104:0/2967973152 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7fd9a8067fc0 msgr2=0x7fd9a806a480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:24.959 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:24.955+0000 7fd9cdb25640 1 --2- 172.21.15.104:0/2967973152 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7fd9a8067fc0 0x7fd9a806a480 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fd9b8009790 tx=0x7fd9b8008040 comp rx=0 tx=0).stop 2024-03-28T14:08:24.959 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:24.955+0000 7fd9cdb25640 1 -- 172.21.15.104:0/2967973152 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd9c8145ce0 msgr2=0x7fd9c8146e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:24.959 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:24.955+0000 7fd9cdb25640 1 --2- 172.21.15.104:0/2967973152 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd9c8145ce0 0x7fd9c8146e00 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7fd9c00b0040 tx=0x7fd9c00195f0 comp rx=0 tx=0).stop 2024-03-28T14:08:24.959 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:24.955+0000 7fd9cdb25640 1 -- 172.21.15.104:0/2967973152 shutdown_connections 2024-03-28T14:08:24.959 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:24.955+0000 7fd9cdb25640 1 --2- 172.21.15.104:0/2967973152 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7fd9a8067fc0 0x7fd9a806a480 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:24.959 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:24.955+0000 7fd9cdb25640 1 --2- 172.21.15.104:0/2967973152 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd9c8145ce0 0x7fd9c8146e00 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:24.959 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:24.955+0000 7fd9cdb25640 1 --2- 172.21.15.104:0/2967973152 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fd9c8072bf0 0x7fd9c8149290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:24.959 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:24.955+0000 7fd9cdb25640 1 --2- 172.21.15.104:0/2967973152 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fd9c8071be0 0x7fd9c8148410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:24.959 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:24.955+0000 7fd9cdb25640 1 -- 172.21.15.104:0/2967973152 >> 172.21.15.104:0/2967973152 conn(0x7fd9c80b7b10 msgr2=0x7fd9c80b6160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T14:08:24.960 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:24.955+0000 7fd9cdb25640 1 -- 172.21.15.104:0/2967973152 shutdown_connections 2024-03-28T14:08:24.960 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:24.955+0000 7fd9cdb25640 1 -- 172.21.15.104:0/2967973152 wait complete. 2024-03-28T14:08:24.975 DEBUG:teuthology.orchestra.run.smithi104:> set -ex 2024-03-28T14:08:24.975 DEBUG:teuthology.orchestra.run.smithi104:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-28T14:08:24.981 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-28T14:08:25.135 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:25.131+0000 7fb762c1d640 1 Processor -- start 2024-03-28T14:08:25.135 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:25.131+0000 7fb762c1d640 1 -- start start 2024-03-28T14:08:25.138 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:25.135+0000 7fb762c1d640 1 --2- >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fb75c06de20 0x7fb75c144730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:25.138 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:25.135+0000 7fb762c1d640 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fb75c0733f0 0x7fb75c0b02f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:25.138 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:25.135+0000 7fb762c1d640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb75c06e200 con 0x7fb75c0b6260 2024-03-28T14:08:25.138 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:25.135+0000 7fb762c1d640 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 -- 0x7fb75c0b0830 con 0x7fb75c0733f0 2024-03-28T14:08:25.138 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:25.135+0000 7fb762c1d640 1 -- --> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] -- mon_getmap magic: 0 -- 0x7fb75c0af820 con 0x7fb75c06de20 2024-03-28T14:08:25.138 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:25.135+0000 7fb75bfff640 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fb75c0733f0 0x7fb75c0b02f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:25.139 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:25.135+0000 7fb7611ba640 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7fb75c0b6260 0x7fb75c09c3c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:46214/0 (socket says 172.21.15.104:46214) 2024-03-28T14:08:25.139 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:25.135+0000 7fb7611ba640 1 -- 172.21.15.104:0/1510829611 learned_addr learned my addr 172.21.15.104:0/1510829611 (peer_addr_for_me v1:172.21.15.104:0/0) 2024-03-28T14:08:25.139 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:25.135+0000 7fb7609b9640 1 --2- 172.21.15.104:0/1510829611 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fb75c06de20 0x7fb75c144730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:25.139 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:25.135+0000 7fb75bfff640 1 -- 172.21.15.104:0/1510829611 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fb75c06de20 msgr2=0x7fb75c144730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T14:08:25.139 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:25.135+0000 7fb75bfff640 1 --2- 172.21.15.104:0/1510829611 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fb75c06de20 0x7fb75c144730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:25.139 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:25.135+0000 7fb75bfff640 1 -- 172.21.15.104:0/1510829611 >> v1:172.21.15.104:6789/0 conn(0x7fb75c0b6260 legacy=0x7fb75c09c3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:25.140 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:25.135+0000 7fb75b7fe640 1 -- 172.21.15.104:0/1510829611 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1509451938 0 0) 0x7fb75c06e200 con 0x7fb75c0b6260 2024-03-28T14:08:25.140 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:25.135+0000 7fb75bfff640 1 -- 172.21.15.104:0/1510829611 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb75c0af2e0 con 0x7fb75c0733f0 2024-03-28T14:08:25.144 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:25.135+0000 7fb75bfff640 1 --2- 172.21.15.104:0/1510829611 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb75c0733f0 0x7fb75c0b02f0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7fb750009d90 tx=0x7fb75001c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=46e4fe33ac4724b4 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:08:25.144 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:25.135+0000 7fb75b7fe640 1 -- 172.21.15.104:0/1510829611 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7500b2070 con 0x7fb75c0733f0 2024-03-28T14:08:25.144 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:25.135+0000 7fb75b7fe640 1 -- 172.21.15.104:0/1510829611 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb7500a57a0 con 0x7fb75c0733f0 2024-03-28T14:08:25.144 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:25.135+0000 7fb75b7fe640 1 -- 172.21.15.104:0/1510829611 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7500a5b70 con 0x7fb75c0733f0 2024-03-28T14:08:25.144 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:25.139+0000 7fb762c1d640 1 -- 172.21.15.104:0/1510829611 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb75c0733f0 msgr2=0x7fb75c0b02f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:25.144 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:25.139+0000 7fb762c1d640 1 --2- 172.21.15.104:0/1510829611 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb75c0733f0 0x7fb75c0b02f0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7fb750009d90 tx=0x7fb75001c8e0 comp rx=0 tx=0).stop 2024-03-28T14:08:25.144 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:25.139+0000 7fb762c1d640 1 -- 172.21.15.104:0/1510829611 shutdown_connections 2024-03-28T14:08:25.144 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:25.139+0000 7fb762c1d640 1 --2- 172.21.15.104:0/1510829611 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb75c0733f0 0x7fb75c0b02f0 secure :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7fb750009d90 tx=0x7fb75001c8e0 comp rx=0 tx=0).stop 2024-03-28T14:08:25.144 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:25.139+0000 7fb762c1d640 1 --2- 172.21.15.104:0/1510829611 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fb75c06de20 0x7fb75c144730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:25.144 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:25.139+0000 7fb762c1d640 1 -- 172.21.15.104:0/1510829611 >> 172.21.15.104:0/1510829611 conn(0x7fb75c0ba390 msgr2=0x7fb75c0ba7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T14:08:25.144 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:25.139+0000 7fb762c1d640 1 -- 172.21.15.104:0/1510829611 shutdown_connections 2024-03-28T14:08:25.145 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:25.139+0000 7fb762c1d640 1 -- 172.21.15.104:0/1510829611 wait complete. 2024-03-28T14:08:25.145 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:25.139+0000 7fb762c1d640 1 Processor -- start 2024-03-28T14:08:25.145 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:25.139+0000 7fb762c1d640 1 -- start start 2024-03-28T14:08:25.145 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:25.139+0000 7fb762c1d640 1 --2- >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fb75c06de20 0x7fb75c071e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:25.145 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:25.139+0000 7fb762c1d640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fb75c0b6260 0x7fb75c09d9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:25.145 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:25.139+0000 7fb762c1d640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb75c09f4f0 0x7fb75c09e570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:25.145 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:25.139+0000 7fb762c1d640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7fb75c06b820 con 0x7fb75c09f4f0 2024-03-28T14:08:25.145 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:25.139+0000 7fb762c1d640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7fb75c09cee0 con 0x7fb75c0b6260 2024-03-28T14:08:25.145 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:25.139+0000 7fb762c1d640 1 -- --> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] -- mon_getmap magic: 0 -- 0x7fb75c06b9a0 con 0x7fb75c06de20 2024-03-28T14:08:25.145 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:25.139+0000 7fb7611ba640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb75c09f4f0 0x7fb75c09e570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:25.145 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:25.139+0000 7fb7611ba640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb75c09f4f0 0x7fb75c09e570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:42694/0 (socket says 172.21.15.104:42694) 2024-03-28T14:08:25.145 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:25.139+0000 7fb7611ba640 1 -- 172.21.15.104:0/554194955 learned_addr learned my addr 172.21.15.104:0/554194955 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-03-28T14:08:25.145 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:25.139+0000 7fb75bfff640 1 --2- 172.21.15.104:0/554194955 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fb75c0b6260 0x7fb75c09d9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:25.145 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:25.139+0000 7fb7609b9640 1 --2- 172.21.15.104:0/554194955 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fb75c06de20 0x7fb75c071e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:25.145 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:25.139+0000 7fb7611ba640 1 -- 172.21.15.104:0/554194955 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fb75c06de20 msgr2=0x7fb75c071e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:25.146 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:25.139+0000 7fb7611ba640 1 --2- 172.21.15.104:0/554194955 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fb75c06de20 0x7fb75c071e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:25.146 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:25.139+0000 7fb7611ba640 1 -- 172.21.15.104:0/554194955 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fb75c0b6260 msgr2=0x7fb75c09d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:25.146 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:25.139+0000 7fb7611ba640 1 --2- 172.21.15.104:0/554194955 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fb75c0b6260 0x7fb75c09d9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:25.146 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:25.139+0000 7fb7611ba640 1 -- 172.21.15.104:0/554194955 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb75c09df20 con 0x7fb75c09f4f0 2024-03-28T14:08:25.146 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:25.139+0000 7fb75bfff640 1 --2- 172.21.15.104:0/554194955 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fb75c0b6260 0x7fb75c09d9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T14:08:25.146 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:25.139+0000 7fb7611ba640 1 --2- 172.21.15.104:0/554194955 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb75c09f4f0 0x7fb75c09e570 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7fb75400c410 tx=0x7fb75400c810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:08:25.146 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:25.139+0000 7fb7597fa640 1 -- 172.21.15.104:0/554194955 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7540074e0 con 0x7fb75c09f4f0 2024-03-28T14:08:25.146 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:25.139+0000 7fb7597fa640 1 -- 172.21.15.104:0/554194955 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb7540047f0 con 0x7fb75c09f4f0 2024-03-28T14:08:25.146 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:25.139+0000 7fb762c1d640 1 -- 172.21.15.104:0/554194955 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb75c09eab0 con 0x7fb75c09f4f0 2024-03-28T14:08:25.146 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:25.139+0000 7fb7597fa640 1 -- 172.21.15.104:0/554194955 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb754007680 con 0x7fb75c09f4f0 2024-03-28T14:08:25.146 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:25.139+0000 7fb762c1d640 1 -- 172.21.15.104:0/554194955 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb75c09f060 con 0x7fb75c09f4f0 2024-03-28T14:08:25.146 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:25.139+0000 7fb762c1d640 1 -- 172.21.15.104:0/554194955 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb75c09d0b0 con 0x7fb75c09f4f0 2024-03-28T14:08:25.146 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:25.143+0000 7fb7597fa640 1 -- 172.21.15.104:0/554194955 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) ==== 87803+0+0 (secure 0 0 0) 0x7fb754011070 con 0x7fb75c09f4f0 2024-03-28T14:08:25.146 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:25.143+0000 7fb7597fa640 1 --2- 172.21.15.104:0/554194955 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7fb748067fc0 0x7fb74806a480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:25.147 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:25.143+0000 7fb7597fa640 1 -- 172.21.15.104:0/554194955 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7fb754085fa0 con 0x7fb75c09f4f0 2024-03-28T14:08:25.147 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:25.143+0000 7fb7609b9640 1 --2- 172.21.15.104:0/554194955 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7fb748067fc0 0x7fb74806a480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:25.147 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:25.143+0000 7fb7609b9640 1 --2- 172.21.15.104:0/554194955 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7fb748067fc0 0x7fb74806a480 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fb75c0729b0 tx=0x7fb74c008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:08:25.149 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:25.147+0000 7fb7597fa640 1 -- 172.21.15.104:0/554194955 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179117 (secure 0 0 0) 0x7fb7540567e0 con 0x7fb75c09f4f0 2024-03-28T14:08:25.299 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:25.295+0000 7fb762c1d640 1 -- 172.21.15.104:0/554194955 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fb75c13ec80 con 0x7fb75c09f4f0 2024-03-28T14:08:25.300 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:25.299+0000 7fb7597fa640 1 -- 172.21.15.104:0/554194955 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) ==== 74+0+16629 (secure 0 0 0) 0x7fb75c13ec80 con 0x7fb75c09f4f0 2024-03-28T14:08:25.300 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-03-28T14:08:25.300 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":21,"fsid":"428a90f2-3351-435f-8fcf-f7e66c78a13e","created":"2024-03-28T14:07:34.862458+0000","modified":"2024-03-28T14:08:00.671587+0000","last_up_change":"2024-03-28T14:07:46.028122+0000","last_in_change":"2024-03-28T14:07:37.157486+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-28T14:07:47.622988+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-28T14:07:57.147450+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-28T14:07:57.990975+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"1cb70540-4e42-402f-a446-aaec4ca83007","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6800","nonce":3745690918},{"type":"v1","addr":"172.21.15.104:6801","nonce":3745690918}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":3745690918},{"type":"v1","addr":"172.21.15.104:6803","nonce":3745690918}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":3745690918},{"type":"v1","addr":"172.21.15.104:6807","nonce":3745690918}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":3745690918},{"type":"v1","addr":"172.21.15.104:6805","nonce":3745690918}]},"public_addr":"172.21.15.104:6801/3745690918","cluster_addr":"172.21.15.104:6803/3745690918","heartbeat_back_addr":"172.21.15.104:6807/3745690918","heartbeat_front_addr":"172.21.15.104:6805/3745690918","state":["exists","up"]},{"osd":1,"uuid":"3ab7afdd-b980-418c-81f2-004e16de3dca","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":19438674},{"type":"v1","addr":"172.21.15.104:6810","nonce":19438674}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":19438674},{"type":"v1","addr":"172.21.15.104:6814","nonce":19438674}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":19438674},{"type":"v1","addr":"172.21.15.104:6823","nonce":19438674}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":19438674},{"type":"v1","addr":"172.21.15.104:6819","nonce":19438674}]},"public_addr":"172.21.15.104:6810/19438674","cluster_addr":"172.21.15.104:6814/19438674","heartbeat_back_addr":"172.21.15.104:6823/19438674","heartbeat_front_addr":"172.21.15.104:6819/19438674","state":["exists","up"]},{"osd":2,"uuid":"3e2651c8-ecae-4a9e-a053-27240a79eac2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":2352012626},{"type":"v1","addr":"172.21.15.104:6825","nonce":2352012626}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6826","nonce":2352012626},{"type":"v1","addr":"172.21.15.104:6827","nonce":2352012626}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":2352012626},{"type":"v1","addr":"172.21.15.104:6831","nonce":2352012626}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6828","nonce":2352012626},{"type":"v1","addr":"172.21.15.104:6829","nonce":2352012626}]},"public_addr":"172.21.15.104:6825/2352012626","cluster_addr":"172.21.15.104:6827/2352012626","heartbeat_back_addr":"172.21.15.104:6831/2352012626","heartbeat_front_addr":"172.21.15.104:6829/2352012626","state":["exists","up"]},{"osd":3,"uuid":"925c7e0a-38ef-47cd-b288-8bec8456fb17","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6809","nonce":3568347381},{"type":"v1","addr":"172.21.15.104:6811","nonce":3568347381}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6813","nonce":3568347381},{"type":"v1","addr":"172.21.15.104:6815","nonce":3568347381}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":3568347381},{"type":"v1","addr":"172.21.15.104:6822","nonce":3568347381}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6817","nonce":3568347381},{"type":"v1","addr":"172.21.15.104:6818","nonce":3568347381}]},"public_addr":"172.21.15.104:6811/3568347381","cluster_addr":"172.21.15.104:6815/3568347381","heartbeat_back_addr":"172.21.15.104:6822/3568347381","heartbeat_front_addr":"172.21.15.104:6818/3568347381","state":["exists","up"]},{"osd":4,"uuid":"4a8fac86-e595-4a0e-96e3-57edb3cc3f57","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6800","nonce":2107001077},{"type":"v1","addr":"172.21.15.171:6801","nonce":2107001077}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6802","nonce":2107001077},{"type":"v1","addr":"172.21.15.171:6803","nonce":2107001077}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6806","nonce":2107001077},{"type":"v1","addr":"172.21.15.171:6807","nonce":2107001077}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6804","nonce":2107001077},{"type":"v1","addr":"172.21.15.171:6805","nonce":2107001077}]},"public_addr":"172.21.15.171:6801/2107001077","cluster_addr":"172.21.15.171:6803/2107001077","heartbeat_back_addr":"172.21.15.171:6807/2107001077","heartbeat_front_addr":"172.21.15.171:6805/2107001077","state":["exists","up"]},{"osd":5,"uuid":"e48d50b0-beae-40ae-b15a-9a84dbfe88ab","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6808","nonce":2934826925},{"type":"v1","addr":"172.21.15.171:6810","nonce":2934826925}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6812","nonce":2934826925},{"type":"v1","addr":"172.21.15.171:6814","nonce":2934826925}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6819","nonce":2934826925},{"type":"v1","addr":"172.21.15.171:6821","nonce":2934826925}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6816","nonce":2934826925},{"type":"v1","addr":"172.21.15.171:6817","nonce":2934826925}]},"public_addr":"172.21.15.171:6810/2934826925","cluster_addr":"172.21.15.171:6814/2934826925","heartbeat_back_addr":"172.21.15.171:6821/2934826925","heartbeat_front_addr":"172.21.15.171:6817/2934826925","state":["exists","up"]},{"osd":6,"uuid":"5e7c3292-9936-4f35-b267-aed9d4e49323","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6824","nonce":436742919},{"type":"v1","addr":"172.21.15.171:6825","nonce":436742919}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6826","nonce":436742919},{"type":"v1","addr":"172.21.15.171:6827","nonce":436742919}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6830","nonce":436742919},{"type":"v1","addr":"172.21.15.171:6831","nonce":436742919}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6828","nonce":436742919},{"type":"v1","addr":"172.21.15.171:6829","nonce":436742919}]},"public_addr":"172.21.15.171:6825/436742919","cluster_addr":"172.21.15.171:6827/436742919","heartbeat_back_addr":"172.21.15.171:6831/436742919","heartbeat_front_addr":"172.21.15.171:6829/436742919","state":["exists","up"]},{"osd":7,"uuid":"339b8414-d7f0-4fb0-a446-efaf635cb7ff","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6809","nonce":2055631175},{"type":"v1","addr":"172.21.15.171:6811","nonce":2055631175}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6813","nonce":2055631175},{"type":"v1","addr":"172.21.15.171:6815","nonce":2055631175}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6822","nonce":2055631175},{"type":"v1","addr":"172.21.15.171:6823","nonce":2055631175}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6818","nonce":2055631175},{"type":"v1","addr":"172.21.15.171:6820","nonce":2055631175}]},"public_addr":"172.21.15.171:6811/2055631175","cluster_addr":"172.21.15.171:6815/2055631175","heartbeat_back_addr":"172.21.15.171:6823/2055631175","heartbeat_front_addr":"172.21.15.171:6820/2055631175","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T14:07:42.668881+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T14:07:42.708397+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T14:07:42.634191+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T14:07:42.733662+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T14:07:42.651541+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T14:07:42.783181+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T14:07:42.766310+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T14:07:42.670797+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-28T14:08:25.302 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:25.299+0000 7fb762c1d640 1 -- 172.21.15.104:0/554194955 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7fb748067fc0 msgr2=0x7fb74806a480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:25.302 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:25.299+0000 7fb762c1d640 1 --2- 172.21.15.104:0/554194955 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7fb748067fc0 0x7fb74806a480 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fb75c0729b0 tx=0x7fb74c008040 comp rx=0 tx=0).stop 2024-03-28T14:08:25.303 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:25.299+0000 7fb762c1d640 1 -- 172.21.15.104:0/554194955 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb75c09f4f0 msgr2=0x7fb75c09e570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:25.303 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:25.299+0000 7fb762c1d640 1 --2- 172.21.15.104:0/554194955 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb75c09f4f0 0x7fb75c09e570 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7fb75400c410 tx=0x7fb75400c810 comp rx=0 tx=0).stop 2024-03-28T14:08:25.303 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:25.299+0000 7fb762c1d640 1 -- 172.21.15.104:0/554194955 shutdown_connections 2024-03-28T14:08:25.303 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:25.299+0000 7fb762c1d640 1 --2- 172.21.15.104:0/554194955 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7fb748067fc0 0x7fb74806a480 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:25.303 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:25.299+0000 7fb762c1d640 1 --2- 172.21.15.104:0/554194955 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb75c09f4f0 0x7fb75c09e570 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:25.303 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:25.299+0000 7fb762c1d640 1 --2- 172.21.15.104:0/554194955 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fb75c0b6260 0x7fb75c09d9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:25.303 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:25.299+0000 7fb762c1d640 1 --2- 172.21.15.104:0/554194955 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fb75c06de20 0x7fb75c071e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:25.303 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:25.299+0000 7fb762c1d640 1 -- 172.21.15.104:0/554194955 >> 172.21.15.104:0/554194955 conn(0x7fb75c0ba390 msgr2=0x7fb75c0b4620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T14:08:25.303 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:25.299+0000 7fb762c1d640 1 -- 172.21.15.104:0/554194955 shutdown_connections 2024-03-28T14:08:25.303 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:25.299+0000 7fb762c1d640 1 -- 172.21.15.104:0/554194955 wait complete. 2024-03-28T14:08:25.317 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-28T14:07:47.622988+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-28T14:07:57.147450+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-28T14:07:57.990975+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-28T14:08:25.317 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-28T14:08:25.432 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:25.431+0000 7fb25373a640 1 Processor -- start 2024-03-28T14:08:25.432 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:25.431+0000 7fb25373a640 1 -- start start 2024-03-28T14:08:25.433 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:25.431+0000 7fb25373a640 1 --2- >> v2:172.21.15.171:3300/0 conn(0x7fb24c072bf0 0x7fb24c0b1e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:25.433 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:25.431+0000 7fb25373a640 1 --2- >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fb24c071be0 0x7fb24c0b12f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:25.433 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:25.431+0000 7fb25373a640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb24c072020 con 0x7fb24c0b1830 2024-03-28T14:08:25.433 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:25.431+0000 7fb25373a640 1 -- --> v2:172.21.15.171:3300/0 -- mon_getmap magic: 0 -- 0x7fb24c073250 con 0x7fb24c072bf0 2024-03-28T14:08:25.433 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:25.431+0000 7fb25373a640 1 -- --> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] -- mon_getmap magic: 0 -- 0x7fb24c072fd0 con 0x7fb24c071be0 2024-03-28T14:08:25.434 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:25.431+0000 7fb250cd5640 1 --2- >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fb24c071be0 0x7fb24c0b12f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:25.434 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:25.431+0000 7fb250cd5640 1 --2- >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fb24c071be0 0x7fb24c0b12f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.171:3301/0 says I am v2:172.21.15.104:39334/0 (socket says 172.21.15.104:39334) 2024-03-28T14:08:25.434 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:25.431+0000 7fb250cd5640 1 -- 172.21.15.104:0/147557128 learned_addr learned my addr 172.21.15.104:0/147557128 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-03-28T14:08:25.435 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:25.431+0000 7fb2514d6640 1 --2- >> v2:172.21.15.171:3300/0 conn(0x7fb24c072bf0 0x7fb24c0b1e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:25.435 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:25.431+0000 7fb250cd5640 1 -- 172.21.15.104:0/147557128 >> v2:172.21.15.171:3300/0 conn(0x7fb24c072bf0 msgr2=0x7fb24c0b1e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T14:08:25.436 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:25.431+0000 7fb250cd5640 1 --2- 172.21.15.104:0/147557128 >> v2:172.21.15.171:3300/0 conn(0x7fb24c072bf0 0x7fb24c0b1e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:25.436 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:25.431+0000 7fb250cd5640 1 -- 172.21.15.104:0/147557128 >> v1:172.21.15.171:6789/0 conn(0x7fb24c0b1830 legacy=0x7fb24c145ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:25.436 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:25.431+0000 7fb250cd5640 1 -- 172.21.15.104:0/147557128 --> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb24c0734a0 con 0x7fb24c071be0 2024-03-28T14:08:25.436 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:25.435+0000 7fb250cd5640 1 --2- 172.21.15.104:0/147557128 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fb24c071be0 0x7fb24c0b12f0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fb23c009870 tx=0x7fb23c01c440 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9a7b34a4b39010b3 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:08:25.437 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:25.435+0000 7fb243fff640 1 -- 172.21.15.104:0/147557128 <== mon.2 v2:172.21.15.171:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb23c0b0070 con 0x7fb24c071be0 2024-03-28T14:08:25.437 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:25.435+0000 7fb243fff640 1 -- 172.21.15.104:0/147557128 <== mon.2 v2:172.21.15.171:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb23c0ab600 con 0x7fb24c071be0 2024-03-28T14:08:25.437 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:25.435+0000 7fb243fff640 1 -- 172.21.15.104:0/147557128 <== mon.2 v2:172.21.15.171:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb23c0ab9d0 con 0x7fb24c071be0 2024-03-28T14:08:25.437 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:25.435+0000 7fb25373a640 1 -- 172.21.15.104:0/147557128 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fb24c071be0 msgr2=0x7fb24c0b12f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:25.437 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:25.435+0000 7fb25373a640 1 --2- 172.21.15.104:0/147557128 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fb24c071be0 0x7fb24c0b12f0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fb23c009870 tx=0x7fb23c01c440 comp rx=0 tx=0).stop 2024-03-28T14:08:25.438 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:25.435+0000 7fb25373a640 1 -- 172.21.15.104:0/147557128 shutdown_connections 2024-03-28T14:08:25.438 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:25.435+0000 7fb25373a640 1 --2- 172.21.15.104:0/147557128 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fb24c071be0 0x7fb24c0b12f0 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:25.438 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:25.435+0000 7fb25373a640 1 --2- 172.21.15.104:0/147557128 >> v2:172.21.15.171:3300/0 conn(0x7fb24c072bf0 0x7fb24c0b1e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:25.438 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:25.435+0000 7fb25373a640 1 -- 172.21.15.104:0/147557128 >> 172.21.15.104:0/147557128 conn(0x7fb24c0b7b10 msgr2=0x7fb24c0b7f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T14:08:25.438 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:25.435+0000 7fb25373a640 1 -- 172.21.15.104:0/147557128 shutdown_connections 2024-03-28T14:08:25.438 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:25.435+0000 7fb25373a640 1 -- 172.21.15.104:0/147557128 wait complete. 2024-03-28T14:08:25.438 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:25.435+0000 7fb25373a640 1 Processor -- start 2024-03-28T14:08:25.438 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:25.435+0000 7fb25373a640 1 -- start start 2024-03-28T14:08:25.440 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:25.435+0000 7fb25373a640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb24c071be0 0x7fb24c13e630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:25.440 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:25.435+0000 7fb25373a640 1 --2- >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fb24c072bf0 0x7fb24c1e3a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:25.441 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:25.435+0000 7fb25373a640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fb24c0b1830 0x7fb24c1350f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:25.441 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:25.435+0000 7fb25373a640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7fb24c152830 con 0x7fb24c071be0 2024-03-28T14:08:25.441 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:25.435+0000 7fb25373a640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7fb24c1526b0 con 0x7fb24c0b1830 2024-03-28T14:08:25.441 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:25.435+0000 7fb25373a640 1 -- --> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] -- mon_getmap magic: 0 -- 0x7fb24c1529b0 con 0x7fb24c072bf0 2024-03-28T14:08:25.441 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:25.439+0000 7fb2514d6640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb24c071be0 0x7fb24c13e630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:25.441 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:25.439+0000 7fb2514d6640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb24c071be0 0x7fb24c13e630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:42702/0 (socket says 172.21.15.104:42702) 2024-03-28T14:08:25.441 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:25.439+0000 7fb2514d6640 1 -- 172.21.15.104:0/1713363773 learned_addr learned my addr 172.21.15.104:0/1713363773 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-03-28T14:08:25.441 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:25.439+0000 7fb250cd5640 1 --2- 172.21.15.104:0/1713363773 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fb24c072bf0 0x7fb24c1e3a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:25.441 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:25.439+0000 7fb251cd7640 1 --2- 172.21.15.104:0/1713363773 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fb24c0b1830 0x7fb24c1350f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:25.441 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:25.439+0000 7fb2514d6640 1 -- 172.21.15.104:0/1713363773 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fb24c072bf0 msgr2=0x7fb24c1e3a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:25.441 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:25.439+0000 7fb2514d6640 1 --2- 172.21.15.104:0/1713363773 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fb24c072bf0 0x7fb24c1e3a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:25.441 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:25.439+0000 7fb2514d6640 1 -- 172.21.15.104:0/1713363773 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fb24c0b1830 msgr2=0x7fb24c1350f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:25.441 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:25.439+0000 7fb2514d6640 1 --2- 172.21.15.104:0/1713363773 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fb24c0b1830 0x7fb24c1350f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:25.442 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:25.439+0000 7fb2514d6640 1 -- 172.21.15.104:0/1713363773 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb24c0a0d20 con 0x7fb24c071be0 2024-03-28T14:08:25.442 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:25.439+0000 7fb251cd7640 1 --2- 172.21.15.104:0/1713363773 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fb24c0b1830 0x7fb24c1350f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T14:08:25.442 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:25.439+0000 7fb250cd5640 1 --2- 172.21.15.104:0/1713363773 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fb24c072bf0 0x7fb24c1e3a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T14:08:25.442 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:25.439+0000 7fb2514d6640 1 --2- 172.21.15.104:0/1713363773 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb24c071be0 0x7fb24c13e630 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7fb24800e9b0 tx=0x7fb24800ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:08:25.442 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:25.439+0000 7fb241ffb640 1 -- 172.21.15.104:0/1713363773 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb248017070 con 0x7fb24c071be0 2024-03-28T14:08:25.442 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:25.439+0000 7fb241ffb640 1 -- 172.21.15.104:0/1713363773 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb24800c9a0 con 0x7fb24c071be0 2024-03-28T14:08:25.442 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:25.439+0000 7fb25373a640 1 -- 172.21.15.104:0/1713363773 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb24c0b4620 con 0x7fb24c071be0 2024-03-28T14:08:25.442 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:25.439+0000 7fb241ffb640 1 -- 172.21.15.104:0/1713363773 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb24800cd70 con 0x7fb24c071be0 2024-03-28T14:08:25.442 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:25.439+0000 7fb25373a640 1 -- 172.21.15.104:0/1713363773 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb24c0b1e10 con 0x7fb24c071be0 2024-03-28T14:08:25.443 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:25.439+0000 7fb25373a640 1 -- 172.21.15.104:0/1713363773 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb24c0687c0 con 0x7fb24c071be0 2024-03-28T14:08:25.443 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:25.439+0000 7fb241ffb640 1 -- 172.21.15.104:0/1713363773 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) ==== 87803+0+0 (secure 0 0 0) 0x7fb24801a070 con 0x7fb24c071be0 2024-03-28T14:08:25.443 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:25.439+0000 7fb241ffb640 1 --2- 172.21.15.104:0/1713363773 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7fb230067fc0 0x7fb23006a480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:25.443 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:25.439+0000 7fb241ffb640 1 -- 172.21.15.104:0/1713363773 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7fb248086860 con 0x7fb24c071be0 2024-03-28T14:08:25.443 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:25.439+0000 7fb250cd5640 1 --2- 172.21.15.104:0/1713363773 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7fb230067fc0 0x7fb23006a480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:25.444 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:25.443+0000 7fb250cd5640 1 --2- 172.21.15.104:0/1713363773 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7fb230067fc0 0x7fb23006a480 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fb23c0ad000 tx=0x7fb23c008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:08:25.446 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:25.443+0000 7fb241ffb640 1 -- 172.21.15.104:0/1713363773 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179117 (secure 0 0 0) 0x7fb248021030 con 0x7fb24c071be0 2024-03-28T14:08:25.726 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:25.723+0000 7fb25373a640 1 -- 172.21.15.104:0/1713363773 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fb24c0b9290 con 0x7fb24c071be0 2024-03-28T14:08:25.727 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:25.723+0000 7fb241ffb640 1 -- 172.21.15.104:0/1713363773 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v21) ==== 93+0+10 (secure 0 0 0) 0x7fb248086020 con 0x7fb24c071be0 2024-03-28T14:08:25.727 INFO:teuthology.orchestra.run.smithi104.stdout:pg_num: 1 2024-03-28T14:08:25.730 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:25.727+0000 7fb25373a640 1 -- 172.21.15.104:0/1713363773 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7fb230067fc0 msgr2=0x7fb23006a480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:25.731 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:25.727+0000 7fb25373a640 1 --2- 172.21.15.104:0/1713363773 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7fb230067fc0 0x7fb23006a480 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fb23c0ad000 tx=0x7fb23c008040 comp rx=0 tx=0).stop 2024-03-28T14:08:25.731 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:25.727+0000 7fb25373a640 1 -- 172.21.15.104:0/1713363773 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb24c071be0 msgr2=0x7fb24c13e630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:25.731 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:25.727+0000 7fb25373a640 1 --2- 172.21.15.104:0/1713363773 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb24c071be0 0x7fb24c13e630 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7fb24800e9b0 tx=0x7fb24800ee70 comp rx=0 tx=0).stop 2024-03-28T14:08:25.731 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:25.727+0000 7fb25373a640 1 -- 172.21.15.104:0/1713363773 shutdown_connections 2024-03-28T14:08:25.731 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:25.727+0000 7fb25373a640 1 --2- 172.21.15.104:0/1713363773 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fb24c0b1830 0x7fb24c1350f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:25.731 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:25.727+0000 7fb25373a640 1 --2- 172.21.15.104:0/1713363773 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fb24c072bf0 0x7fb24c1e3a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:25.731 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:25.727+0000 7fb25373a640 1 --2- 172.21.15.104:0/1713363773 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7fb230067fc0 0x7fb23006a480 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:25.731 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:25.727+0000 7fb25373a640 1 --2- 172.21.15.104:0/1713363773 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb24c071be0 0x7fb24c13e630 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:25.731 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:25.727+0000 7fb25373a640 1 -- 172.21.15.104:0/1713363773 >> 172.21.15.104:0/1713363773 conn(0x7fb24c0b7b10 msgr2=0x7fb24c13f230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T14:08:25.731 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:25.727+0000 7fb25373a640 1 -- 172.21.15.104:0/1713363773 shutdown_connections 2024-03-28T14:08:25.731 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:25.727+0000 7fb25373a640 1 -- 172.21.15.104:0/1713363773 wait complete. 2024-03-28T14:08:25.746 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-28T14:08:25.868 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:25.867+0000 7f6fc62af640 1 Processor -- start 2024-03-28T14:08:25.869 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:25.867+0000 7f6fc62af640 1 -- start start 2024-03-28T14:08:25.873 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:25.867+0000 7f6fc62af640 1 --2- >> v2:172.21.15.171:3300/0 conn(0x7f6fc006e020 0x7f6fc00af020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:25.873 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:25.867+0000 7f6fc62af640 1 --2- >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f6fc00b58b0 0x7f6fc0072d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:25.873 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:25.867+0000 7f6fc62af640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6fc00b1170 con 0x7f6fc00b0d90 2024-03-28T14:08:25.874 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:25.867+0000 7f6fc62af640 1 -- --> v2:172.21.15.171:3300/0 -- mon_getmap magic: 0 -- 0x7f6fc00af560 con 0x7f6fc006e020 2024-03-28T14:08:25.874 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:25.867+0000 7f6fc62af640 1 -- --> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] -- mon_getmap magic: 0 -- 0x7f6fc006db90 con 0x7f6fc00b58b0 2024-03-28T14:08:25.874 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:25.867+0000 7f6fbffff640 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f6fc00b0d90 0x7f6fc014adb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:46226/0 (socket says 172.21.15.104:46226) 2024-03-28T14:08:25.874 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:25.867+0000 7f6fbffff640 1 -- 172.21.15.104:0/1250061893 learned_addr learned my addr 172.21.15.104:0/1250061893 (peer_addr_for_me v1:172.21.15.104:0/0) 2024-03-28T14:08:25.874 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:25.867+0000 7f6fbeffd640 1 -- 172.21.15.104:0/1250061893 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3723143141 0 0) 0x7f6fc00b1170 con 0x7f6fc00b0d90 2024-03-28T14:08:25.874 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:25.867+0000 7f6fbeffd640 1 -- 172.21.15.104:0/1250061893 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6fac003530 con 0x7f6fc00b0d90 2024-03-28T14:08:25.874 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:25.867+0000 7f6fbf7fe640 1 --2- 172.21.15.104:0/1250061893 >> v2:172.21.15.171:3300/0 conn(0x7f6fc006e020 0x7f6fc00af020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:25.874 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:25.867+0000 7f6fc484c640 1 --2- 172.21.15.104:0/1250061893 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f6fc00b58b0 0x7f6fc0072d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:25.874 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:25.867+0000 7f6fbeffd640 1 -- 172.21.15.104:0/1250061893 <== mon.0 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1239294431 0 0) 0x7f6fac003530 con 0x7f6fc00b0d90 2024-03-28T14:08:25.874 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:25.867+0000 7f6fbeffd640 1 -- 172.21.15.104:0/1250061893 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f6fc00b58b0 msgr2=0x7f6fc0072d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T14:08:25.874 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:25.867+0000 7f6fbeffd640 1 --2- 172.21.15.104:0/1250061893 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f6fc00b58b0 0x7f6fc0072d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:25.874 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:25.867+0000 7f6fbeffd640 1 -- 172.21.15.104:0/1250061893 >> v2:172.21.15.171:3300/0 conn(0x7f6fc006e020 msgr2=0x7f6fc00af020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T14:08:25.874 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:25.867+0000 7f6fbeffd640 1 --2- 172.21.15.104:0/1250061893 >> v2:172.21.15.171:3300/0 conn(0x7f6fc006e020 0x7f6fc00af020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:25.875 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:25.867+0000 7f6fbeffd640 1 -- 172.21.15.104:0/1250061893 --> v1:172.21.15.104:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6fc00b51e0 con 0x7f6fc00b0d90 2024-03-28T14:08:25.875 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:25.867+0000 7f6fbeffd640 1 -- 172.21.15.104:0/1250061893 <== mon.0 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3745580849 0 0) 0x7f6fb8001680 con 0x7f6fc00b0d90 2024-03-28T14:08:25.875 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:25.867+0000 7f6fbeffd640 1 -- 172.21.15.104:0/1250061893 >> v1:172.21.15.104:6789/0 conn(0x7f6fc00b0d90 legacy=0x7f6fc014adb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:25.875 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:25.867+0000 7f6fbeffd640 1 --2- 172.21.15.104:0/1250061893 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6fac0039c0 0x7f6fac003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:25.875 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:25.867+0000 7f6fbeffd640 1 --2- 172.21.15.104:0/1250061893 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f6fc00b58b0 0x7f6fac004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:25.875 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:25.867+0000 7f6fbeffd640 1 --2- 172.21.15.104:0/1250061893 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f6fac004d50 0x7f6fac0051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:25.875 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:25.867+0000 7f6fbeffd640 1 -- 172.21.15.104:0/1250061893 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f6fc00af560 con 0x7f6fac0039c0 2024-03-28T14:08:25.875 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:25.867+0000 7f6fbeffd640 1 -- 172.21.15.104:0/1250061893 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f6fc006db90 con 0x7f6fac004d50 2024-03-28T14:08:25.875 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:25.867+0000 7f6fbeffd640 1 -- 172.21.15.104:0/1250061893 --> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] -- mon_getmap magic: 0 -- 0x7f6fac00bb00 con 0x7f6fc00b58b0 2024-03-28T14:08:25.875 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:25.867+0000 7f6fbffff640 1 --2- 172.21.15.104:0/1250061893 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6fac0039c0 0x7f6fac003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:25.875 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:25.867+0000 7f6fbf7fe640 1 --2- 172.21.15.104:0/1250061893 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f6fc00b58b0 0x7f6fac004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:25.875 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:25.867+0000 7f6fbffff640 1 -- 172.21.15.104:0/1250061893 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f6fc00b58b0 msgr2=0x7f6fac004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T14:08:25.876 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:25.867+0000 7f6fbffff640 1 --2- 172.21.15.104:0/1250061893 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f6fc00b58b0 0x7f6fac004810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:25.876 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:25.867+0000 7f6fc484c640 1 --2- 172.21.15.104:0/1250061893 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f6fac004d50 0x7f6fac0051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:25.876 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:25.867+0000 7f6fbffff640 1 -- 172.21.15.104:0/1250061893 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f6fac004d50 msgr2=0x7f6fac0051d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T14:08:25.876 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:25.867+0000 7f6fbffff640 1 --2- 172.21.15.104:0/1250061893 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f6fac004d50 0x7f6fac0051d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:25.876 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:25.867+0000 7f6fbffff640 1 -- 172.21.15.104:0/1250061893 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6fac003530 con 0x7f6fac0039c0 2024-03-28T14:08:25.876 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:25.867+0000 7f6fbf7fe640 1 --2- 172.21.15.104:0/1250061893 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f6fc00b58b0 0x7f6fac004810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T14:08:25.877 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:25.871+0000 7f6fbffff640 1 --2- 172.21.15.104:0/1250061893 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6fac0039c0 0x7f6fac003de0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f6fb800c240 tx=0x7f6fb801ccc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=65c15b03db78ca21 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:08:25.877 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:25.871+0000 7f6fbeffd640 1 -- 172.21.15.104:0/1250061893 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6fb8007d60 con 0x7f6fac0039c0 2024-03-28T14:08:25.877 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:25.871+0000 7f6fbeffd640 1 -- 172.21.15.104:0/1250061893 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6fb80a9020 con 0x7f6fac0039c0 2024-03-28T14:08:25.877 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:25.871+0000 7f6fbeffd640 1 -- 172.21.15.104:0/1250061893 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6fb8005ae0 con 0x7f6fac0039c0 2024-03-28T14:08:25.877 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:25.871+0000 7f6fc62af640 1 -- 172.21.15.104:0/1250061893 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6fac0039c0 msgr2=0x7f6fac003de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:25.877 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:25.871+0000 7f6fc62af640 1 --2- 172.21.15.104:0/1250061893 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6fac0039c0 0x7f6fac003de0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f6fb800c240 tx=0x7f6fb801ccc0 comp rx=0 tx=0).stop 2024-03-28T14:08:25.878 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:25.871+0000 7f6fc484c640 1 -- 172.21.15.104:0/1250061893 reap_dead start 2024-03-28T14:08:25.878 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:25.871+0000 7f6fc62af640 1 -- 172.21.15.104:0/1250061893 shutdown_connections 2024-03-28T14:08:25.878 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:25.871+0000 7f6fc62af640 1 -- 172.21.15.104:0/1250061893 >> 172.21.15.104:0/1250061893 conn(0x7f6fc00ba400 msgr2=0x7f6fc00ba810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T14:08:25.878 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:25.871+0000 7f6fc62af640 1 -- 172.21.15.104:0/1250061893 shutdown_connections 2024-03-28T14:08:25.878 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:25.871+0000 7f6fc62af640 1 -- 172.21.15.104:0/1250061893 wait complete. 2024-03-28T14:08:25.880 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:25.871+0000 7f6fc62af640 1 Processor -- start 2024-03-28T14:08:25.881 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:25.871+0000 7f6fc62af640 1 -- start start 2024-03-28T14:08:25.881 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:25.871+0000 7f6fc62af640 1 --2- >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f6fac0039c0 0x7f6fc01e0aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:25.881 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:25.871+0000 7f6fc62af640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f6fc00b32d0 0x7f6fc00b3750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:25.881 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:25.871+0000 7f6fc62af640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6fc01e3720 0x7f6fc01e3bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:25.881 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:25.871+0000 7f6fc62af640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f6fc0152c80 con 0x7f6fc01e3720 2024-03-28T14:08:25.881 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:25.871+0000 7f6fc62af640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f6fc0152b00 con 0x7f6fc00b32d0 2024-03-28T14:08:25.881 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:25.871+0000 7f6fc62af640 1 -- --> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] -- mon_getmap magic: 0 -- 0x7f6fc0152e00 con 0x7f6fac0039c0 2024-03-28T14:08:25.881 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:25.871+0000 7f6fc484c640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6fc01e3720 0x7f6fc01e3bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:25.881 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:25.871+0000 7f6fc484c640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6fc01e3720 0x7f6fc01e3bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:42720/0 (socket says 172.21.15.104:42720) 2024-03-28T14:08:25.882 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:25.871+0000 7f6fc484c640 1 -- 172.21.15.104:0/633228038 learned_addr learned my addr 172.21.15.104:0/633228038 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-03-28T14:08:25.882 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:25.871+0000 7f6fbffff640 1 --2- 172.21.15.104:0/633228038 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f6fac0039c0 0x7f6fc01e0aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:25.882 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:25.871+0000 7f6fbf7fe640 1 --2- 172.21.15.104:0/633228038 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f6fc00b32d0 0x7f6fc00b3750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:25.882 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:25.871+0000 7f6fc484c640 1 -- 172.21.15.104:0/633228038 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f6fac0039c0 msgr2=0x7f6fc01e0aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:25.882 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:25.871+0000 7f6fc484c640 1 --2- 172.21.15.104:0/633228038 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f6fac0039c0 0x7f6fc01e0aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:25.882 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:25.871+0000 7f6fc484c640 1 -- 172.21.15.104:0/633228038 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f6fc00b32d0 msgr2=0x7f6fc00b3750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:25.882 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:25.871+0000 7f6fc484c640 1 --2- 172.21.15.104:0/633228038 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f6fc00b32d0 0x7f6fc00b3750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:25.883 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:25.871+0000 7f6fc484c640 1 -- 172.21.15.104:0/633228038 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6fc01485e0 con 0x7f6fc01e3720 2024-03-28T14:08:25.883 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:25.871+0000 7f6fbf7fe640 1 --2- 172.21.15.104:0/633228038 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f6fc00b32d0 0x7f6fc00b3750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T14:08:25.883 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:25.871+0000 7f6fc484c640 1 --2- 172.21.15.104:0/633228038 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6fc01e3720 0x7f6fc01e3bd0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f6fc00b62e0 tx=0x7f6fb0009cd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:08:25.883 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:25.875+0000 7f6fbcff9640 1 -- 172.21.15.104:0/633228038 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6fb0019070 con 0x7f6fc01e3720 2024-03-28T14:08:25.883 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:25.875+0000 7f6fbcff9640 1 -- 172.21.15.104:0/633228038 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6fb0007c90 con 0x7f6fc01e3720 2024-03-28T14:08:25.883 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:25.875+0000 7f6fc62af640 1 -- 172.21.15.104:0/633228038 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6fc01e02c0 con 0x7f6fc01e3720 2024-03-28T14:08:25.883 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:25.875+0000 7f6fbcff9640 1 -- 172.21.15.104:0/633228038 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6fb0014640 con 0x7f6fc01e3720 2024-03-28T14:08:25.883 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:25.875+0000 7f6fc62af640 1 -- 172.21.15.104:0/633228038 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6fc00b1560 con 0x7f6fc01e3720 2024-03-28T14:08:25.883 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:25.875+0000 7f6fc62af640 1 -- 172.21.15.104:0/633228038 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6fc00687c0 con 0x7f6fc01e3720 2024-03-28T14:08:25.883 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:25.875+0000 7f6fbcff9640 1 -- 172.21.15.104:0/633228038 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) ==== 87803+0+0 (secure 0 0 0) 0x7f6fb001c070 con 0x7f6fc01e3720 2024-03-28T14:08:25.883 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:25.875+0000 7f6fbcff9640 1 --2- 172.21.15.104:0/633228038 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f6fa0067fc0 0x7f6fa006a480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:25.883 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:25.875+0000 7f6fbcff9640 1 -- 172.21.15.104:0/633228038 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f6fb0085ea0 con 0x7f6fc01e3720 2024-03-28T14:08:25.883 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:25.875+0000 7f6fbffff640 1 --2- 172.21.15.104:0/633228038 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f6fa0067fc0 0x7f6fa006a480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:25.884 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:25.875+0000 7f6fbffff640 1 --2- 172.21.15.104:0/633228038 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f6fa0067fc0 0x7f6fa006a480 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f6fb80aa000 tx=0x7f6fb8001ac0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:08:25.884 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:25.879+0000 7f6fbcff9640 1 -- 172.21.15.104:0/633228038 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179117 (secure 0 0 0) 0x7f6fb00566e0 con 0x7f6fc01e3720 2024-03-28T14:08:26.065 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:26.063+0000 7f6fc62af640 1 -- 172.21.15.104:0/633228038 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f6fc002d400 con 0x7f6fc01e3720 2024-03-28T14:08:26.066 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:26.063+0000 7f6fbcff9640 1 -- 172.21.15.104:0/633228038 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v21) ==== 104+0+11 (secure 0 0 0) 0x7f6fb0055fe0 con 0x7f6fc01e3720 2024-03-28T14:08:26.066 INFO:teuthology.orchestra.run.smithi104.stdout:pg_num: 32 2024-03-28T14:08:26.069 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:26.067+0000 7f6fc62af640 1 -- 172.21.15.104:0/633228038 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f6fa0067fc0 msgr2=0x7f6fa006a480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:26.069 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:26.067+0000 7f6fc62af640 1 --2- 172.21.15.104:0/633228038 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f6fa0067fc0 0x7f6fa006a480 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f6fb80aa000 tx=0x7f6fb8001ac0 comp rx=0 tx=0).stop 2024-03-28T14:08:26.070 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:26.067+0000 7f6fc62af640 1 -- 172.21.15.104:0/633228038 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6fc01e3720 msgr2=0x7f6fc01e3bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:26.070 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:26.067+0000 7f6fc62af640 1 --2- 172.21.15.104:0/633228038 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6fc01e3720 0x7f6fc01e3bd0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f6fc00b62e0 tx=0x7f6fb0009cd0 comp rx=0 tx=0).stop 2024-03-28T14:08:26.070 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:26.067+0000 7f6fc62af640 1 -- 172.21.15.104:0/633228038 shutdown_connections 2024-03-28T14:08:26.070 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:26.067+0000 7f6fc62af640 1 --2- 172.21.15.104:0/633228038 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f6fa0067fc0 0x7f6fa006a480 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:26.070 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:26.067+0000 7f6fc62af640 1 --2- 172.21.15.104:0/633228038 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6fc01e3720 0x7f6fc01e3bd0 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:26.070 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:26.067+0000 7f6fc62af640 1 --2- 172.21.15.104:0/633228038 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f6fc00b32d0 0x7f6fc00b3750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:26.070 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:26.067+0000 7f6fc62af640 1 --2- 172.21.15.104:0/633228038 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f6fac0039c0 0x7f6fc01e0aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:26.070 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:26.067+0000 7f6fc62af640 1 -- 172.21.15.104:0/633228038 >> 172.21.15.104:0/633228038 conn(0x7f6fc00ba400 msgr2=0x7f6fc006bd20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T14:08:26.070 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:26.067+0000 7f6fc62af640 1 -- 172.21.15.104:0/633228038 shutdown_connections 2024-03-28T14:08:26.071 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:26.067+0000 7f6fc62af640 1 -- 172.21.15.104:0/633228038 wait complete. 2024-03-28T14:08:26.091 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-28T14:08:26.200 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:26.199+0000 7efd44a4e640 1 Processor -- start 2024-03-28T14:08:26.200 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:26.199+0000 7efd44a4e640 1 -- start start 2024-03-28T14:08:26.202 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:26.199+0000 7efd44a4e640 1 --2- >> v2:172.21.15.171:3300/0 conn(0x7efd4002c460 0x7efd400b54e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:26.202 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:26.199+0000 7efd44a4e640 1 --2- >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7efd401487e0 0x7efd400b5a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:26.203 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:26.199+0000 7efd44a4e640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efd4002c840 con 0x7efd4009e8e0 2024-03-28T14:08:26.203 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:26.199+0000 7efd44a4e640 1 -- --> v2:172.21.15.171:3300/0 -- mon_getmap magic: 0 -- 0x7efd40147c40 con 0x7efd4002c460 2024-03-28T14:08:26.203 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:26.199+0000 7efd44a4e640 1 -- --> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] -- mon_getmap magic: 0 -- 0x7efd400aec70 con 0x7efd401487e0 2024-03-28T14:08:26.203 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:26.199+0000 7efd3dd9b640 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7efd4009e8e0 0x7efd401480d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:46242/0 (socket says 172.21.15.104:46242) 2024-03-28T14:08:26.203 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:26.199+0000 7efd3dd9b640 1 -- 172.21.15.104:0/4043280788 learned_addr learned my addr 172.21.15.104:0/4043280788 (peer_addr_for_me v1:172.21.15.104:0/0) 2024-03-28T14:08:26.203 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:26.199+0000 7efd3ed9d640 1 --2- 172.21.15.104:0/4043280788 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7efd401487e0 0x7efd400b5a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:26.203 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:26.199+0000 7efd3e59c640 1 --2- 172.21.15.104:0/4043280788 >> v2:172.21.15.171:3300/0 conn(0x7efd4002c460 0x7efd400b54e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:26.203 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:26.199+0000 7efd3d59a640 1 -- 172.21.15.104:0/4043280788 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2434600836 0 0) 0x7efd4002c840 con 0x7efd4009e8e0 2024-03-28T14:08:26.203 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:26.199+0000 7efd3d59a640 1 -- 172.21.15.104:0/4043280788 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7efd28003530 con 0x7efd4009e8e0 2024-03-28T14:08:26.203 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:26.199+0000 7efd3d59a640 1 -- 172.21.15.104:0/4043280788 <== mon.0 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 676255507 0 0) 0x7efd28003530 con 0x7efd4009e8e0 2024-03-28T14:08:26.203 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:26.199+0000 7efd3d59a640 1 -- 172.21.15.104:0/4043280788 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7efd401487e0 msgr2=0x7efd400b5a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T14:08:26.203 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:26.199+0000 7efd3d59a640 1 --2- 172.21.15.104:0/4043280788 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7efd401487e0 0x7efd400b5a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:26.203 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:26.199+0000 7efd3d59a640 1 -- 172.21.15.104:0/4043280788 >> v2:172.21.15.171:3300/0 conn(0x7efd4002c460 msgr2=0x7efd400b54e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T14:08:26.204 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:26.199+0000 7efd3d59a640 1 --2- 172.21.15.104:0/4043280788 >> v2:172.21.15.171:3300/0 conn(0x7efd4002c460 0x7efd400b54e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:26.204 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:26.199+0000 7efd3d59a640 1 -- 172.21.15.104:0/4043280788 --> v1:172.21.15.104:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efd400aef50 con 0x7efd4009e8e0 2024-03-28T14:08:26.204 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:26.199+0000 7efd3d59a640 1 -- 172.21.15.104:0/4043280788 <== mon.0 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3745580849 0 0) 0x7efd34001680 con 0x7efd4009e8e0 2024-03-28T14:08:26.204 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:26.199+0000 7efd3d59a640 1 -- 172.21.15.104:0/4043280788 >> v1:172.21.15.104:6789/0 conn(0x7efd4009e8e0 legacy=0x7efd401480d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:26.204 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:26.199+0000 7efd3d59a640 1 --2- 172.21.15.104:0/4043280788 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7efd280039c0 0x7efd28003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:26.204 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:26.199+0000 7efd3d59a640 1 --2- 172.21.15.104:0/4043280788 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7efd28004810 0x7efd28004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:26.204 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:26.199+0000 7efd3d59a640 1 --2- 172.21.15.104:0/4043280788 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7efd401487e0 0x7efd280051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:26.204 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:26.199+0000 7efd3d59a640 1 -- 172.21.15.104:0/4043280788 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7efd40147c40 con 0x7efd28004810 2024-03-28T14:08:26.204 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:26.199+0000 7efd3d59a640 1 -- 172.21.15.104:0/4043280788 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7efd400aec70 con 0x7efd280039c0 2024-03-28T14:08:26.204 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:26.199+0000 7efd3d59a640 1 -- 172.21.15.104:0/4043280788 --> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] -- mon_getmap magic: 0 -- 0x7efd2800b9d0 con 0x7efd401487e0 2024-03-28T14:08:26.204 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:26.199+0000 7efd3dd9b640 1 --2- 172.21.15.104:0/4043280788 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7efd28004810 0x7efd28004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:26.204 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:26.199+0000 7efd3ed9d640 1 --2- 172.21.15.104:0/4043280788 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7efd401487e0 0x7efd280051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:26.204 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:26.199+0000 7efd3e59c640 1 --2- 172.21.15.104:0/4043280788 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7efd280039c0 0x7efd28003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:26.204 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:26.199+0000 7efd3dd9b640 1 -- 172.21.15.104:0/4043280788 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7efd401487e0 msgr2=0x7efd280051d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T14:08:26.205 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:26.199+0000 7efd3dd9b640 1 --2- 172.21.15.104:0/4043280788 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7efd401487e0 0x7efd280051d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:26.205 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:26.199+0000 7efd3dd9b640 1 -- 172.21.15.104:0/4043280788 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7efd280039c0 msgr2=0x7efd28003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T14:08:26.205 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:26.199+0000 7efd3dd9b640 1 --2- 172.21.15.104:0/4043280788 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7efd280039c0 0x7efd28003de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:26.205 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:26.199+0000 7efd3dd9b640 1 -- 172.21.15.104:0/4043280788 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efd28003530 con 0x7efd28004810 2024-03-28T14:08:26.205 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:26.199+0000 7efd3e59c640 1 --2- 172.21.15.104:0/4043280788 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7efd280039c0 0x7efd28003de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T14:08:26.209 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:26.203+0000 7efd3dd9b640 1 --2- 172.21.15.104:0/4043280788 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7efd28004810 0x7efd28004c90 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7efd3400c240 tx=0x7efd3401f410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f3d9f2a2842d16bd server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:08:26.209 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:26.203+0000 7efd3d59a640 1 -- 172.21.15.104:0/4043280788 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd34007d40 con 0x7efd28004810 2024-03-28T14:08:26.209 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:26.203+0000 7efd3d59a640 1 -- 172.21.15.104:0/4043280788 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efd34005b00 con 0x7efd28004810 2024-03-28T14:08:26.209 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:26.203+0000 7efd3d59a640 1 -- 172.21.15.104:0/4043280788 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd34007070 con 0x7efd28004810 2024-03-28T14:08:26.209 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:26.203+0000 7efd44a4e640 1 -- 172.21.15.104:0/4043280788 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7efd28004810 msgr2=0x7efd28004c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:26.209 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:26.203+0000 7efd44a4e640 1 --2- 172.21.15.104:0/4043280788 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7efd28004810 0x7efd28004c90 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7efd3400c240 tx=0x7efd3401f410 comp rx=0 tx=0).stop 2024-03-28T14:08:26.209 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:26.203+0000 7efd3ed9d640 1 -- 172.21.15.104:0/4043280788 reap_dead start 2024-03-28T14:08:26.210 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:26.203+0000 7efd44a4e640 1 -- 172.21.15.104:0/4043280788 shutdown_connections 2024-03-28T14:08:26.210 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:26.203+0000 7efd44a4e640 1 -- 172.21.15.104:0/4043280788 >> 172.21.15.104:0/4043280788 conn(0x7efd400b95f0 msgr2=0x7efd400b99d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T14:08:26.210 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:26.203+0000 7efd44a4e640 1 -- 172.21.15.104:0/4043280788 shutdown_connections 2024-03-28T14:08:26.210 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:26.203+0000 7efd44a4e640 1 -- 172.21.15.104:0/4043280788 wait complete. 2024-03-28T14:08:26.210 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:26.203+0000 7efd44a4e640 1 Processor -- start 2024-03-28T14:08:26.210 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:26.203+0000 7efd44a4e640 1 -- start start 2024-03-28T14:08:26.210 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:26.203+0000 7efd44a4e640 1 --2- >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7efd28004810 0x7efd4013f210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:26.210 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:26.203+0000 7efd44a4e640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7efd401410a0 0x7efd400b64a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:26.210 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:26.203+0000 7efd44a4e640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7efd401db5f0 0x7efd401dba70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:26.210 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:26.203+0000 7efd44a4e640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7efd400b07a0 con 0x7efd401db5f0 2024-03-28T14:08:26.210 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:26.203+0000 7efd44a4e640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7efd400b0620 con 0x7efd401410a0 2024-03-28T14:08:26.210 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:26.203+0000 7efd44a4e640 1 -- --> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] -- mon_getmap magic: 0 -- 0x7efd400b5a20 con 0x7efd28004810 2024-03-28T14:08:26.210 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:26.203+0000 7efd3ed9d640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7efd401db5f0 0x7efd401dba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:26.211 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:26.203+0000 7efd3ed9d640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7efd401db5f0 0x7efd401dba70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:42732/0 (socket says 172.21.15.104:42732) 2024-03-28T14:08:26.211 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:26.203+0000 7efd3ed9d640 1 -- 172.21.15.104:0/3811972426 learned_addr learned my addr 172.21.15.104:0/3811972426 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-03-28T14:08:26.211 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:26.203+0000 7efd3dd9b640 1 --2- 172.21.15.104:0/3811972426 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7efd401410a0 0x7efd400b64a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:26.211 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:26.203+0000 7efd3ed9d640 1 -- 172.21.15.104:0/3811972426 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7efd28004810 msgr2=0x7efd4013f210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:26.211 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:26.203+0000 7efd3ed9d640 1 --2- 172.21.15.104:0/3811972426 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7efd28004810 0x7efd4013f210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:26.211 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:26.203+0000 7efd3ed9d640 1 -- 172.21.15.104:0/3811972426 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7efd401410a0 msgr2=0x7efd400b64a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:26.211 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:26.203+0000 7efd3ed9d640 1 --2- 172.21.15.104:0/3811972426 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7efd401410a0 0x7efd400b64a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:26.211 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:26.203+0000 7efd3ed9d640 1 -- 172.21.15.104:0/3811972426 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efd40140380 con 0x7efd401db5f0 2024-03-28T14:08:26.211 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:26.203+0000 7efd3dd9b640 1 --2- 172.21.15.104:0/3811972426 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7efd401410a0 0x7efd400b64a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T14:08:26.211 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:26.203+0000 7efd3ed9d640 1 --2- 172.21.15.104:0/3811972426 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7efd401db5f0 0x7efd401dba70 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7efd400af7a0 tx=0x7efd380115a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:08:26.211 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:26.203+0000 7efd2effd640 1 -- 172.21.15.104:0/3811972426 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd38011e10 con 0x7efd401db5f0 2024-03-28T14:08:26.211 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:26.203+0000 7efd2effd640 1 -- 172.21.15.104:0/3811972426 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efd3801b070 con 0x7efd401db5f0 2024-03-28T14:08:26.212 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:26.203+0000 7efd44a4e640 1 -- 172.21.15.104:0/3811972426 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efd401415f0 con 0x7efd401db5f0 2024-03-28T14:08:26.212 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:26.203+0000 7efd2effd640 1 -- 172.21.15.104:0/3811972426 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd380039a0 con 0x7efd401db5f0 2024-03-28T14:08:26.212 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:26.203+0000 7efd44a4e640 1 -- 172.21.15.104:0/3811972426 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efd40141f90 con 0x7efd401db5f0 2024-03-28T14:08:26.212 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:26.207+0000 7efd44a4e640 1 -- 172.21.15.104:0/3811972426 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efd400687c0 con 0x7efd401db5f0 2024-03-28T14:08:26.212 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:26.207+0000 7efd2effd640 1 -- 172.21.15.104:0/3811972426 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) ==== 87803+0+0 (secure 0 0 0) 0x7efd38003b40 con 0x7efd401db5f0 2024-03-28T14:08:26.212 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:26.207+0000 7efd2effd640 1 --2- 172.21.15.104:0/3811972426 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7efd20067ff0 0x7efd2006a4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:26.212 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:26.207+0000 7efd2effd640 1 -- 172.21.15.104:0/3811972426 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7efd38086bf0 con 0x7efd401db5f0 2024-03-28T14:08:26.212 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:26.207+0000 7efd3e59c640 1 --2- 172.21.15.104:0/3811972426 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7efd20067ff0 0x7efd2006a4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:26.213 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:26.207+0000 7efd3e59c640 1 --2- 172.21.15.104:0/3811972426 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7efd20067ff0 0x7efd2006a4b0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7efd40073340 tx=0x7efd30008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:08:26.218 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:26.215+0000 7efd2effd640 1 -- 172.21.15.104:0/3811972426 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179117 (secure 0 0 0) 0x7efd38057430 con 0x7efd401db5f0 2024-03-28T14:08:26.379 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:26.375+0000 7efd44a4e640 1 -- 172.21.15.104:0/3811972426 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7efd4009e150 con 0x7efd401db5f0 2024-03-28T14:08:26.380 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:26.379+0000 7efd2effd640 1 -- 172.21.15.104:0/3811972426 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v21) ==== 100+0+11 (secure 0 0 0) 0x7efd38056d30 con 0x7efd401db5f0 2024-03-28T14:08:26.380 INFO:teuthology.orchestra.run.smithi104.stdout:pg_num: 64 2024-03-28T14:08:26.382 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:26.379+0000 7efd44a4e640 1 -- 172.21.15.104:0/3811972426 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7efd20067ff0 msgr2=0x7efd2006a4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:26.383 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:26.379+0000 7efd44a4e640 1 --2- 172.21.15.104:0/3811972426 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7efd20067ff0 0x7efd2006a4b0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7efd40073340 tx=0x7efd30008040 comp rx=0 tx=0).stop 2024-03-28T14:08:26.383 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:26.379+0000 7efd44a4e640 1 -- 172.21.15.104:0/3811972426 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7efd401db5f0 msgr2=0x7efd401dba70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:26.383 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:26.379+0000 7efd44a4e640 1 --2- 172.21.15.104:0/3811972426 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7efd401db5f0 0x7efd401dba70 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7efd400af7a0 tx=0x7efd380115a0 comp rx=0 tx=0).stop 2024-03-28T14:08:26.383 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:26.379+0000 7efd44a4e640 1 -- 172.21.15.104:0/3811972426 shutdown_connections 2024-03-28T14:08:26.383 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:26.379+0000 7efd44a4e640 1 --2- 172.21.15.104:0/3811972426 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7efd20067ff0 0x7efd2006a4b0 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:26.383 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:26.379+0000 7efd44a4e640 1 --2- 172.21.15.104:0/3811972426 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7efd401db5f0 0x7efd401dba70 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:26.384 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:26.379+0000 7efd44a4e640 1 --2- 172.21.15.104:0/3811972426 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7efd401410a0 0x7efd400b64a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:26.384 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:26.379+0000 7efd44a4e640 1 --2- 172.21.15.104:0/3811972426 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7efd28004810 0x7efd4013f210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:26.384 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:26.379+0000 7efd44a4e640 1 -- 172.21.15.104:0/3811972426 >> 172.21.15.104:0/3811972426 conn(0x7efd400b95f0 msgr2=0x7efd400b7bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T14:08:26.384 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:26.379+0000 7efd44a4e640 1 -- 172.21.15.104:0/3811972426 shutdown_connections 2024-03-28T14:08:26.384 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:26.379+0000 7efd44a4e640 1 -- 172.21.15.104:0/3811972426 wait complete. 2024-03-28T14:08:26.397 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-28T14:08:26.510 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:26.507+0000 7f9c32517640 1 Processor -- start 2024-03-28T14:08:26.510 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:26.507+0000 7f9c32517640 1 -- start start 2024-03-28T14:08:26.513 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:26.507+0000 7f9c32517640 1 --2- >> v2:172.21.15.171:3300/0 conn(0x7f9c2c0b12a0 0x7f9c2c0b1df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:26.513 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:26.507+0000 7f9c32517640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9c2c0b1680 con 0x7f9c2c1462e0 2024-03-28T14:08:26.513 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:26.507+0000 7f9c32517640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9c2c071fc0 con 0x7f9c2c145e60 2024-03-28T14:08:26.513 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:26.507+0000 7f9c32517640 1 -- --> v2:172.21.15.171:3300/0 -- mon_getmap magic: 0 -- 0x7f9c2c145940 con 0x7f9c2c0b12a0 2024-03-28T14:08:26.513 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:26.507+0000 7f9c30ab4640 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f9c2c1462e0 0x7f9c2c14feb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:46248/0 (socket says 172.21.15.104:46248) 2024-03-28T14:08:26.513 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:26.507+0000 7f9c30ab4640 1 -- 172.21.15.104:0/2195264888 learned_addr learned my addr 172.21.15.104:0/2195264888 (peer_addr_for_me v1:172.21.15.104:0/0) 2024-03-28T14:08:26.513 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:26.507+0000 7f9c2bfff640 1 --2- 172.21.15.104:0/2195264888 >> v2:172.21.15.171:3300/0 conn(0x7f9c2c0b12a0 0x7f9c2c0b1df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:26.513 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:26.507+0000 7f9c2affd640 1 -- 172.21.15.104:0/2195264888 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2827456702 0 0) 0x7f9c2c0b1680 con 0x7f9c2c1462e0 2024-03-28T14:08:26.513 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:26.507+0000 7f9c2affd640 1 -- 172.21.15.104:0/2195264888 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9c18003530 con 0x7f9c2c1462e0 2024-03-28T14:08:26.514 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:26.511+0000 7f9c2affd640 1 -- 172.21.15.104:0/2195264888 <== mon.0 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 315649559 0 0) 0x7f9c18003530 con 0x7f9c2c1462e0 2024-03-28T14:08:26.514 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:26.511+0000 7f9c2affd640 1 -- 172.21.15.104:0/2195264888 >> v2:172.21.15.171:3300/0 conn(0x7f9c2c0b12a0 msgr2=0x7f9c2c0b1df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T14:08:26.514 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:26.511+0000 7f9c2affd640 1 --2- 172.21.15.104:0/2195264888 >> v2:172.21.15.171:3300/0 conn(0x7f9c2c0b12a0 0x7f9c2c0b1df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:26.514 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:26.511+0000 7f9c2affd640 1 -- 172.21.15.104:0/2195264888 >> v1:172.21.15.171:6789/0 conn(0x7f9c2c145e60 legacy=0x7f9c2c072f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:26.514 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:26.511+0000 7f9c2affd640 1 -- 172.21.15.104:0/2195264888 --> v1:172.21.15.104:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9c2c1455f0 con 0x7f9c2c1462e0 2024-03-28T14:08:26.514 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:26.511+0000 7f9c2affd640 1 -- 172.21.15.104:0/2195264888 <== mon.0 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3745580849 0 0) 0x7f9c24001a00 con 0x7f9c2c1462e0 2024-03-28T14:08:26.514 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:26.511+0000 7f9c2affd640 1 -- 172.21.15.104:0/2195264888 >> v1:172.21.15.104:6789/0 conn(0x7f9c2c1462e0 legacy=0x7f9c2c14feb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:26.515 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:26.511+0000 7f9c2affd640 1 --2- 172.21.15.104:0/2195264888 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f9c180039c0 0x7f9c18003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:26.515 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:26.511+0000 7f9c2affd640 1 --2- 172.21.15.104:0/2195264888 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f9c18004810 0x7f9c18004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:26.515 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:26.511+0000 7f9c2affd640 1 --2- 172.21.15.104:0/2195264888 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9c180051d0 0x7f9c1800b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:26.515 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:26.511+0000 7f9c2affd640 1 -- 172.21.15.104:0/2195264888 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f9c2c145940 con 0x7f9c180051d0 2024-03-28T14:08:26.515 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:26.511+0000 7f9c2affd640 1 -- 172.21.15.104:0/2195264888 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f9c1800c170 con 0x7f9c18004810 2024-03-28T14:08:26.515 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:26.511+0000 7f9c2affd640 1 -- 172.21.15.104:0/2195264888 --> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] -- mon_getmap magic: 0 -- 0x7f9c1800c370 con 0x7f9c180039c0 2024-03-28T14:08:26.515 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:26.511+0000 7f9c30ab4640 1 --2- 172.21.15.104:0/2195264888 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9c180051d0 0x7f9c1800b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:26.515 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:26.511+0000 7f9c2bfff640 1 --2- 172.21.15.104:0/2195264888 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f9c180039c0 0x7f9c18003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:26.515 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:26.511+0000 7f9c30ab4640 1 -- 172.21.15.104:0/2195264888 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f9c180039c0 msgr2=0x7f9c18003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T14:08:26.516 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:26.511+0000 7f9c30ab4640 1 --2- 172.21.15.104:0/2195264888 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f9c180039c0 0x7f9c18003de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:26.516 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:26.511+0000 7f9c30ab4640 1 -- 172.21.15.104:0/2195264888 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f9c18004810 msgr2=0x7f9c18004c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T14:08:26.516 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:26.511+0000 7f9c30ab4640 1 --2- 172.21.15.104:0/2195264888 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f9c18004810 0x7f9c18004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:26.516 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:26.511+0000 7f9c30ab4640 1 -- 172.21.15.104:0/2195264888 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9c18003530 con 0x7f9c180051d0 2024-03-28T14:08:26.520 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:26.511+0000 7f9c30ab4640 1 -- 172.21.15.104:0/2195264888 reap_dead start 2024-03-28T14:08:26.520 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:26.511+0000 7f9c30ab4640 1 --2- 172.21.15.104:0/2195264888 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9c180051d0 0x7f9c1800b9d0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f9c2400a860 tx=0x7f9c2401fae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5f5b430f43b70ee server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:08:26.520 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:26.511+0000 7f9c2affd640 1 -- 172.21.15.104:0/2195264888 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c2400a480 con 0x7f9c180051d0 2024-03-28T14:08:26.520 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:26.511+0000 7f9c2affd640 1 -- 172.21.15.104:0/2195264888 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9c240aa710 con 0x7f9c180051d0 2024-03-28T14:08:26.520 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:26.511+0000 7f9c2affd640 1 -- 172.21.15.104:0/2195264888 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c240aaae0 con 0x7f9c180051d0 2024-03-28T14:08:26.520 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:26.511+0000 7f9c32517640 1 -- 172.21.15.104:0/2195264888 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9c180051d0 msgr2=0x7f9c1800b9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:26.520 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:26.511+0000 7f9c32517640 1 --2- 172.21.15.104:0/2195264888 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9c180051d0 0x7f9c1800b9d0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f9c2400a860 tx=0x7f9c2401fae0 comp rx=0 tx=0).stop 2024-03-28T14:08:26.521 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:26.511+0000 7f9c32517640 1 -- 172.21.15.104:0/2195264888 shutdown_connections 2024-03-28T14:08:26.521 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:26.511+0000 7f9c32517640 1 --2- 172.21.15.104:0/2195264888 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9c180051d0 0x7f9c1800b9d0 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:26.521 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:26.511+0000 7f9c32517640 1 -- 172.21.15.104:0/2195264888 >> 172.21.15.104:0/2195264888 conn(0x7f9c2c0b7b10 msgr2=0x7f9c2c0b7f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T14:08:26.521 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:26.511+0000 7f9c32517640 1 -- 172.21.15.104:0/2195264888 shutdown_connections 2024-03-28T14:08:26.521 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:26.511+0000 7f9c32517640 1 -- 172.21.15.104:0/2195264888 wait complete. 2024-03-28T14:08:26.521 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:26.515+0000 7f9c32517640 1 Processor -- start 2024-03-28T14:08:26.521 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:26.515+0000 7f9c32517640 1 -- start start 2024-03-28T14:08:26.521 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:26.515+0000 7f9c32517640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9c180051d0 0x7f9c2c1cb670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:26.521 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:26.515+0000 7f9c32517640 1 --2- >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f9c2c1cf1f0 0x7f9c2c1deaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:26.521 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:26.515+0000 7f9c32517640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f9c2c0b2530 0x7f9c2c1cdf90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:26.521 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:26.515+0000 7f9c32517640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f9c2c0b7960 con 0x7f9c180051d0 2024-03-28T14:08:26.521 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:26.515+0000 7f9c32517640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f9c2c0b77e0 con 0x7f9c2c0b2530 2024-03-28T14:08:26.521 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:26.515+0000 7f9c32517640 1 -- --> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] -- mon_getmap magic: 0 -- 0x7f9c2c071be0 con 0x7f9c2c1cf1f0 2024-03-28T14:08:26.521 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:26.515+0000 7f9c2bfff640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9c180051d0 0x7f9c2c1cb670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:26.522 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:26.515+0000 7f9c2bfff640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9c180051d0 0x7f9c2c1cb670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:42748/0 (socket says 172.21.15.104:42748) 2024-03-28T14:08:26.522 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:26.515+0000 7f9c2bfff640 1 -- 172.21.15.104:0/3579004200 learned_addr learned my addr 172.21.15.104:0/3579004200 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-03-28T14:08:26.522 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:26.515+0000 7f9c30ab4640 1 --2- 172.21.15.104:0/3579004200 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f9c2c0b2530 0x7f9c2c1cdf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:26.522 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:26.515+0000 7f9c2bfff640 1 -- 172.21.15.104:0/3579004200 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f9c2c1cf1f0 msgr2=0x7f9c2c1deaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:26.522 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:26.515+0000 7f9c2b7fe640 1 --2- 172.21.15.104:0/3579004200 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f9c2c1cf1f0 0x7f9c2c1deaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:26.522 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:26.515+0000 7f9c2bfff640 1 --2- 172.21.15.104:0/3579004200 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f9c2c1cf1f0 0x7f9c2c1deaa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:26.522 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:26.515+0000 7f9c2bfff640 1 -- 172.21.15.104:0/3579004200 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f9c2c0b2530 msgr2=0x7f9c2c1cdf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:26.522 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:26.515+0000 7f9c2bfff640 1 --2- 172.21.15.104:0/3579004200 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f9c2c0b2530 0x7f9c2c1cdf90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:26.522 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:26.515+0000 7f9c2bfff640 1 -- 172.21.15.104:0/3579004200 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9c2c1de2a0 con 0x7f9c180051d0 2024-03-28T14:08:26.522 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:26.515+0000 7f9c30ab4640 1 --2- 172.21.15.104:0/3579004200 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f9c2c0b2530 0x7f9c2c1cdf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T14:08:26.522 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:26.515+0000 7f9c2bfff640 1 --2- 172.21.15.104:0/3579004200 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9c180051d0 0x7f9c2c1cb670 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f9c14010890 tx=0x7f9c14010d50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:08:26.522 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:26.515+0000 7f9c28ff9640 1 -- 172.21.15.104:0/3579004200 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c1400f040 con 0x7f9c180051d0 2024-03-28T14:08:26.522 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:26.515+0000 7f9c28ff9640 1 -- 172.21.15.104:0/3579004200 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9c1400eb60 con 0x7f9c180051d0 2024-03-28T14:08:26.522 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:26.515+0000 7f9c32517640 1 -- 172.21.15.104:0/3579004200 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9c2c0b37f0 con 0x7f9c180051d0 2024-03-28T14:08:26.523 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:26.515+0000 7f9c28ff9640 1 -- 172.21.15.104:0/3579004200 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c1401c400 con 0x7f9c180051d0 2024-03-28T14:08:26.523 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:26.515+0000 7f9c32517640 1 -- 172.21.15.104:0/3579004200 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9c2c143250 con 0x7f9c180051d0 2024-03-28T14:08:26.523 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:26.515+0000 7f9c32517640 1 -- 172.21.15.104:0/3579004200 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9c2c0687c0 con 0x7f9c180051d0 2024-03-28T14:08:26.523 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:26.519+0000 7f9c28ff9640 1 -- 172.21.15.104:0/3579004200 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) ==== 87803+0+0 (secure 0 0 0) 0x7f9c1401c5a0 con 0x7f9c180051d0 2024-03-28T14:08:26.523 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:26.519+0000 7f9c28ff9640 1 --2- 172.21.15.104:0/3579004200 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f9c0c067fc0 0x7f9c0c06a480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:26.523 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:26.519+0000 7f9c28ff9640 1 -- 172.21.15.104:0/3579004200 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f9c14086b40 con 0x7f9c180051d0 2024-03-28T14:08:26.523 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:26.519+0000 7f9c2b7fe640 1 --2- 172.21.15.104:0/3579004200 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f9c0c067fc0 0x7f9c0c06a480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:26.523 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:26.519+0000 7f9c2b7fe640 1 --2- 172.21.15.104:0/3579004200 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f9c0c067fc0 0x7f9c0c06a480 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f9c2c1cceb0 tx=0x7f9c1c009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:08:26.524 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:26.523+0000 7f9c28ff9640 1 -- 172.21.15.104:0/3579004200 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179117 (secure 0 0 0) 0x7f9c14018030 con 0x7f9c180051d0 2024-03-28T14:08:26.692 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:26.691+0000 7f9c32517640 1 -- 172.21.15.104:0/3579004200 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f9c2c145290 con 0x7f9c180051d0 2024-03-28T14:08:26.693 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:26.691+0000 7f9c28ff9640 1 -- 172.21.15.104:0/3579004200 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v21) ==== 80+0+0 (secure 0 0 0) 0x7f9c14057380 con 0x7f9c180051d0 2024-03-28T14:08:26.693 INFO:teuthology.orchestra.run.smithi104.stderr:listed 0 entries 2024-03-28T14:08:26.695 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:26.691+0000 7f9c32517640 1 -- 172.21.15.104:0/3579004200 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f9c0c067fc0 msgr2=0x7f9c0c06a480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:26.696 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:26.691+0000 7f9c32517640 1 --2- 172.21.15.104:0/3579004200 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f9c0c067fc0 0x7f9c0c06a480 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f9c2c1cceb0 tx=0x7f9c1c009040 comp rx=0 tx=0).stop 2024-03-28T14:08:26.696 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:26.691+0000 7f9c32517640 1 -- 172.21.15.104:0/3579004200 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9c180051d0 msgr2=0x7f9c2c1cb670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:26.696 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:26.691+0000 7f9c32517640 1 --2- 172.21.15.104:0/3579004200 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9c180051d0 0x7f9c2c1cb670 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f9c14010890 tx=0x7f9c14010d50 comp rx=0 tx=0).stop 2024-03-28T14:08:26.696 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:26.691+0000 7f9c32517640 1 -- 172.21.15.104:0/3579004200 shutdown_connections 2024-03-28T14:08:26.696 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:26.691+0000 7f9c32517640 1 --2- 172.21.15.104:0/3579004200 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f9c2c0b2530 0x7f9c2c1cdf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:26.696 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:26.691+0000 7f9c32517640 1 --2- 172.21.15.104:0/3579004200 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f9c2c1cf1f0 0x7f9c2c1deaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:26.696 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:26.691+0000 7f9c32517640 1 --2- 172.21.15.104:0/3579004200 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f9c0c067fc0 0x7f9c0c06a480 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:26.696 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:26.691+0000 7f9c32517640 1 --2- 172.21.15.104:0/3579004200 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9c180051d0 0x7f9c2c1cb670 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:26.696 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:26.691+0000 7f9c32517640 1 -- 172.21.15.104:0/3579004200 >> 172.21.15.104:0/3579004200 conn(0x7f9c2c0b7b10 msgr2=0x7f9c2c1d18b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T14:08:26.696 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:26.695+0000 7f9c32517640 1 -- 172.21.15.104:0/3579004200 shutdown_connections 2024-03-28T14:08:26.696 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:26.695+0000 7f9c32517640 1 -- 172.21.15.104:0/3579004200 wait complete. 2024-03-28T14:08:26.710 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi104.front.sepia.ceph.com... 2024-03-28T14:08:26.711 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T14:08:26.711 DEBUG:teuthology.orchestra.run.smithi104:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2024-03-28T14:08:26.736 INFO:teuthology.orchestra.run:waiting for 300 2024-03-28T14:08:26.737 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi104.front.sepia.ceph.com 2024-03-28T14:08:26.737 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T14:08:26.737 DEBUG:teuthology.orchestra.run.smithi104:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-03-28T14:08:26.786 INFO:tasks.cephfs.cephfs_test_case:[] 2024-03-28T14:08:26.786 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T14:08:26.951 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:26.951+0000 7f92999dc640 1 Processor -- start 2024-03-28T14:08:26.952 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:26.951+0000 7f92999dc640 1 -- start start 2024-03-28T14:08:26.955 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:26.951+0000 7f92999dc640 1 --2- >> v2:172.21.15.171:3300/0 conn(0x7f92940ae1a0 0x7f9294072e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:26.955 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:26.951+0000 7f92999dc640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f92940b0be0 con 0x7f9294143ef0 2024-03-28T14:08:26.955 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:26.951+0000 7f92999dc640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f92940ae6c0 con 0x7f92940addc0 2024-03-28T14:08:26.955 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:26.951+0000 7f92999dc640 1 -- --> v2:172.21.15.171:3300/0 -- mon_getmap magic: 0 -- 0x7f92940aec80 con 0x7f92940ae1a0 2024-03-28T14:08:26.955 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:26.951+0000 7f92937fe640 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f9294143ef0 0x7f9294144340 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:41548/0 (socket says 172.21.15.104:41548) 2024-03-28T14:08:26.955 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:26.951+0000 7f92937fe640 1 -- 172.21.15.104:0/1227123390 learned_addr learned my addr 172.21.15.104:0/1227123390 (peer_addr_for_me v1:172.21.15.104:0/0) 2024-03-28T14:08:26.955 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:26.951+0000 7f92927fc640 1 --2- 172.21.15.104:0/1227123390 >> v2:172.21.15.171:3300/0 conn(0x7f92940ae1a0 0x7f9294072e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:26.955 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:26.951+0000 7f9291ffb640 1 -- 172.21.15.104:0/1227123390 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 414469302 0 0) 0x7f92940b0be0 con 0x7f9294143ef0 2024-03-28T14:08:26.955 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:26.951+0000 7f9291ffb640 1 -- 172.21.15.104:0/1227123390 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f927c003530 con 0x7f9294143ef0 2024-03-28T14:08:26.955 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:26.951+0000 7f9291ffb640 1 -- 172.21.15.104:0/1227123390 <== mon.0 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4202317022 0 0) 0x7f927c003530 con 0x7f9294143ef0 2024-03-28T14:08:26.955 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:26.951+0000 7f9291ffb640 1 -- 172.21.15.104:0/1227123390 >> v2:172.21.15.171:3300/0 conn(0x7f92940ae1a0 msgr2=0x7f9294072e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T14:08:26.955 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:26.951+0000 7f9291ffb640 1 --2- 172.21.15.104:0/1227123390 >> v2:172.21.15.171:3300/0 conn(0x7f92940ae1a0 0x7f9294072e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:26.956 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:26.951+0000 7f9291ffb640 1 -- 172.21.15.104:0/1227123390 >> v1:172.21.15.171:6789/0 conn(0x7f92940addc0 legacy=0x7f92940af6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:26.956 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:26.951+0000 7f9291ffb640 1 -- 172.21.15.104:0/1227123390 --> v1:172.21.15.104:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f92940ae9a0 con 0x7f9294143ef0 2024-03-28T14:08:26.956 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:26.951+0000 7f9291ffb640 1 -- 172.21.15.104:0/1227123390 <== mon.0 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3745580849 0 0) 0x7f928c001a00 con 0x7f9294143ef0 2024-03-28T14:08:26.956 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:26.951+0000 7f9291ffb640 1 -- 172.21.15.104:0/1227123390 >> v1:172.21.15.104:6789/0 conn(0x7f9294143ef0 legacy=0x7f9294144340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:26.956 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:26.951+0000 7f9291ffb640 1 --2- 172.21.15.104:0/1227123390 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f927c003a10 0x7f927c003e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:26.956 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:26.951+0000 7f9291ffb640 1 --2- 172.21.15.104:0/1227123390 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f927c004810 0x7f927c004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:26.956 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:26.951+0000 7f9291ffb640 1 --2- 172.21.15.104:0/1227123390 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f927c005220 0x7f927c00b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:26.956 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:26.951+0000 7f9291ffb640 1 -- 172.21.15.104:0/1227123390 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f92940aec80 con 0x7f927c005220 2024-03-28T14:08:26.956 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:26.951+0000 7f9291ffb640 1 -- 172.21.15.104:0/1227123390 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f927c00bf10 con 0x7f927c003a10 2024-03-28T14:08:26.956 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:26.951+0000 7f9291ffb640 1 -- 172.21.15.104:0/1227123390 --> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] -- mon_getmap magic: 0 -- 0x7f927c00c110 con 0x7f927c004810 2024-03-28T14:08:26.956 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:26.951+0000 7f92937fe640 1 --2- 172.21.15.104:0/1227123390 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f927c005220 0x7f927c00b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:26.956 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:26.951+0000 7f9292ffd640 1 --2- 172.21.15.104:0/1227123390 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f927c003a10 0x7f927c003e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:26.957 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:26.951+0000 7f92927fc640 1 --2- 172.21.15.104:0/1227123390 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f927c004810 0x7f927c004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:26.957 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:26.951+0000 7f92937fe640 1 -- 172.21.15.104:0/1227123390 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f927c004810 msgr2=0x7f927c004c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T14:08:26.957 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:26.951+0000 7f92937fe640 1 --2- 172.21.15.104:0/1227123390 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f927c004810 0x7f927c004c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:26.957 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:26.951+0000 7f92937fe640 1 -- 172.21.15.104:0/1227123390 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f927c003a10 msgr2=0x7f927c003e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T14:08:26.957 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:26.951+0000 7f92937fe640 1 --2- 172.21.15.104:0/1227123390 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f927c003a10 0x7f927c003e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:26.957 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:26.951+0000 7f92937fe640 1 -- 172.21.15.104:0/1227123390 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f927c003530 con 0x7f927c005220 2024-03-28T14:08:26.960 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:26.955+0000 7f92937fe640 1 -- 172.21.15.104:0/1227123390 reap_dead start 2024-03-28T14:08:26.961 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:26.955+0000 7f92937fe640 1 --2- 172.21.15.104:0/1227123390 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f927c005220 0x7f927c00b9d0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f928c00a760 tx=0x7f928c01f9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a57da591220dc5db server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:08:26.961 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:26.955+0000 7f9291ffb640 1 -- 172.21.15.104:0/1227123390 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f928c00a3b0 con 0x7f927c005220 2024-03-28T14:08:26.961 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:26.955+0000 7f9291ffb640 1 -- 172.21.15.104:0/1227123390 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f928c0ac470 con 0x7f927c005220 2024-03-28T14:08:26.961 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:26.955+0000 7f9291ffb640 1 -- 172.21.15.104:0/1227123390 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f928c0ac860 con 0x7f927c005220 2024-03-28T14:08:26.961 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:26.955+0000 7f92999dc640 1 -- 172.21.15.104:0/1227123390 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f927c005220 msgr2=0x7f927c00b9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:26.961 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:26.955+0000 7f92999dc640 1 --2- 172.21.15.104:0/1227123390 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f927c005220 0x7f927c00b9d0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f928c00a760 tx=0x7f928c01f9c0 comp rx=0 tx=0).stop 2024-03-28T14:08:26.961 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:26.955+0000 7f92999dc640 1 -- 172.21.15.104:0/1227123390 shutdown_connections 2024-03-28T14:08:26.961 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:26.955+0000 7f92999dc640 1 --2- 172.21.15.104:0/1227123390 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f927c005220 0x7f927c00b9d0 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:26.961 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:26.955+0000 7f92999dc640 1 -- 172.21.15.104:0/1227123390 >> 172.21.15.104:0/1227123390 conn(0x7f92940b58a0 msgr2=0x7f92940b5cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T14:08:26.961 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:26.955+0000 7f92999dc640 1 -- 172.21.15.104:0/1227123390 shutdown_connections 2024-03-28T14:08:26.961 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:26.955+0000 7f92999dc640 1 -- 172.21.15.104:0/1227123390 wait complete. 2024-03-28T14:08:26.961 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:26.955+0000 7f92999dc640 1 Processor -- start 2024-03-28T14:08:26.961 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:26.955+0000 7f92999dc640 1 -- start start 2024-03-28T14:08:26.961 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:26.955+0000 7f92999dc640 1 --2- >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f927c005220 0x7f9294142770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:26.962 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:26.955+0000 7f92999dc640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9294140c40 0x7f92941d8820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:26.962 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:26.955+0000 7f92999dc640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f92941d8d60 0x7f92941dbd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:26.962 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:26.955+0000 7f92999dc640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f92941444c0 con 0x7f9294140c40 2024-03-28T14:08:26.962 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:26.955+0000 7f92999dc640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f9294144340 con 0x7f92941d8d60 2024-03-28T14:08:26.962 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:26.955+0000 7f92999dc640 1 -- --> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] -- mon_getmap magic: 0 -- 0x7f9294144640 con 0x7f927c005220 2024-03-28T14:08:26.962 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:26.955+0000 7f92927fc640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9294140c40 0x7f92941d8820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:26.962 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:26.955+0000 7f92927fc640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9294140c40 0x7f92941d8820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:58840/0 (socket says 172.21.15.104:58840) 2024-03-28T14:08:26.962 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:26.955+0000 7f92927fc640 1 -- 172.21.15.104:0/3642099639 learned_addr learned my addr 172.21.15.104:0/3642099639 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-03-28T14:08:26.962 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:26.955+0000 7f9292ffd640 1 --2- 172.21.15.104:0/3642099639 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f927c005220 0x7f9294142770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:26.962 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:26.955+0000 7f92937fe640 1 --2- 172.21.15.104:0/3642099639 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f92941d8d60 0x7f92941dbd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:26.962 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:26.955+0000 7f92927fc640 1 -- 172.21.15.104:0/3642099639 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f927c005220 msgr2=0x7f9294142770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:26.962 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:26.955+0000 7f92927fc640 1 --2- 172.21.15.104:0/3642099639 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f927c005220 0x7f9294142770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:26.962 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:26.955+0000 7f92927fc640 1 -- 172.21.15.104:0/3642099639 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f92941d8d60 msgr2=0x7f92941dbd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:26.963 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:26.955+0000 7f92927fc640 1 --2- 172.21.15.104:0/3642099639 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f92941d8d60 0x7f92941dbd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:26.963 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:26.955+0000 7f92927fc640 1 -- 172.21.15.104:0/3642099639 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f92941d91f0 con 0x7f9294140c40 2024-03-28T14:08:26.963 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:26.955+0000 7f92937fe640 1 --2- 172.21.15.104:0/3642099639 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f92941d8d60 0x7f92941dbd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T14:08:26.963 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:26.955+0000 7f92927fc640 1 --2- 172.21.15.104:0/3642099639 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9294140c40 0x7f92941d8820 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f9284010690 tx=0x7f9284010b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:08:26.963 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:26.955+0000 7f927bfff640 1 -- 172.21.15.104:0/3642099639 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9284019070 con 0x7f9294140c40 2024-03-28T14:08:26.963 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:26.959+0000 7f927bfff640 1 -- 172.21.15.104:0/3642099639 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f928400ec00 con 0x7f9294140c40 2024-03-28T14:08:26.963 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:26.959+0000 7f92999dc640 1 -- 172.21.15.104:0/3642099639 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f929406fa50 con 0x7f9294140c40 2024-03-28T14:08:26.963 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:26.959+0000 7f927bfff640 1 -- 172.21.15.104:0/3642099639 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f928400b040 con 0x7f9294140c40 2024-03-28T14:08:26.963 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:26.959+0000 7f92999dc640 1 -- 172.21.15.104:0/3642099639 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9294141240 con 0x7f9294140c40 2024-03-28T14:08:26.963 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:26.959+0000 7f92999dc640 1 -- 172.21.15.104:0/3642099639 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f929406f840 con 0x7f9294140c40 2024-03-28T14:08:26.963 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:26.959+0000 7f927bfff640 1 -- 172.21.15.104:0/3642099639 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) ==== 87803+0+0 (secure 0 0 0) 0x7f928401c070 con 0x7f9294140c40 2024-03-28T14:08:26.964 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:26.959+0000 7f927bfff640 1 --2- 172.21.15.104:0/3642099639 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f9280067fc0 0x7f928006a480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:26.964 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:26.959+0000 7f927bfff640 1 -- 172.21.15.104:0/3642099639 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f9284086640 con 0x7f9294140c40 2024-03-28T14:08:26.964 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:26.959+0000 7f9292ffd640 1 --2- 172.21.15.104:0/3642099639 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f9280067fc0 0x7f928006a480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:26.964 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:26.959+0000 7f9292ffd640 1 --2- 172.21.15.104:0/3642099639 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f9280067fc0 0x7f928006a480 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f9288007fd0 tx=0x7f9288009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:08:26.970 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:26.967+0000 7f927bfff640 1 -- 172.21.15.104:0/3642099639 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179117 (secure 0 0 0) 0x7f9284023030 con 0x7f9294140c40 2024-03-28T14:08:27.153 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:27.151+0000 7f92999dc640 1 -- 172.21.15.104:0/3642099639 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f92940687c0 con 0x7f9294140c40 2024-03-28T14:08:27.154 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:27.151+0000 7f927bfff640 1 -- 172.21.15.104:0/3642099639 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 7 v7) ==== 93+0+3533 (secure 0 0 0) 0x7f9284025300 con 0x7f9294140c40 2024-03-28T14:08:27.154 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-03-28T14:08:27.155 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":7,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4229,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.171:6835/1775867467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6834","nonce":1775867467},{"type":"v1","addr":"172.21.15.171:6835","nonce":1775867467}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":7,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T14:07:59.027458+0000","modified":"2024-03-28T14:08:02.057923+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4245},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4245":{"gid":4245,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.104:6833/3184673325","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":3184673325},{"type":"v1","addr":"172.21.15.104:6833","nonce":3184673325}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4245,"qdb_cluster":[4245]},"id":1}]} 2024-03-28T14:08:27.155 INFO:teuthology.orchestra.run.smithi104.stderr:dumped fsmap epoch 7 2024-03-28T14:08:27.156 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:27.155+0000 7f92999dc640 1 -- 172.21.15.104:0/3642099639 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f9280067fc0 msgr2=0x7f928006a480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:27.156 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:27.155+0000 7f92999dc640 1 --2- 172.21.15.104:0/3642099639 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f9280067fc0 0x7f928006a480 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f9288007fd0 tx=0x7f9288009040 comp rx=0 tx=0).stop 2024-03-28T14:08:27.156 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:27.155+0000 7f92999dc640 1 -- 172.21.15.104:0/3642099639 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9294140c40 msgr2=0x7f92941d8820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:27.157 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:27.155+0000 7f92999dc640 1 --2- 172.21.15.104:0/3642099639 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9294140c40 0x7f92941d8820 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f9284010690 tx=0x7f9284010b50 comp rx=0 tx=0).stop 2024-03-28T14:08:27.157 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:27.155+0000 7f92999dc640 1 -- 172.21.15.104:0/3642099639 shutdown_connections 2024-03-28T14:08:27.157 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:27.155+0000 7f92999dc640 1 --2- 172.21.15.104:0/3642099639 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f92941d8d60 0x7f92941dbd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:27.157 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:27.155+0000 7f92999dc640 1 --2- 172.21.15.104:0/3642099639 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f9280067fc0 0x7f928006a480 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:27.157 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:27.155+0000 7f92999dc640 1 --2- 172.21.15.104:0/3642099639 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9294140c40 0x7f92941d8820 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:27.157 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:27.155+0000 7f92999dc640 1 --2- 172.21.15.104:0/3642099639 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f927c005220 0x7f9294142770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:27.157 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:27.155+0000 7f92999dc640 1 -- 172.21.15.104:0/3642099639 >> 172.21.15.104:0/3642099639 conn(0x7f92940b58a0 msgr2=0x7f92940b2150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T14:08:27.157 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:27.155+0000 7f92999dc640 1 -- 172.21.15.104:0/3642099639 shutdown_connections 2024-03-28T14:08:27.157 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:27.155+0000 7f92999dc640 1 -- 172.21.15.104:0/3642099639 wait complete. 2024-03-28T14:08:27.176 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-28T14:08:27.304 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:27.303+0000 7f6995551640 1 Processor -- start 2024-03-28T14:08:27.304 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:27.303+0000 7f6995551640 1 -- start start 2024-03-28T14:08:27.306 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:27.303+0000 7f6995551640 1 --2- >> v2:172.21.15.171:3300/0 conn(0x7f6990144cf0 0x7f69900b0010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:27.307 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:27.303+0000 7f6995551640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f69900b6780 con 0x7f69900af9b0 2024-03-28T14:08:27.307 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:27.303+0000 7f6995551640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f699006dde0 con 0x7f69900b63a0 2024-03-28T14:08:27.307 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:27.303+0000 7f6995551640 1 -- --> v2:172.21.15.171:3300/0 -- mon_getmap magic: 0 -- 0x7f69900af610 con 0x7f6990144cf0 2024-03-28T14:08:27.307 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:27.303+0000 7f698e7fc640 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f69900af9b0 0x7f69901445e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:41562/0 (socket says 172.21.15.104:41562) 2024-03-28T14:08:27.307 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:27.303+0000 7f698e7fc640 1 -- 172.21.15.104:0/17315628 learned_addr learned my addr 172.21.15.104:0/17315628 (peer_addr_for_me v1:172.21.15.104:0/0) 2024-03-28T14:08:27.307 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:27.303+0000 7f698f7fe640 1 --2- 172.21.15.104:0/17315628 >> v2:172.21.15.171:3300/0 conn(0x7f6990144cf0 0x7f69900b0010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:27.307 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:27.303+0000 7f698dffb640 1 -- 172.21.15.104:0/17315628 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1059320589 0 0) 0x7f69900b6780 con 0x7f69900af9b0 2024-03-28T14:08:27.307 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:27.303+0000 7f698dffb640 1 -- 172.21.15.104:0/17315628 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f697c003530 con 0x7f69900af9b0 2024-03-28T14:08:27.307 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:27.303+0000 7f698dffb640 1 -- 172.21.15.104:0/17315628 <== mon.0 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 766020749 0 0) 0x7f697c003530 con 0x7f69900af9b0 2024-03-28T14:08:27.307 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:27.303+0000 7f698dffb640 1 -- 172.21.15.104:0/17315628 >> v2:172.21.15.171:3300/0 conn(0x7f6990144cf0 msgr2=0x7f69900b0010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T14:08:27.307 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:27.303+0000 7f698dffb640 1 --2- 172.21.15.104:0/17315628 >> v2:172.21.15.171:3300/0 conn(0x7f6990144cf0 0x7f69900b0010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:27.307 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:27.303+0000 7f698dffb640 1 -- 172.21.15.104:0/17315628 >> v1:172.21.15.171:6789/0 conn(0x7f69900b63a0 legacy=0x7f6990141670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:27.308 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:27.303+0000 7f698dffb640 1 -- 172.21.15.104:0/17315628 --> v1:172.21.15.104:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f699006e0c0 con 0x7f69900af9b0 2024-03-28T14:08:27.308 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:27.303+0000 7f698dffb640 1 -- 172.21.15.104:0/17315628 <== mon.0 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3745580849 0 0) 0x7f6978001680 con 0x7f69900af9b0 2024-03-28T14:08:27.308 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:27.303+0000 7f698dffb640 1 -- 172.21.15.104:0/17315628 >> v1:172.21.15.104:6789/0 conn(0x7f69900af9b0 legacy=0x7f69901445e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:27.308 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:27.303+0000 7f698dffb640 1 --2- 172.21.15.104:0/17315628 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f697c0039c0 0x7f697c003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:27.308 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:27.303+0000 7f698dffb640 1 --2- 172.21.15.104:0/17315628 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f697c004810 0x7f697c004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:27.308 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:27.303+0000 7f698dffb640 1 --2- 172.21.15.104:0/17315628 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f697c0051d0 0x7f697c00dbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:27.308 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:27.303+0000 7f698dffb640 1 -- 172.21.15.104:0/17315628 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f697c00e350 con 0x7f697c004810 2024-03-28T14:08:27.308 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:27.303+0000 7f698dffb640 1 -- 172.21.15.104:0/17315628 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f697c00e500 con 0x7f697c0039c0 2024-03-28T14:08:27.308 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:27.303+0000 7f698dffb640 1 -- 172.21.15.104:0/17315628 --> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] -- mon_getmap magic: 0 -- 0x7f697c00e700 con 0x7f697c0051d0 2024-03-28T14:08:27.308 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:27.303+0000 7f698e7fc640 1 --2- 172.21.15.104:0/17315628 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f697c004810 0x7f697c004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:27.308 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:27.303+0000 7f698effd640 1 --2- 172.21.15.104:0/17315628 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f697c0039c0 0x7f697c003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:27.308 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:27.303+0000 7f698f7fe640 1 --2- 172.21.15.104:0/17315628 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f697c0051d0 0x7f697c00dbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:27.308 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:27.303+0000 7f698e7fc640 1 -- 172.21.15.104:0/17315628 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f697c0051d0 msgr2=0x7f697c00dbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T14:08:27.309 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:27.303+0000 7f698e7fc640 1 --2- 172.21.15.104:0/17315628 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f697c0051d0 0x7f697c00dbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:27.309 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:27.303+0000 7f698e7fc640 1 -- 172.21.15.104:0/17315628 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f697c0039c0 msgr2=0x7f697c003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T14:08:27.309 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:27.303+0000 7f698e7fc640 1 --2- 172.21.15.104:0/17315628 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f697c0039c0 0x7f697c003de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:27.309 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:27.303+0000 7f698e7fc640 1 -- 172.21.15.104:0/17315628 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f697c003530 con 0x7f697c004810 2024-03-28T14:08:27.309 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:27.303+0000 7f698f7fe640 1 -- 172.21.15.104:0/17315628 reap_dead start 2024-03-28T14:08:27.310 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:27.307+0000 7f698e7fc640 1 --2- 172.21.15.104:0/17315628 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f697c004810 0x7f697c004c90 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f697800c240 tx=0x7f697801cc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c0ebe8357c0794b2 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:08:27.310 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:27.307+0000 7f698dffb640 1 -- 172.21.15.104:0/17315628 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6978007d20 con 0x7f697c004810 2024-03-28T14:08:27.311 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:27.307+0000 7f698dffb640 1 -- 172.21.15.104:0/17315628 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f69780a9070 con 0x7f697c004810 2024-03-28T14:08:27.311 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:27.307+0000 7f698dffb640 1 -- 172.21.15.104:0/17315628 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6978005da0 con 0x7f697c004810 2024-03-28T14:08:27.313 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:27.307+0000 7f6995551640 1 -- 172.21.15.104:0/17315628 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f697c004810 msgr2=0x7f697c004c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:27.313 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:27.307+0000 7f6995551640 1 --2- 172.21.15.104:0/17315628 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f697c004810 0x7f697c004c90 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f697800c240 tx=0x7f697801cc00 comp rx=0 tx=0).stop 2024-03-28T14:08:27.313 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:27.307+0000 7f6995551640 1 -- 172.21.15.104:0/17315628 shutdown_connections 2024-03-28T14:08:27.313 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:27.307+0000 7f6995551640 1 --2- 172.21.15.104:0/17315628 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f697c004810 0x7f697c004c90 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:27.313 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:27.307+0000 7f6995551640 1 -- 172.21.15.104:0/17315628 >> 172.21.15.104:0/17315628 conn(0x7f69900ba400 msgr2=0x7f69900ba810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T14:08:27.313 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:27.307+0000 7f6995551640 1 -- 172.21.15.104:0/17315628 shutdown_connections 2024-03-28T14:08:27.313 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:27.307+0000 7f6995551640 1 -- 172.21.15.104:0/17315628 wait complete. 2024-03-28T14:08:27.316 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:27.307+0000 7f6995551640 1 Processor -- start 2024-03-28T14:08:27.316 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:27.311+0000 7f6995551640 1 -- start start 2024-03-28T14:08:27.317 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:27.311+0000 7f6995551640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f697c004810 0x7f6990147290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:27.317 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:27.311+0000 7f6995551640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6990140cd0 0x7f699013f550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:27.317 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:27.311+0000 7f6995551640 1 --2- >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f69901df350 0x7f6990145d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:27.317 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:27.311+0000 7f6995551640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f6990147bd0 con 0x7f6990140cd0 2024-03-28T14:08:27.317 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:27.311+0000 7f6995551640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f6990147a50 con 0x7f697c004810 2024-03-28T14:08:27.317 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:27.311+0000 7f6995551640 1 -- --> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] -- mon_getmap magic: 0 -- 0x7f6990147d50 con 0x7f69901df350 2024-03-28T14:08:27.317 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:27.311+0000 7f698e7fc640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6990140cd0 0x7f699013f550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:27.317 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:27.311+0000 7f698e7fc640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6990140cd0 0x7f699013f550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:58858/0 (socket says 172.21.15.104:58858) 2024-03-28T14:08:27.317 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:27.311+0000 7f698e7fc640 1 -- 172.21.15.104:0/2448812995 learned_addr learned my addr 172.21.15.104:0/2448812995 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-03-28T14:08:27.317 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:27.311+0000 7f698effd640 1 --2- 172.21.15.104:0/2448812995 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f697c004810 0x7f6990147290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:27.318 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:27.311+0000 7f698f7fe640 1 --2- 172.21.15.104:0/2448812995 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f69901df350 0x7f6990145d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:27.318 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:27.311+0000 7f698e7fc640 1 -- 172.21.15.104:0/2448812995 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f69901df350 msgr2=0x7f6990145d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:27.318 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:27.311+0000 7f698e7fc640 1 --2- 172.21.15.104:0/2448812995 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f69901df350 0x7f6990145d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:27.318 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:27.311+0000 7f698e7fc640 1 -- 172.21.15.104:0/2448812995 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f697c004810 msgr2=0x7f6990147290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:27.318 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:27.311+0000 7f698e7fc640 1 --2- 172.21.15.104:0/2448812995 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f697c004810 0x7f6990147290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:27.318 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:27.311+0000 7f698e7fc640 1 -- 172.21.15.104:0/2448812995 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f699013fc30 con 0x7f6990140cd0 2024-03-28T14:08:27.318 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:27.311+0000 7f698f7fe640 1 --2- 172.21.15.104:0/2448812995 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f69901df350 0x7f6990145d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T14:08:27.318 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:27.311+0000 7f698e7fc640 1 --2- 172.21.15.104:0/2448812995 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6990140cd0 0x7f699013f550 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f697801c670 tx=0x7f697c005840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:08:27.318 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:27.311+0000 7f69877fe640 1 -- 172.21.15.104:0/2448812995 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69780bc070 con 0x7f6990140cd0 2024-03-28T14:08:27.318 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:27.311+0000 7f69877fe640 1 -- 172.21.15.104:0/2448812995 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6978007140 con 0x7f6990140cd0 2024-03-28T14:08:27.318 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:27.311+0000 7f6995551640 1 -- 172.21.15.104:0/2448812995 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f699013ccc0 con 0x7f6990140cd0 2024-03-28T14:08:27.319 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:27.311+0000 7f69877fe640 1 -- 172.21.15.104:0/2448812995 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6978007d20 con 0x7f6990140cd0 2024-03-28T14:08:27.319 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:27.311+0000 7f6995551640 1 -- 172.21.15.104:0/2448812995 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f69901404d0 con 0x7f6990140cd0 2024-03-28T14:08:27.319 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:27.311+0000 7f6995551640 1 -- 172.21.15.104:0/2448812995 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f69900687c0 con 0x7f6990140cd0 2024-03-28T14:08:27.319 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:27.311+0000 7f69877fe640 1 -- 172.21.15.104:0/2448812995 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) ==== 87803+0+0 (secure 0 0 0) 0x7f69780bf070 con 0x7f6990140cd0 2024-03-28T14:08:27.319 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:27.311+0000 7f69877fe640 1 --2- 172.21.15.104:0/2448812995 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f6970067fc0 0x7f697006a480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:27.319 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:27.311+0000 7f69877fe640 1 -- 172.21.15.104:0/2448812995 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f697811fcc0 con 0x7f6990140cd0 2024-03-28T14:08:27.319 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:27.315+0000 7f698effd640 1 --2- 172.21.15.104:0/2448812995 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f6970067fc0 0x7f697006a480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:27.319 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:27.315+0000 7f698effd640 1 --2- 172.21.15.104:0/2448812995 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f6970067fc0 0x7f697006a480 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f6990051260 tx=0x7f6980009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:08:27.320 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:27.319+0000 7f69877fe640 1 -- 172.21.15.104:0/2448812995 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179117 (secure 0 0 0) 0x7f6978016910 con 0x7f6990140cd0 2024-03-28T14:08:27.480 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:27.479+0000 7f6995551640 1 -- 172.21.15.104:0/2448812995 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f6990145640 con 0x7f6990140cd0 2024-03-28T14:08:27.483 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:27.479+0000 7f69877fe640 1 -- 172.21.15.104:0/2448812995 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) ==== 74+0+16629 (secure 0 0 0) 0x7f69780efeb0 con 0x7f6990140cd0 2024-03-28T14:08:27.483 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-03-28T14:08:27.483 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":21,"fsid":"428a90f2-3351-435f-8fcf-f7e66c78a13e","created":"2024-03-28T14:07:34.862458+0000","modified":"2024-03-28T14:08:00.671587+0000","last_up_change":"2024-03-28T14:07:46.028122+0000","last_in_change":"2024-03-28T14:07:37.157486+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-28T14:07:47.622988+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-28T14:07:57.147450+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-28T14:07:57.990975+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"1cb70540-4e42-402f-a446-aaec4ca83007","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6800","nonce":3745690918},{"type":"v1","addr":"172.21.15.104:6801","nonce":3745690918}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":3745690918},{"type":"v1","addr":"172.21.15.104:6803","nonce":3745690918}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":3745690918},{"type":"v1","addr":"172.21.15.104:6807","nonce":3745690918}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":3745690918},{"type":"v1","addr":"172.21.15.104:6805","nonce":3745690918}]},"public_addr":"172.21.15.104:6801/3745690918","cluster_addr":"172.21.15.104:6803/3745690918","heartbeat_back_addr":"172.21.15.104:6807/3745690918","heartbeat_front_addr":"172.21.15.104:6805/3745690918","state":["exists","up"]},{"osd":1,"uuid":"3ab7afdd-b980-418c-81f2-004e16de3dca","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":19438674},{"type":"v1","addr":"172.21.15.104:6810","nonce":19438674}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":19438674},{"type":"v1","addr":"172.21.15.104:6814","nonce":19438674}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":19438674},{"type":"v1","addr":"172.21.15.104:6823","nonce":19438674}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":19438674},{"type":"v1","addr":"172.21.15.104:6819","nonce":19438674}]},"public_addr":"172.21.15.104:6810/19438674","cluster_addr":"172.21.15.104:6814/19438674","heartbeat_back_addr":"172.21.15.104:6823/19438674","heartbeat_front_addr":"172.21.15.104:6819/19438674","state":["exists","up"]},{"osd":2,"uuid":"3e2651c8-ecae-4a9e-a053-27240a79eac2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":2352012626},{"type":"v1","addr":"172.21.15.104:6825","nonce":2352012626}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6826","nonce":2352012626},{"type":"v1","addr":"172.21.15.104:6827","nonce":2352012626}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":2352012626},{"type":"v1","addr":"172.21.15.104:6831","nonce":2352012626}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6828","nonce":2352012626},{"type":"v1","addr":"172.21.15.104:6829","nonce":2352012626}]},"public_addr":"172.21.15.104:6825/2352012626","cluster_addr":"172.21.15.104:6827/2352012626","heartbeat_back_addr":"172.21.15.104:6831/2352012626","heartbeat_front_addr":"172.21.15.104:6829/2352012626","state":["exists","up"]},{"osd":3,"uuid":"925c7e0a-38ef-47cd-b288-8bec8456fb17","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6809","nonce":3568347381},{"type":"v1","addr":"172.21.15.104:6811","nonce":3568347381}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6813","nonce":3568347381},{"type":"v1","addr":"172.21.15.104:6815","nonce":3568347381}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":3568347381},{"type":"v1","addr":"172.21.15.104:6822","nonce":3568347381}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6817","nonce":3568347381},{"type":"v1","addr":"172.21.15.104:6818","nonce":3568347381}]},"public_addr":"172.21.15.104:6811/3568347381","cluster_addr":"172.21.15.104:6815/3568347381","heartbeat_back_addr":"172.21.15.104:6822/3568347381","heartbeat_front_addr":"172.21.15.104:6818/3568347381","state":["exists","up"]},{"osd":4,"uuid":"4a8fac86-e595-4a0e-96e3-57edb3cc3f57","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6800","nonce":2107001077},{"type":"v1","addr":"172.21.15.171:6801","nonce":2107001077}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6802","nonce":2107001077},{"type":"v1","addr":"172.21.15.171:6803","nonce":2107001077}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6806","nonce":2107001077},{"type":"v1","addr":"172.21.15.171:6807","nonce":2107001077}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6804","nonce":2107001077},{"type":"v1","addr":"172.21.15.171:6805","nonce":2107001077}]},"public_addr":"172.21.15.171:6801/2107001077","cluster_addr":"172.21.15.171:6803/2107001077","heartbeat_back_addr":"172.21.15.171:6807/2107001077","heartbeat_front_addr":"172.21.15.171:6805/2107001077","state":["exists","up"]},{"osd":5,"uuid":"e48d50b0-beae-40ae-b15a-9a84dbfe88ab","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6808","nonce":2934826925},{"type":"v1","addr":"172.21.15.171:6810","nonce":2934826925}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6812","nonce":2934826925},{"type":"v1","addr":"172.21.15.171:6814","nonce":2934826925}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6819","nonce":2934826925},{"type":"v1","addr":"172.21.15.171:6821","nonce":2934826925}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6816","nonce":2934826925},{"type":"v1","addr":"172.21.15.171:6817","nonce":2934826925}]},"public_addr":"172.21.15.171:6810/2934826925","cluster_addr":"172.21.15.171:6814/2934826925","heartbeat_back_addr":"172.21.15.171:6821/2934826925","heartbeat_front_addr":"172.21.15.171:6817/2934826925","state":["exists","up"]},{"osd":6,"uuid":"5e7c3292-9936-4f35-b267-aed9d4e49323","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6824","nonce":436742919},{"type":"v1","addr":"172.21.15.171:6825","nonce":436742919}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6826","nonce":436742919},{"type":"v1","addr":"172.21.15.171:6827","nonce":436742919}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6830","nonce":436742919},{"type":"v1","addr":"172.21.15.171:6831","nonce":436742919}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6828","nonce":436742919},{"type":"v1","addr":"172.21.15.171:6829","nonce":436742919}]},"public_addr":"172.21.15.171:6825/436742919","cluster_addr":"172.21.15.171:6827/436742919","heartbeat_back_addr":"172.21.15.171:6831/436742919","heartbeat_front_addr":"172.21.15.171:6829/436742919","state":["exists","up"]},{"osd":7,"uuid":"339b8414-d7f0-4fb0-a446-efaf635cb7ff","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6809","nonce":2055631175},{"type":"v1","addr":"172.21.15.171:6811","nonce":2055631175}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6813","nonce":2055631175},{"type":"v1","addr":"172.21.15.171:6815","nonce":2055631175}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6822","nonce":2055631175},{"type":"v1","addr":"172.21.15.171:6823","nonce":2055631175}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6818","nonce":2055631175},{"type":"v1","addr":"172.21.15.171:6820","nonce":2055631175}]},"public_addr":"172.21.15.171:6811/2055631175","cluster_addr":"172.21.15.171:6815/2055631175","heartbeat_back_addr":"172.21.15.171:6823/2055631175","heartbeat_front_addr":"172.21.15.171:6820/2055631175","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T14:07:42.668881+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T14:07:42.708397+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T14:07:42.634191+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T14:07:42.733662+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T14:07:42.651541+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T14:07:42.783181+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T14:07:42.766310+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T14:07:42.670797+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-28T14:08:27.485 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:27.483+0000 7f6995551640 1 -- 172.21.15.104:0/2448812995 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f6970067fc0 msgr2=0x7f697006a480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:27.485 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:27.483+0000 7f6995551640 1 --2- 172.21.15.104:0/2448812995 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f6970067fc0 0x7f697006a480 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f6990051260 tx=0x7f6980009040 comp rx=0 tx=0).stop 2024-03-28T14:08:27.485 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:27.483+0000 7f6995551640 1 -- 172.21.15.104:0/2448812995 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6990140cd0 msgr2=0x7f699013f550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:27.485 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:27.483+0000 7f6995551640 1 --2- 172.21.15.104:0/2448812995 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6990140cd0 0x7f699013f550 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f697801c670 tx=0x7f697c005840 comp rx=0 tx=0).stop 2024-03-28T14:08:27.486 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:27.483+0000 7f6995551640 1 -- 172.21.15.104:0/2448812995 shutdown_connections 2024-03-28T14:08:27.486 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:27.483+0000 7f6995551640 1 --2- 172.21.15.104:0/2448812995 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f69901df350 0x7f6990145d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:27.486 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:27.483+0000 7f6995551640 1 --2- 172.21.15.104:0/2448812995 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f6970067fc0 0x7f697006a480 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:27.486 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:27.483+0000 7f6995551640 1 --2- 172.21.15.104:0/2448812995 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6990140cd0 0x7f699013f550 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:27.486 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:27.483+0000 7f6995551640 1 --2- 172.21.15.104:0/2448812995 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f697c004810 0x7f6990147290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:27.486 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:27.483+0000 7f6995551640 1 -- 172.21.15.104:0/2448812995 >> 172.21.15.104:0/2448812995 conn(0x7f69900ba400 msgr2=0x7f69900ba7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T14:08:27.486 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:27.483+0000 7f6995551640 1 -- 172.21.15.104:0/2448812995 shutdown_connections 2024-03-28T14:08:27.486 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:27.483+0000 7f6995551640 1 -- 172.21.15.104:0/2448812995 wait complete. 2024-03-28T14:08:27.501 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-28T14:07:47.622988+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-28T14:07:57.147450+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-28T14:07:57.990975+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-28T14:08:27.502 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-28T14:08:27.615 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:27.611+0000 7f78ce878640 1 Processor -- start 2024-03-28T14:08:27.615 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:27.611+0000 7f78ce878640 1 -- start start 2024-03-28T14:08:27.618 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:27.615+0000 7f78ce878640 1 --2- >> v2:172.21.15.171:3300/0 conn(0x7f78c806dd10 0x7f78c8072cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:27.618 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:27.615+0000 7f78ce878640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f78c8073230 con 0x7f78c80b5120 2024-03-28T14:08:27.618 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:27.615+0000 7f78ce878640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f78c806e0f0 con 0x7f78c80b5600 2024-03-28T14:08:27.618 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:27.615+0000 7f78ce878640 1 -- --> v2:172.21.15.171:3300/0 -- mon_getmap magic: 0 -- 0x7f78c80b0ee0 con 0x7f78c806dd10 2024-03-28T14:08:27.618 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:27.615+0000 7f78cce15640 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f78c80b5120 0x7f78c8152280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:41570/0 (socket says 172.21.15.104:41570) 2024-03-28T14:08:27.618 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:27.615+0000 7f78cce15640 1 -- 172.21.15.104:0/836916057 learned_addr learned my addr 172.21.15.104:0/836916057 (peer_addr_for_me v1:172.21.15.104:0/0) 2024-03-28T14:08:27.618 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:27.615+0000 7f78c7fff640 1 --2- 172.21.15.104:0/836916057 >> v2:172.21.15.171:3300/0 conn(0x7f78c806dd10 0x7f78c8072cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:27.618 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:27.615+0000 7f78c6ffd640 1 -- 172.21.15.104:0/836916057 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 490919979 0 0) 0x7f78c8073230 con 0x7f78c80b5120 2024-03-28T14:08:27.618 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:27.615+0000 7f78c6ffd640 1 -- 172.21.15.104:0/836916057 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f78b0003530 con 0x7f78c80b5120 2024-03-28T14:08:27.618 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:27.615+0000 7f78c6ffd640 1 -- 172.21.15.104:0/836916057 <== mon.0 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3656785449 0 0) 0x7f78b0003530 con 0x7f78c80b5120 2024-03-28T14:08:27.618 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:27.615+0000 7f78c6ffd640 1 -- 172.21.15.104:0/836916057 >> v2:172.21.15.171:3300/0 conn(0x7f78c806dd10 msgr2=0x7f78c8072cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T14:08:27.619 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:27.615+0000 7f78c6ffd640 1 --2- 172.21.15.104:0/836916057 >> v2:172.21.15.171:3300/0 conn(0x7f78c806dd10 0x7f78c8072cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:27.619 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:27.615+0000 7f78c6ffd640 1 -- 172.21.15.104:0/836916057 >> v1:172.21.15.171:6789/0 conn(0x7f78c80b5600 legacy=0x7f78c81417b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:27.619 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:27.615+0000 7f78c6ffd640 1 -- 172.21.15.104:0/836916057 --> v1:172.21.15.104:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f78c80b0b70 con 0x7f78c80b5120 2024-03-28T14:08:27.619 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:27.615+0000 7f78c6ffd640 1 -- 172.21.15.104:0/836916057 <== mon.0 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3745580849 0 0) 0x7f78c0001a00 con 0x7f78c80b5120 2024-03-28T14:08:27.619 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:27.615+0000 7f78c6ffd640 1 -- 172.21.15.104:0/836916057 >> v1:172.21.15.104:6789/0 conn(0x7f78c80b5120 legacy=0x7f78c8152280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:27.619 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:27.615+0000 7f78c6ffd640 1 --2- 172.21.15.104:0/836916057 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f78b00039c0 0x7f78b0003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:27.619 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:27.615+0000 7f78c6ffd640 1 --2- 172.21.15.104:0/836916057 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f78b0004810 0x7f78b0004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:27.619 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:27.615+0000 7f78c6ffd640 1 --2- 172.21.15.104:0/836916057 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f78b00051d0 0x7f78b000b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:27.619 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:27.615+0000 7f78c6ffd640 1 -- 172.21.15.104:0/836916057 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f78c80b0ee0 con 0x7f78b00051d0 2024-03-28T14:08:27.619 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:27.615+0000 7f78c6ffd640 1 -- 172.21.15.104:0/836916057 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f78b000c070 con 0x7f78b0004810 2024-03-28T14:08:27.619 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:27.615+0000 7f78c6ffd640 1 -- 172.21.15.104:0/836916057 --> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] -- mon_getmap magic: 0 -- 0x7f78b000c270 con 0x7f78b00039c0 2024-03-28T14:08:27.619 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:27.615+0000 7f78cce15640 1 --2- 172.21.15.104:0/836916057 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f78b00051d0 0x7f78b000b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:27.619 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:27.615+0000 7f78c7fff640 1 --2- 172.21.15.104:0/836916057 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f78b00039c0 0x7f78b0003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:27.619 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:27.615+0000 7f78c77fe640 1 --2- 172.21.15.104:0/836916057 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f78b0004810 0x7f78b0004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:27.620 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:27.615+0000 7f78cce15640 1 -- 172.21.15.104:0/836916057 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f78b00039c0 msgr2=0x7f78b0003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T14:08:27.620 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:27.615+0000 7f78cce15640 1 --2- 172.21.15.104:0/836916057 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f78b00039c0 0x7f78b0003de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:27.620 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:27.615+0000 7f78cce15640 1 -- 172.21.15.104:0/836916057 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f78b0004810 msgr2=0x7f78b0004c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T14:08:27.620 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:27.615+0000 7f78cce15640 1 --2- 172.21.15.104:0/836916057 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f78b0004810 0x7f78b0004c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:27.620 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:27.615+0000 7f78cce15640 1 -- 172.21.15.104:0/836916057 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f78b0003530 con 0x7f78b00051d0 2024-03-28T14:08:27.620 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:27.615+0000 7f78c7fff640 1 --2- 172.21.15.104:0/836916057 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f78b00039c0 0x7f78b0003de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T14:08:27.620 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:27.615+0000 7f78c77fe640 1 --2- 172.21.15.104:0/836916057 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f78b0004810 0x7f78b0004c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T14:08:27.623 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:27.615+0000 7f78cce15640 1 -- 172.21.15.104:0/836916057 reap_dead start 2024-03-28T14:08:27.623 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:27.615+0000 7f78cce15640 1 --2- 172.21.15.104:0/836916057 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f78b00051d0 0x7f78b000b9d0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f78c000a850 tx=0x7f78c001d3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6eaed11ef38d09bd server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:08:27.623 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:27.619+0000 7f78c6ffd640 1 -- 172.21.15.104:0/836916057 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78c000a4a0 con 0x7f78b00051d0 2024-03-28T14:08:27.623 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:27.619+0000 7f78c6ffd640 1 -- 172.21.15.104:0/836916057 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f78c00ad650 con 0x7f78b00051d0 2024-03-28T14:08:27.623 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:27.619+0000 7f78c6ffd640 1 -- 172.21.15.104:0/836916057 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78c00ada40 con 0x7f78b00051d0 2024-03-28T14:08:27.623 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:27.619+0000 7f78ce878640 1 -- 172.21.15.104:0/836916057 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f78b00051d0 msgr2=0x7f78b000b9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:27.624 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:27.619+0000 7f78ce878640 1 --2- 172.21.15.104:0/836916057 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f78b00051d0 0x7f78b000b9d0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f78c000a850 tx=0x7f78c001d3a0 comp rx=0 tx=0).stop 2024-03-28T14:08:27.624 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:27.619+0000 7f78ce878640 1 -- 172.21.15.104:0/836916057 shutdown_connections 2024-03-28T14:08:27.624 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:27.619+0000 7f78ce878640 1 --2- 172.21.15.104:0/836916057 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f78b00051d0 0x7f78b000b9d0 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:27.624 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:27.619+0000 7f78ce878640 1 -- 172.21.15.104:0/836916057 >> 172.21.15.104:0/836916057 conn(0x7f78c80ba390 msgr2=0x7f78c80ba7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T14:08:27.624 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:27.619+0000 7f78ce878640 1 -- 172.21.15.104:0/836916057 shutdown_connections 2024-03-28T14:08:27.624 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:27.619+0000 7f78ce878640 1 -- 172.21.15.104:0/836916057 wait complete. 2024-03-28T14:08:27.624 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:27.619+0000 7f78ce878640 1 Processor -- start 2024-03-28T14:08:27.624 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:27.619+0000 7f78ce878640 1 -- start start 2024-03-28T14:08:27.624 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:27.619+0000 7f78ce878640 1 --2- >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f78b00051d0 0x7f78c813cec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:27.624 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:27.619+0000 7f78ce878640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f78c81e02f0 0x7f78c8142410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:27.624 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:27.619+0000 7f78ce878640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f78c813a0c0 0x7f78c813a540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:27.624 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:27.619+0000 7f78ce878640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f78c80af300 con 0x7f78c81e02f0 2024-03-28T14:08:27.624 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:27.619+0000 7f78ce878640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f78c80b6340 con 0x7f78c813a0c0 2024-03-28T14:08:27.624 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:27.619+0000 7f78ce878640 1 -- --> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] -- mon_getmap magic: 0 -- 0x7f78c80af480 con 0x7f78b00051d0 2024-03-28T14:08:27.625 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:27.619+0000 7f78c77fe640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f78c81e02f0 0x7f78c8142410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:27.625 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:27.619+0000 7f78c77fe640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f78c81e02f0 0x7f78c8142410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:58882/0 (socket says 172.21.15.104:58882) 2024-03-28T14:08:27.625 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:27.619+0000 7f78c77fe640 1 -- 172.21.15.104:0/786086096 learned_addr learned my addr 172.21.15.104:0/786086096 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-03-28T14:08:27.625 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:27.619+0000 7f78cce15640 1 --2- 172.21.15.104:0/786086096 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f78c813a0c0 0x7f78c813a540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:27.625 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:27.619+0000 7f78c7fff640 1 --2- 172.21.15.104:0/786086096 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f78b00051d0 0x7f78c813cec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:27.625 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:27.619+0000 7f78c77fe640 1 -- 172.21.15.104:0/786086096 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f78b00051d0 msgr2=0x7f78c813cec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:27.625 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:27.619+0000 7f78c77fe640 1 --2- 172.21.15.104:0/786086096 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f78b00051d0 0x7f78c813cec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:27.625 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:27.619+0000 7f78c77fe640 1 -- 172.21.15.104:0/786086096 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f78c813a0c0 msgr2=0x7f78c813a540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:27.625 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:27.619+0000 7f78c77fe640 1 --2- 172.21.15.104:0/786086096 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f78c813a0c0 0x7f78c813a540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:27.625 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:27.619+0000 7f78c77fe640 1 -- 172.21.15.104:0/786086096 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f78c813d400 con 0x7f78c81e02f0 2024-03-28T14:08:27.625 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:27.619+0000 7f78c7fff640 1 --2- 172.21.15.104:0/786086096 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f78b00051d0 0x7f78c813cec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T14:08:27.625 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:27.619+0000 7f78c77fe640 1 --2- 172.21.15.104:0/786086096 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f78c81e02f0 0x7f78c8142410 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f78bc00d570 tx=0x7f78bc00da30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:08:27.625 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:27.619+0000 7f78c4ff9640 1 -- 172.21.15.104:0/786086096 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78bc005480 con 0x7f78c81e02f0 2024-03-28T14:08:27.625 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:27.619+0000 7f78c4ff9640 1 -- 172.21.15.104:0/786086096 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f78bc017070 con 0x7f78c81e02f0 2024-03-28T14:08:27.626 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:27.619+0000 7f78ce878640 1 -- 172.21.15.104:0/786086096 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f78c813d680 con 0x7f78c81e02f0 2024-03-28T14:08:27.626 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:27.619+0000 7f78c4ff9640 1 -- 172.21.15.104:0/786086096 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78bc003840 con 0x7f78c81e02f0 2024-03-28T14:08:27.626 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:27.619+0000 7f78ce878640 1 -- 172.21.15.104:0/786086096 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f78c8144250 con 0x7f78c81e02f0 2024-03-28T14:08:27.626 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:27.623+0000 7f78ce878640 1 -- 172.21.15.104:0/786086096 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f78c813c540 con 0x7f78c81e02f0 2024-03-28T14:08:27.627 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:27.623+0000 7f78c4ff9640 1 -- 172.21.15.104:0/786086096 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) ==== 87803+0+0 (secure 0 0 0) 0x7f78bc0039e0 con 0x7f78c81e02f0 2024-03-28T14:08:27.627 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:27.623+0000 7f78c4ff9640 1 --2- 172.21.15.104:0/786086096 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f78b4067fc0 0x7f78b406a480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:27.627 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:27.623+0000 7f78c4ff9640 1 -- 172.21.15.104:0/786086096 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f78bc086340 con 0x7f78c81e02f0 2024-03-28T14:08:27.627 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:27.623+0000 7f78c7fff640 1 --2- 172.21.15.104:0/786086096 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f78b4067fc0 0x7f78b406a480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:27.627 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:27.623+0000 7f78c7fff640 1 --2- 172.21.15.104:0/786086096 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f78b4067fc0 0x7f78b406a480 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f78b8009ac0 tx=0x7f78b8008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:08:27.628 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:27.627+0000 7f78c4ff9640 1 -- 172.21.15.104:0/786086096 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179117 (secure 0 0 0) 0x7f78c813d400 con 0x7f78c81e02f0 2024-03-28T14:08:27.844 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:27.843+0000 7f78ce878640 1 -- 172.21.15.104:0/786086096 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f78c80687c0 con 0x7f78c81e02f0 2024-03-28T14:08:27.845 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:27.843+0000 7f78c4ff9640 1 -- 172.21.15.104:0/786086096 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v21) ==== 93+0+10 (secure 0 0 0) 0x7f78bc019310 con 0x7f78c81e02f0 2024-03-28T14:08:27.845 INFO:teuthology.orchestra.run.smithi104.stdout:pg_num: 1 2024-03-28T14:08:27.847 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:27.843+0000 7f78ce878640 1 -- 172.21.15.104:0/786086096 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f78b4067fc0 msgr2=0x7f78b406a480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:27.847 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:27.843+0000 7f78ce878640 1 --2- 172.21.15.104:0/786086096 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f78b4067fc0 0x7f78b406a480 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f78b8009ac0 tx=0x7f78b8008040 comp rx=0 tx=0).stop 2024-03-28T14:08:27.847 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:27.843+0000 7f78ce878640 1 -- 172.21.15.104:0/786086096 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f78c81e02f0 msgr2=0x7f78c8142410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:27.847 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:27.843+0000 7f78ce878640 1 --2- 172.21.15.104:0/786086096 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f78c81e02f0 0x7f78c8142410 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f78bc00d570 tx=0x7f78bc00da30 comp rx=0 tx=0).stop 2024-03-28T14:08:27.847 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:27.843+0000 7f78ce878640 1 -- 172.21.15.104:0/786086096 shutdown_connections 2024-03-28T14:08:27.847 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:27.843+0000 7f78ce878640 1 --2- 172.21.15.104:0/786086096 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f78c813a0c0 0x7f78c813a540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:27.847 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:27.843+0000 7f78ce878640 1 --2- 172.21.15.104:0/786086096 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f78b4067fc0 0x7f78b406a480 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:27.848 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:27.843+0000 7f78ce878640 1 --2- 172.21.15.104:0/786086096 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f78c81e02f0 0x7f78c8142410 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:27.848 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:27.843+0000 7f78ce878640 1 --2- 172.21.15.104:0/786086096 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f78b00051d0 0x7f78c813cec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:27.848 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:27.843+0000 7f78ce878640 1 -- 172.21.15.104:0/786086096 >> 172.21.15.104:0/786086096 conn(0x7f78c80ba390 msgr2=0x7f78c8071440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T14:08:27.848 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:27.843+0000 7f78ce878640 1 -- 172.21.15.104:0/786086096 shutdown_connections 2024-03-28T14:08:27.848 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:27.843+0000 7f78ce878640 1 -- 172.21.15.104:0/786086096 wait complete. 2024-03-28T14:08:27.862 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-28T14:08:27.972 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:27.971+0000 7fb2d9ea5640 1 Processor -- start 2024-03-28T14:08:27.972 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:27.971+0000 7fb2d9ea5640 1 -- start start 2024-03-28T14:08:27.974 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:27.971+0000 7fb2d9ea5640 1 --2- >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fb2d40af9b0 0x7fb2d40b0010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:27.974 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:27.971+0000 7fb2d9ea5640 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fb2d41445e0 0x7fb2d4144a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:27.974 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:27.971+0000 7fb2d9ea5640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb2d4144f70 con 0x7fb2d40b63a0 2024-03-28T14:08:27.975 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:27.971+0000 7fb2d9ea5640 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 -- 0x7fb2d40af610 con 0x7fb2d41445e0 2024-03-28T14:08:27.975 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:27.971+0000 7fb2d9ea5640 1 -- --> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] -- mon_getmap magic: 0 -- 0x7fb2d40b6780 con 0x7fb2d40af9b0 2024-03-28T14:08:27.975 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:27.971+0000 7fb2d3fff640 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fb2d41445e0 0x7fb2d4144a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:27.975 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:27.971+0000 7fb2d37fe640 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7fb2d40b63a0 0x7fb2d4141670 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:41584/0 (socket says 172.21.15.104:41584) 2024-03-28T14:08:27.975 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:27.971+0000 7fb2d3fff640 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fb2d41445e0 0x7fb2d4144a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:58890/0 (socket says 172.21.15.104:58890) 2024-03-28T14:08:27.975 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:27.971+0000 7fb2d37fe640 1 -- 172.21.15.104:0/537404074 learned_addr learned my addr 172.21.15.104:0/537404074 (peer_addr_for_me v1:172.21.15.104:0/0) 2024-03-28T14:08:27.975 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:27.971+0000 7fb2cbfff640 1 --2- 172.21.15.104:0/537404074 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fb2d40af9b0 0x7fb2d40b0010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:27.975 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:27.971+0000 7fb2d3fff640 1 -- 172.21.15.104:0/537404074 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fb2d40af9b0 msgr2=0x7fb2d40b0010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T14:08:27.975 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:27.971+0000 7fb2d3fff640 1 --2- 172.21.15.104:0/537404074 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fb2d40af9b0 0x7fb2d40b0010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:27.975 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:27.971+0000 7fb2d3fff640 1 -- 172.21.15.104:0/537404074 >> v1:172.21.15.104:6789/0 conn(0x7fb2d40b63a0 legacy=0x7fb2d4141670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:27.975 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:27.971+0000 7fb2d2ffd640 1 -- 172.21.15.104:0/537404074 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1118015105 0 0) 0x7fb2d4144f70 con 0x7fb2d40b63a0 2024-03-28T14:08:27.975 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:27.971+0000 7fb2d3fff640 1 -- 172.21.15.104:0/537404074 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb2d406dde0 con 0x7fb2d41445e0 2024-03-28T14:08:27.977 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:27.975+0000 7fb2d3fff640 1 --2- 172.21.15.104:0/537404074 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb2d41445e0 0x7fb2d4144a30 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7fb2cc009d90 tx=0x7fb2cc01c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=91d34c47471f044b server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:08:27.977 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:27.975+0000 7fb2d2ffd640 1 -- 172.21.15.104:0/537404074 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2cc0b2070 con 0x7fb2d41445e0 2024-03-28T14:08:27.977 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:27.975+0000 7fb2d2ffd640 1 -- 172.21.15.104:0/537404074 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb2cc0a57a0 con 0x7fb2d41445e0 2024-03-28T14:08:27.977 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:27.975+0000 7fb2d2ffd640 1 -- 172.21.15.104:0/537404074 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2cc0a5b70 con 0x7fb2d41445e0 2024-03-28T14:08:27.979 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:27.975+0000 7fb2d9ea5640 1 -- 172.21.15.104:0/537404074 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb2d41445e0 msgr2=0x7fb2d4144a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:27.979 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:27.975+0000 7fb2d9ea5640 1 --2- 172.21.15.104:0/537404074 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb2d41445e0 0x7fb2d4144a30 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7fb2cc009d90 tx=0x7fb2cc01c8e0 comp rx=0 tx=0).stop 2024-03-28T14:08:27.979 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:27.975+0000 7fb2d9ea5640 1 -- 172.21.15.104:0/537404074 shutdown_connections 2024-03-28T14:08:27.979 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:27.975+0000 7fb2d9ea5640 1 --2- 172.21.15.104:0/537404074 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb2d41445e0 0x7fb2d4144a30 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:27.979 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:27.975+0000 7fb2d9ea5640 1 --2- 172.21.15.104:0/537404074 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fb2d40af9b0 0x7fb2d40b0010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:27.979 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:27.975+0000 7fb2d9ea5640 1 -- 172.21.15.104:0/537404074 >> 172.21.15.104:0/537404074 conn(0x7fb2d40ba400 msgr2=0x7fb2d40ba810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T14:08:27.979 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:27.975+0000 7fb2d9ea5640 1 -- 172.21.15.104:0/537404074 shutdown_connections 2024-03-28T14:08:27.979 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:27.975+0000 7fb2d9ea5640 1 -- 172.21.15.104:0/537404074 wait complete. 2024-03-28T14:08:27.982 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:27.979+0000 7fb2d9ea5640 1 Processor -- start 2024-03-28T14:08:27.982 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:27.979+0000 7fb2d9ea5640 1 -- start start 2024-03-28T14:08:27.982 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:27.979+0000 7fb2d9ea5640 1 --2- >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fb2d40af9b0 0x7fb2d4072050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:27.982 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:27.979+0000 7fb2d9ea5640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fb2d40b63a0 0x7fb2d413bd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:27.982 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:27.979+0000 7fb2d9ea5640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb2d41445e0 0x7fb2d41e1370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:27.982 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:27.979+0000 7fb2d9ea5640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7fb2d41517b0 con 0x7fb2d41445e0 2024-03-28T14:08:27.982 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:27.979+0000 7fb2d9ea5640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7fb2d4141f90 con 0x7fb2d40b63a0 2024-03-28T14:08:27.982 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:27.979+0000 7fb2d9ea5640 1 -- --> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] -- mon_getmap magic: 0 -- 0x7fb2d4151930 con 0x7fb2d40af9b0 2024-03-28T14:08:27.982 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:27.979+0000 7fb2d3fff640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb2d41445e0 0x7fb2d41e1370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:27.982 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:27.979+0000 7fb2d3fff640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb2d41445e0 0x7fb2d41e1370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:58904/0 (socket says 172.21.15.104:58904) 2024-03-28T14:08:27.982 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:27.979+0000 7fb2d3fff640 1 -- 172.21.15.104:0/3614742365 learned_addr learned my addr 172.21.15.104:0/3614742365 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-03-28T14:08:27.982 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:27.979+0000 7fb2d37fe640 1 --2- 172.21.15.104:0/3614742365 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fb2d40af9b0 0x7fb2d4072050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:27.983 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:27.979+0000 7fb2cbfff640 1 --2- 172.21.15.104:0/3614742365 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fb2d40b63a0 0x7fb2d413bd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:27.983 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:27.979+0000 7fb2d3fff640 1 -- 172.21.15.104:0/3614742365 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fb2d40af9b0 msgr2=0x7fb2d4072050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:27.983 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:27.979+0000 7fb2d3fff640 1 --2- 172.21.15.104:0/3614742365 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fb2d40af9b0 0x7fb2d4072050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:27.983 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:27.979+0000 7fb2d3fff640 1 -- 172.21.15.104:0/3614742365 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fb2d40b63a0 msgr2=0x7fb2d413bd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:27.983 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:27.979+0000 7fb2d3fff640 1 --2- 172.21.15.104:0/3614742365 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fb2d40b63a0 0x7fb2d413bd90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:27.983 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:27.979+0000 7fb2d3fff640 1 -- 172.21.15.104:0/3614742365 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb2d4072950 con 0x7fb2d41445e0 2024-03-28T14:08:27.983 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:27.979+0000 7fb2d3fff640 1 --2- 172.21.15.104:0/3614742365 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb2d41445e0 0x7fb2d41e1370 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7fb2cc0b0040 tx=0x7fb2cc019680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:08:27.983 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:27.979+0000 7fb2d0ff9640 1 -- 172.21.15.104:0/3614742365 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2cc0bd020 con 0x7fb2d41445e0 2024-03-28T14:08:27.983 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:27.979+0000 7fb2d0ff9640 1 -- 172.21.15.104:0/3614742365 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb2cc0b8470 con 0x7fb2d41445e0 2024-03-28T14:08:27.983 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:27.979+0000 7fb2d9ea5640 1 -- 172.21.15.104:0/3614742365 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb2d413b8b0 con 0x7fb2d41445e0 2024-03-28T14:08:27.983 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:27.979+0000 7fb2d0ff9640 1 -- 172.21.15.104:0/3614742365 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2cc0b2040 con 0x7fb2d41445e0 2024-03-28T14:08:27.983 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:27.979+0000 7fb2d9ea5640 1 -- 172.21.15.104:0/3614742365 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb2d413cca0 con 0x7fb2d41445e0 2024-03-28T14:08:27.985 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:27.979+0000 7fb2d9ea5640 1 -- 172.21.15.104:0/3614742365 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb2d413ba80 con 0x7fb2d41445e0 2024-03-28T14:08:27.985 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:27.979+0000 7fb2d0ff9640 1 -- 172.21.15.104:0/3614742365 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) ==== 87803+0+0 (secure 0 0 0) 0x7fb2cc0c0020 con 0x7fb2d41445e0 2024-03-28T14:08:27.985 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:27.983+0000 7fb2d0ff9640 1 --2- 172.21.15.104:0/3614742365 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7fb2b4067fc0 0x7fb2b406a480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:27.985 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:27.983+0000 7fb2d0ff9640 1 -- 172.21.15.104:0/3614742365 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7fb2cc1213b0 con 0x7fb2d41445e0 2024-03-28T14:08:27.985 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:27.983+0000 7fb2d37fe640 1 --2- 172.21.15.104:0/3614742365 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7fb2b4067fc0 0x7fb2b406a480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:27.985 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:27.983+0000 7fb2d37fe640 1 --2- 172.21.15.104:0/3614742365 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7fb2b4067fc0 0x7fb2b406a480 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fb2c4000f70 tx=0x7fb2c4009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:08:27.992 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:27.991+0000 7fb2d0ff9640 1 -- 172.21.15.104:0/3614742365 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179117 (secure 0 0 0) 0x7fb2cc0c7030 con 0x7fb2d41445e0 2024-03-28T14:08:28.246 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:28.243+0000 7fb2d9ea5640 1 -- 172.21.15.104:0/3614742365 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fb2d40687c0 con 0x7fb2d41445e0 2024-03-28T14:08:28.246 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:28.243+0000 7fb2d0ff9640 1 -- 172.21.15.104:0/3614742365 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v21) ==== 104+0+11 (secure 0 0 0) 0x7fb2cc016560 con 0x7fb2d41445e0 2024-03-28T14:08:28.247 INFO:teuthology.orchestra.run.smithi104.stdout:pg_num: 32 2024-03-28T14:08:28.249 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:28.247+0000 7fb2d9ea5640 1 -- 172.21.15.104:0/3614742365 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7fb2b4067fc0 msgr2=0x7fb2b406a480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:28.249 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:28.247+0000 7fb2d9ea5640 1 --2- 172.21.15.104:0/3614742365 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7fb2b4067fc0 0x7fb2b406a480 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fb2c4000f70 tx=0x7fb2c4009040 comp rx=0 tx=0).stop 2024-03-28T14:08:28.249 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:28.247+0000 7fb2d9ea5640 1 -- 172.21.15.104:0/3614742365 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb2d41445e0 msgr2=0x7fb2d41e1370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:28.249 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:28.247+0000 7fb2d9ea5640 1 --2- 172.21.15.104:0/3614742365 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb2d41445e0 0x7fb2d41e1370 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7fb2cc0b0040 tx=0x7fb2cc019680 comp rx=0 tx=0).stop 2024-03-28T14:08:28.249 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:28.247+0000 7fb2d9ea5640 1 -- 172.21.15.104:0/3614742365 shutdown_connections 2024-03-28T14:08:28.249 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:28.247+0000 7fb2d9ea5640 1 --2- 172.21.15.104:0/3614742365 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7fb2b4067fc0 0x7fb2b406a480 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:28.249 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:28.247+0000 7fb2d9ea5640 1 --2- 172.21.15.104:0/3614742365 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb2d41445e0 0x7fb2d41e1370 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:28.250 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:28.247+0000 7fb2d9ea5640 1 --2- 172.21.15.104:0/3614742365 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fb2d40b63a0 0x7fb2d413bd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:28.250 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:28.247+0000 7fb2d9ea5640 1 --2- 172.21.15.104:0/3614742365 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fb2d40af9b0 0x7fb2d4072050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:28.250 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:28.247+0000 7fb2d9ea5640 1 -- 172.21.15.104:0/3614742365 >> 172.21.15.104:0/3614742365 conn(0x7fb2d40ba400 msgr2=0x7fb2d4137060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T14:08:28.250 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:28.247+0000 7fb2d9ea5640 1 -- 172.21.15.104:0/3614742365 shutdown_connections 2024-03-28T14:08:28.250 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:28.247+0000 7fb2d9ea5640 1 -- 172.21.15.104:0/3614742365 wait complete. 2024-03-28T14:08:28.264 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-28T14:08:28.386 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:28.383+0000 7f8e36232640 1 Processor -- start 2024-03-28T14:08:28.387 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:28.383+0000 7f8e36232640 1 -- start start 2024-03-28T14:08:28.388 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:28.383+0000 7f8e36232640 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f8e3006b8e0 0x7f8e300b0b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:28.388 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:28.383+0000 7f8e36232640 1 --2- >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f8e3014b260 0x7f8e3014b6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:28.388 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:28.387+0000 7f8e36232640 1 --2- >> v2:172.21.15.171:3300/0 conn(0x7f8e3006d8f0 0x7f8e3006dd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:28.388 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:28.387+0000 7f8e36232640 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 -- 0x7f8e3006bcc0 con 0x7f8e3006b8e0 2024-03-28T14:08:28.388 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:28.387+0000 7f8e36232640 1 -- --> v2:172.21.15.171:3300/0 -- mon_getmap magic: 0 -- 0x7f8e300b5000 con 0x7f8e3006d8f0 2024-03-28T14:08:28.388 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:28.387+0000 7f8e36232640 1 -- --> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] -- mon_getmap magic: 0 -- 0x7f8e300b0520 con 0x7f8e3014b260 2024-03-28T14:08:28.389 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:28.387+0000 7f8e2f7fe640 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f8e3006b8e0 0x7f8e300b0b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:28.389 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:28.387+0000 7f8e2f7fe640 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f8e3006b8e0 0x7f8e300b0b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:58908/0 (socket says 172.21.15.104:58908) 2024-03-28T14:08:28.389 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:28.387+0000 7f8e2f7fe640 1 -- 172.21.15.104:0/2307306537 learned_addr learned my addr 172.21.15.104:0/2307306537 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-03-28T14:08:28.389 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:28.387+0000 7f8e2ffff640 1 --2- 172.21.15.104:0/2307306537 >> v2:172.21.15.171:3300/0 conn(0x7f8e3006d8f0 0x7f8e3006dd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:28.389 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:28.387+0000 7f8e2effd640 1 --2- 172.21.15.104:0/2307306537 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f8e3014b260 0x7f8e3014b6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:28.389 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:28.387+0000 7f8e2f7fe640 1 -- 172.21.15.104:0/2307306537 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f8e3014b260 msgr2=0x7f8e3014b6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T14:08:28.389 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:28.387+0000 7f8e2f7fe640 1 --2- 172.21.15.104:0/2307306537 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f8e3014b260 0x7f8e3014b6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:28.389 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:28.387+0000 7f8e2f7fe640 1 -- 172.21.15.104:0/2307306537 >> v2:172.21.15.171:3300/0 conn(0x7f8e3006d8f0 msgr2=0x7f8e3006dd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T14:08:28.389 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:28.387+0000 7f8e2f7fe640 1 --2- 172.21.15.104:0/2307306537 >> v2:172.21.15.171:3300/0 conn(0x7f8e3006d8f0 0x7f8e3006dd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:28.389 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:28.387+0000 7f8e2f7fe640 1 -- 172.21.15.104:0/2307306537 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8e3014bc00 con 0x7f8e3006b8e0 2024-03-28T14:08:28.389 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:28.387+0000 7f8e2effd640 1 --2- 172.21.15.104:0/2307306537 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f8e3014b260 0x7f8e3014b6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T14:08:28.389 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:28.387+0000 7f8e2ffff640 1 --2- 172.21.15.104:0/2307306537 >> v2:172.21.15.171:3300/0 conn(0x7f8e3006d8f0 0x7f8e3006dd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T14:08:28.391 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:28.387+0000 7f8e2f7fe640 1 --2- 172.21.15.104:0/2307306537 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8e3006b8e0 0x7f8e300b0b20 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f8e300af7a0 tx=0x7f8e2001c400 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4d8964c5b84987b8 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:08:28.391 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:28.387+0000 7f8e2e7fc640 1 -- 172.21.15.104:0/2307306537 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e200b0070 con 0x7f8e3006b8e0 2024-03-28T14:08:28.391 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:28.387+0000 7f8e2e7fc640 1 -- 172.21.15.104:0/2307306537 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8e200a4e50 con 0x7f8e3006b8e0 2024-03-28T14:08:28.391 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:28.387+0000 7f8e2e7fc640 1 -- 172.21.15.104:0/2307306537 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e200ab820 con 0x7f8e3006b8e0 2024-03-28T14:08:28.391 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:28.387+0000 7f8e36232640 1 -- 172.21.15.104:0/2307306537 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8e3006b8e0 msgr2=0x7f8e300b0b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:28.391 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:28.387+0000 7f8e36232640 1 --2- 172.21.15.104:0/2307306537 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8e3006b8e0 0x7f8e300b0b20 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f8e300af7a0 tx=0x7f8e2001c400 comp rx=0 tx=0).stop 2024-03-28T14:08:28.391 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:28.387+0000 7f8e36232640 1 -- 172.21.15.104:0/2307306537 shutdown_connections 2024-03-28T14:08:28.391 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:28.387+0000 7f8e36232640 1 --2- 172.21.15.104:0/2307306537 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f8e3014b260 0x7f8e3014b6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:28.391 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:28.387+0000 7f8e36232640 1 --2- 172.21.15.104:0/2307306537 >> v2:172.21.15.171:3300/0 conn(0x7f8e3006d8f0 0x7f8e3006dd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:28.391 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:28.387+0000 7f8e36232640 1 --2- 172.21.15.104:0/2307306537 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8e3006b8e0 0x7f8e300b0b20 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:28.392 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:28.387+0000 7f8e36232640 1 -- 172.21.15.104:0/2307306537 >> 172.21.15.104:0/2307306537 conn(0x7f8e300ba200 msgr2=0x7f8e300ba610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T14:08:28.392 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:28.387+0000 7f8e36232640 1 -- 172.21.15.104:0/2307306537 shutdown_connections 2024-03-28T14:08:28.392 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:28.387+0000 7f8e36232640 1 -- 172.21.15.104:0/2307306537 wait complete. 2024-03-28T14:08:28.394 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:28.391+0000 7f8e36232640 1 Processor -- start 2024-03-28T14:08:28.394 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:28.391+0000 7f8e36232640 1 -- start start 2024-03-28T14:08:28.394 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:28.391+0000 7f8e36232640 1 --2- >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f8e3006b8e0 0x7f8e3009d130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:28.395 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:28.391+0000 7f8e36232640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f8e3006d8f0 0x7f8e301e0c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:28.395 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:28.391+0000 7f8e36232640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8e3014b260 0x7f8e301e11c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:28.395 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:28.391+0000 7f8e36232640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f8e300af050 con 0x7f8e3014b260 2024-03-28T14:08:28.395 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:28.391+0000 7f8e36232640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f8e300af3f0 con 0x7f8e3006d8f0 2024-03-28T14:08:28.395 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:28.391+0000 7f8e36232640 1 -- --> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] -- mon_getmap magic: 0 -- 0x7f8e300af1d0 con 0x7f8e3006b8e0 2024-03-28T14:08:28.395 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:28.391+0000 7f8e2ffff640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8e3014b260 0x7f8e301e11c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:28.395 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:28.391+0000 7f8e2ffff640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8e3014b260 0x7f8e301e11c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:58924/0 (socket says 172.21.15.104:58924) 2024-03-28T14:08:28.395 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:28.391+0000 7f8e2ffff640 1 -- 172.21.15.104:0/2202835872 learned_addr learned my addr 172.21.15.104:0/2202835872 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-03-28T14:08:28.395 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:28.391+0000 7f8e2effd640 1 --2- 172.21.15.104:0/2202835872 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f8e3006d8f0 0x7f8e301e0c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:28.395 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:28.391+0000 7f8e2f7fe640 1 --2- 172.21.15.104:0/2202835872 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f8e3006b8e0 0x7f8e3009d130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:28.395 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:28.391+0000 7f8e2ffff640 1 -- 172.21.15.104:0/2202835872 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f8e3006b8e0 msgr2=0x7f8e3009d130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:28.395 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:28.391+0000 7f8e2ffff640 1 --2- 172.21.15.104:0/2202835872 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f8e3006b8e0 0x7f8e3009d130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:28.395 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:28.391+0000 7f8e2ffff640 1 -- 172.21.15.104:0/2202835872 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f8e3006d8f0 msgr2=0x7f8e301e0c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:28.395 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:28.391+0000 7f8e2ffff640 1 --2- 172.21.15.104:0/2202835872 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f8e3006d8f0 0x7f8e301e0c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:28.396 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:28.391+0000 7f8e2ffff640 1 -- 172.21.15.104:0/2202835872 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8e3009fa00 con 0x7f8e3014b260 2024-03-28T14:08:28.396 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:28.391+0000 7f8e2effd640 1 --2- 172.21.15.104:0/2202835872 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f8e3006d8f0 0x7f8e301e0c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T14:08:28.396 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:28.391+0000 7f8e2ffff640 1 --2- 172.21.15.104:0/2202835872 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8e3014b260 0x7f8e301e11c0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f8e2800cd10 tx=0x7f8e2800a650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:08:28.396 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:28.391+0000 7f8e17fff640 1 -- 172.21.15.104:0/2202835872 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e2800f680 con 0x7f8e3014b260 2024-03-28T14:08:28.397 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:28.391+0000 7f8e17fff640 1 -- 172.21.15.104:0/2202835872 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8e2800f840 con 0x7f8e3014b260 2024-03-28T14:08:28.397 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:28.391+0000 7f8e36232640 1 -- 172.21.15.104:0/2202835872 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8e3009e200 con 0x7f8e3014b260 2024-03-28T14:08:28.397 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:28.391+0000 7f8e17fff640 1 -- 172.21.15.104:0/2202835872 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e2800fc30 con 0x7f8e3014b260 2024-03-28T14:08:28.397 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:28.391+0000 7f8e36232640 1 -- 172.21.15.104:0/2202835872 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8e300b2cf0 con 0x7f8e3014b260 2024-03-28T14:08:28.397 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:28.391+0000 7f8e36232640 1 -- 172.21.15.104:0/2202835872 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8e300687c0 con 0x7f8e3014b260 2024-03-28T14:08:28.398 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:28.395+0000 7f8e17fff640 1 -- 172.21.15.104:0/2202835872 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) ==== 87803+0+0 (secure 0 0 0) 0x7f8e280213f0 con 0x7f8e3014b260 2024-03-28T14:08:28.398 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:28.395+0000 7f8e17fff640 1 --2- 172.21.15.104:0/2202835872 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f8e1c067fc0 0x7f8e1c06a480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:28.398 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:28.395+0000 7f8e17fff640 1 -- 172.21.15.104:0/2202835872 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f8e28087530 con 0x7f8e3014b260 2024-03-28T14:08:28.398 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:28.395+0000 7f8e2f7fe640 1 --2- 172.21.15.104:0/2202835872 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f8e1c067fc0 0x7f8e1c06a480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:28.398 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:28.395+0000 7f8e2f7fe640 1 --2- 172.21.15.104:0/2202835872 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f8e1c067fc0 0x7f8e1c06a480 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f8e200ad000 tx=0x7f8e20008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:08:28.399 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:28.399+0000 7f8e17fff640 1 -- 172.21.15.104:0/2202835872 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179117 (secure 0 0 0) 0x7f8e28014030 con 0x7f8e3014b260 2024-03-28T14:08:28.549 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:28.547+0000 7f8e36232640 1 -- 172.21.15.104:0/2202835872 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f8e3014b9e0 con 0x7f8e3014b260 2024-03-28T14:08:28.550 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:28.547+0000 7f8e17fff640 1 -- 172.21.15.104:0/2202835872 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v21) ==== 100+0+11 (secure 0 0 0) 0x7f8e28057d50 con 0x7f8e3014b260 2024-03-28T14:08:28.550 INFO:teuthology.orchestra.run.smithi104.stdout:pg_num: 64 2024-03-28T14:08:28.552 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:28.551+0000 7f8e36232640 1 -- 172.21.15.104:0/2202835872 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f8e1c067fc0 msgr2=0x7f8e1c06a480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:28.552 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:28.551+0000 7f8e36232640 1 --2- 172.21.15.104:0/2202835872 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f8e1c067fc0 0x7f8e1c06a480 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f8e200ad000 tx=0x7f8e20008040 comp rx=0 tx=0).stop 2024-03-28T14:08:28.552 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:28.551+0000 7f8e36232640 1 -- 172.21.15.104:0/2202835872 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8e3014b260 msgr2=0x7f8e301e11c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:28.552 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:28.551+0000 7f8e36232640 1 --2- 172.21.15.104:0/2202835872 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8e3014b260 0x7f8e301e11c0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f8e2800cd10 tx=0x7f8e2800a650 comp rx=0 tx=0).stop 2024-03-28T14:08:28.553 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:28.551+0000 7f8e36232640 1 -- 172.21.15.104:0/2202835872 shutdown_connections 2024-03-28T14:08:28.553 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:28.551+0000 7f8e36232640 1 --2- 172.21.15.104:0/2202835872 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f8e1c067fc0 0x7f8e1c06a480 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:28.553 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:28.551+0000 7f8e36232640 1 --2- 172.21.15.104:0/2202835872 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8e3014b260 0x7f8e301e11c0 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:28.553 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:28.551+0000 7f8e36232640 1 --2- 172.21.15.104:0/2202835872 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f8e3006d8f0 0x7f8e301e0c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:28.553 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:28.551+0000 7f8e36232640 1 --2- 172.21.15.104:0/2202835872 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f8e3006b8e0 0x7f8e3009d130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:28.553 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:28.551+0000 7f8e36232640 1 -- 172.21.15.104:0/2202835872 >> 172.21.15.104:0/2202835872 conn(0x7f8e300ba200 msgr2=0x7f8e30147fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T14:08:28.553 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:28.551+0000 7f8e36232640 1 -- 172.21.15.104:0/2202835872 shutdown_connections 2024-03-28T14:08:28.553 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:28.551+0000 7f8e36232640 1 -- 172.21.15.104:0/2202835872 wait complete. 2024-03-28T14:08:28.567 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T14:08:28.691 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:28.687+0000 7f3ca095a640 1 Processor -- start 2024-03-28T14:08:28.691 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:28.687+0000 7f3ca095a640 1 -- start start 2024-03-28T14:08:28.693 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:28.691+0000 7f3ca095a640 1 --2- >> v2:172.21.15.171:3300/0 conn(0x7f3c9c02c460 0x7f3c9c0b54e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:28.693 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:28.691+0000 7f3ca095a640 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f3c9c09e8e0 0x7f3c9c0b5a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:28.693 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:28.691+0000 7f3ca095a640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3c9c02c840 con 0x7f3c9c1480d0 2024-03-28T14:08:28.693 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:28.691+0000 7f3ca095a640 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 -- 0x7f3c9c147c40 con 0x7f3c9c09e8e0 2024-03-28T14:08:28.693 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:28.691+0000 7f3ca095a640 1 -- --> v2:172.21.15.171:3300/0 -- mon_getmap magic: 0 -- 0x7f3c9c0aec70 con 0x7f3c9c02c460 2024-03-28T14:08:28.693 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:28.691+0000 7f3c99d9b640 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f3c9c09e8e0 0x7f3c9c0b5a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:28.693 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:28.691+0000 7f3c9ad9d640 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f3c9c1480d0 0x7f3c9c148560 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:41588/0 (socket says 172.21.15.104:41588) 2024-03-28T14:08:28.693 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:28.691+0000 7f3c9ad9d640 1 -- 172.21.15.104:0/4093534020 learned_addr learned my addr 172.21.15.104:0/4093534020 (peer_addr_for_me v1:172.21.15.104:0/0) 2024-03-28T14:08:28.693 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:28.691+0000 7f3c9a59c640 1 --2- 172.21.15.104:0/4093534020 >> v2:172.21.15.171:3300/0 conn(0x7f3c9c02c460 0x7f3c9c0b54e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:28.693 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:28.691+0000 7f3c99d9b640 1 -- 172.21.15.104:0/4093534020 >> v2:172.21.15.171:3300/0 conn(0x7f3c9c02c460 msgr2=0x7f3c9c0b54e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T14:08:28.693 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:28.691+0000 7f3c99d9b640 1 --2- 172.21.15.104:0/4093534020 >> v2:172.21.15.171:3300/0 conn(0x7f3c9c02c460 0x7f3c9c0b54e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:28.694 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:28.691+0000 7f3c99d9b640 1 -- 172.21.15.104:0/4093534020 >> v1:172.21.15.104:6789/0 conn(0x7f3c9c1480d0 legacy=0x7f3c9c148560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:28.694 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:28.691+0000 7f3c99d9b640 1 -- 172.21.15.104:0/4093534020 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3c9c0aef50 con 0x7f3c9c09e8e0 2024-03-28T14:08:28.694 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:28.691+0000 7f3c9959a640 1 -- 172.21.15.104:0/4093534020 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2828635847 0 0) 0x7f3c9c02c840 con 0x7f3c9c1480d0 2024-03-28T14:08:28.696 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:28.695+0000 7f3c99d9b640 1 --2- 172.21.15.104:0/4093534020 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3c9c09e8e0 0x7f3c9c0b5a20 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f3c9c02d2f0 tx=0x7f3c9001c480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a17f91d2f696db server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:08:28.696 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:28.695+0000 7f3c9959a640 1 -- 172.21.15.104:0/4093534020 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c900b0070 con 0x7f3c9c09e8e0 2024-03-28T14:08:28.696 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:28.695+0000 7f3c9959a640 1 -- 172.21.15.104:0/4093534020 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3c900ab600 con 0x7f3c9c09e8e0 2024-03-28T14:08:28.696 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:28.695+0000 7f3c9959a640 1 -- 172.21.15.104:0/4093534020 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c900ab9d0 con 0x7f3c9c09e8e0 2024-03-28T14:08:28.696 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:28.695+0000 7f3ca095a640 1 -- 172.21.15.104:0/4093534020 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3c9c09e8e0 msgr2=0x7f3c9c0b5a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:28.697 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:28.695+0000 7f3ca095a640 1 --2- 172.21.15.104:0/4093534020 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3c9c09e8e0 0x7f3c9c0b5a20 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f3c9c02d2f0 tx=0x7f3c9001c480 comp rx=0 tx=0).stop 2024-03-28T14:08:28.697 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:28.695+0000 7f3ca095a640 1 -- 172.21.15.104:0/4093534020 shutdown_connections 2024-03-28T14:08:28.697 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:28.695+0000 7f3ca095a640 1 --2- 172.21.15.104:0/4093534020 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3c9c09e8e0 0x7f3c9c0b5a20 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:28.697 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:28.695+0000 7f3ca095a640 1 --2- 172.21.15.104:0/4093534020 >> v2:172.21.15.171:3300/0 conn(0x7f3c9c02c460 0x7f3c9c0b54e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:28.697 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:28.695+0000 7f3ca095a640 1 -- 172.21.15.104:0/4093534020 >> 172.21.15.104:0/4093534020 conn(0x7f3c9c0b95f0 msgr2=0x7f3c9c0b99d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T14:08:28.697 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:28.695+0000 7f3ca095a640 1 -- 172.21.15.104:0/4093534020 shutdown_connections 2024-03-28T14:08:28.697 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:28.695+0000 7f3ca095a640 1 -- 172.21.15.104:0/4093534020 wait complete. 2024-03-28T14:08:28.700 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:28.695+0000 7f3ca095a640 1 Processor -- start 2024-03-28T14:08:28.700 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:28.695+0000 7f3ca095a640 1 -- start start 2024-03-28T14:08:28.700 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:28.695+0000 7f3ca095a640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3c9c02c460 0x7f3c9c146c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:28.700 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:28.695+0000 7f3ca095a640 1 --2- >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f3c9c09e8e0 0x7f3c9c1e3d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:28.700 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:28.695+0000 7f3ca095a640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f3c9c1480d0 0x7f3c9c140000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:28.700 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:28.695+0000 7f3ca095a640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f3c9c06d390 con 0x7f3c9c02c460 2024-03-28T14:08:28.700 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:28.695+0000 7f3ca095a640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f3c9c06d210 con 0x7f3c9c1480d0 2024-03-28T14:08:28.700 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:28.695+0000 7f3ca095a640 1 -- --> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] -- mon_getmap magic: 0 -- 0x7f3c9c06d510 con 0x7f3c9c09e8e0 2024-03-28T14:08:28.700 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:28.695+0000 7f3c9a59c640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3c9c02c460 0x7f3c9c146c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:28.700 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:28.695+0000 7f3c9a59c640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3c9c02c460 0x7f3c9c146c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:58934/0 (socket says 172.21.15.104:58934) 2024-03-28T14:08:28.701 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:28.695+0000 7f3c9a59c640 1 -- 172.21.15.104:0/1261043089 learned_addr learned my addr 172.21.15.104:0/1261043089 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-03-28T14:08:28.701 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:28.695+0000 7f3c99d9b640 1 --2- 172.21.15.104:0/1261043089 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f3c9c09e8e0 0x7f3c9c1e3d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:28.701 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:28.695+0000 7f3c9ad9d640 1 --2- 172.21.15.104:0/1261043089 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f3c9c1480d0 0x7f3c9c140000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:28.701 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:28.695+0000 7f3c9a59c640 1 -- 172.21.15.104:0/1261043089 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f3c9c09e8e0 msgr2=0x7f3c9c1e3d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:28.701 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:28.695+0000 7f3c9a59c640 1 --2- 172.21.15.104:0/1261043089 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f3c9c09e8e0 0x7f3c9c1e3d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:28.701 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:28.695+0000 7f3c9a59c640 1 -- 172.21.15.104:0/1261043089 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f3c9c1480d0 msgr2=0x7f3c9c140000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:28.701 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:28.695+0000 7f3c9a59c640 1 --2- 172.21.15.104:0/1261043089 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f3c9c1480d0 0x7f3c9c140000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:28.701 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:28.695+0000 7f3c9a59c640 1 -- 172.21.15.104:0/1261043089 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3c9c09a590 con 0x7f3c9c02c460 2024-03-28T14:08:28.701 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:28.695+0000 7f3c9ad9d640 1 --2- 172.21.15.104:0/1261043089 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f3c9c1480d0 0x7f3c9c140000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T14:08:28.701 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:28.695+0000 7f3c9a59c640 1 --2- 172.21.15.104:0/1261043089 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3c9c02c460 0x7f3c9c146c20 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f3c8c007f20 tx=0x7f3c8c00e570 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:08:28.701 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:28.695+0000 7f3c8affd640 1 -- 172.21.15.104:0/1261043089 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c8c014620 con 0x7f3c9c02c460 2024-03-28T14:08:28.701 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:28.695+0000 7f3c8affd640 1 -- 172.21.15.104:0/1261043089 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3c8c019070 con 0x7f3c9c02c460 2024-03-28T14:08:28.701 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:28.695+0000 7f3ca095a640 1 -- 172.21.15.104:0/1261043089 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3c9c09a830 con 0x7f3c9c02c460 2024-03-28T14:08:28.702 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:28.695+0000 7f3c8affd640 1 -- 172.21.15.104:0/1261043089 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c8c014a00 con 0x7f3c9c02c460 2024-03-28T14:08:28.702 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:28.695+0000 7f3ca095a640 1 -- 172.21.15.104:0/1261043089 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3c9c09bb00 con 0x7f3c9c02c460 2024-03-28T14:08:28.702 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:28.699+0000 7f3ca095a640 1 -- 172.21.15.104:0/1261043089 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3c9c0687c0 con 0x7f3c9c02c460 2024-03-28T14:08:28.702 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:28.699+0000 7f3c8affd640 1 -- 172.21.15.104:0/1261043089 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) ==== 87803+0+0 (secure 0 0 0) 0x7f3c8c014ba0 con 0x7f3c9c02c460 2024-03-28T14:08:28.703 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:28.699+0000 7f3c8affd640 1 --2- 172.21.15.104:0/1261043089 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f3c7c067ff0 0x7f3c7c06a4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:28.703 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:28.699+0000 7f3c8affd640 1 -- 172.21.15.104:0/1261043089 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f3c8c086e30 con 0x7f3c9c02c460 2024-03-28T14:08:28.703 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:28.699+0000 7f3c99d9b640 1 --2- 172.21.15.104:0/1261043089 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f3c7c067ff0 0x7f3c7c06a4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:28.703 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:28.699+0000 7f3c99d9b640 1 --2- 172.21.15.104:0/1261043089 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f3c7c067ff0 0x7f3c7c06a4b0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f3c900ad000 tx=0x7f3c900a6000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:08:28.705 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:28.703+0000 7f3c8affd640 1 -- 172.21.15.104:0/1261043089 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179117 (secure 0 0 0) 0x7f3c8c0575e0 con 0x7f3c9c02c460 2024-03-28T14:08:28.893 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:28.891+0000 7f3ca095a640 1 -- 172.21.15.104:0/1261043089 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f3c9c14eb70 con 0x7f3c9c02c460 2024-03-28T14:08:28.894 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:28.891+0000 7f3c8affd640 1 -- 172.21.15.104:0/1261043089 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 7 v7) ==== 93+0+3533 (secure 0 0 0) 0x7f3c8c056ee0 con 0x7f3c9c02c460 2024-03-28T14:08:28.894 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-03-28T14:08:28.894 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":7,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4229,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.171:6835/1775867467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6834","nonce":1775867467},{"type":"v1","addr":"172.21.15.171:6835","nonce":1775867467}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":7,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T14:07:59.027458+0000","modified":"2024-03-28T14:08:02.057923+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4245},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4245":{"gid":4245,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.104:6833/3184673325","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":3184673325},{"type":"v1","addr":"172.21.15.104:6833","nonce":3184673325}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4245,"qdb_cluster":[4245]},"id":1}]} 2024-03-28T14:08:28.895 INFO:teuthology.orchestra.run.smithi104.stderr:dumped fsmap epoch 7 2024-03-28T14:08:28.896 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:28.895+0000 7f3ca095a640 1 -- 172.21.15.104:0/1261043089 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f3c7c067ff0 msgr2=0x7f3c7c06a4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:28.896 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:28.895+0000 7f3ca095a640 1 --2- 172.21.15.104:0/1261043089 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f3c7c067ff0 0x7f3c7c06a4b0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f3c900ad000 tx=0x7f3c900a6000 comp rx=0 tx=0).stop 2024-03-28T14:08:28.896 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:28.895+0000 7f3ca095a640 1 -- 172.21.15.104:0/1261043089 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3c9c02c460 msgr2=0x7f3c9c146c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:28.896 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:28.895+0000 7f3ca095a640 1 --2- 172.21.15.104:0/1261043089 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3c9c02c460 0x7f3c9c146c20 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f3c8c007f20 tx=0x7f3c8c00e570 comp rx=0 tx=0).stop 2024-03-28T14:08:28.896 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:28.895+0000 7f3ca095a640 1 -- 172.21.15.104:0/1261043089 shutdown_connections 2024-03-28T14:08:28.897 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:28.895+0000 7f3ca095a640 1 --2- 172.21.15.104:0/1261043089 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f3c9c1480d0 0x7f3c9c140000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:28.897 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:28.895+0000 7f3ca095a640 1 --2- 172.21.15.104:0/1261043089 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f3c9c09e8e0 0x7f3c9c1e3d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:28.897 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:28.895+0000 7f3ca095a640 1 --2- 172.21.15.104:0/1261043089 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f3c7c067ff0 0x7f3c7c06a4b0 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:28.897 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:28.895+0000 7f3ca095a640 1 --2- 172.21.15.104:0/1261043089 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3c9c02c460 0x7f3c9c146c20 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:28.897 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:28.895+0000 7f3ca095a640 1 -- 172.21.15.104:0/1261043089 >> 172.21.15.104:0/1261043089 conn(0x7f3c9c0b95f0 msgr2=0x7f3c9c0b99d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T14:08:28.897 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:28.895+0000 7f3ca095a640 1 -- 172.21.15.104:0/1261043089 shutdown_connections 2024-03-28T14:08:28.897 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:28.895+0000 7f3ca095a640 1 -- 172.21.15.104:0/1261043089 wait complete. 2024-03-28T14:08:28.907 DEBUG:teuthology.orchestra.run.smithi104:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-28T14:08:28.910 DEBUG:teuthology.orchestra.run.smithi171:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-28T14:08:28.912 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-28T14:08:29.094 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:29.091+0000 7fa977d99640 1 Processor -- start 2024-03-28T14:08:29.094 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:29.091+0000 7fa977d99640 1 -- start start 2024-03-28T14:08:29.096 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:29.091+0000 7fa977d99640 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fa97006eb50 0x7fa970148ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:29.096 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:29.091+0000 7fa977d99640 1 --2- >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fa97006ef30 0x7fa97013f440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:29.096 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:29.091+0000 7fa977d99640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa97006fb40 con 0x7fa9700b5ab0 2024-03-28T14:08:29.096 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:29.091+0000 7fa977d99640 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 -- 0x7fa9700b5f90 con 0x7fa97006eb50 2024-03-28T14:08:29.096 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:29.091+0000 7fa977d99640 1 -- --> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] -- mon_getmap magic: 0 -- 0x7fa9700b61e0 con 0x7fa97006ef30 2024-03-28T14:08:29.096 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:29.091+0000 7fa975b35640 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fa97006eb50 0x7fa970148ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:29.096 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:29.095+0000 7fa976336640 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7fa9700b5ab0 0x7fa97013f980 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:41594/0 (socket says 172.21.15.104:41594) 2024-03-28T14:08:29.097 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:29.095+0000 7fa976336640 1 -- 172.21.15.104:0/1007227718 learned_addr learned my addr 172.21.15.104:0/1007227718 (peer_addr_for_me v1:172.21.15.104:0/0) 2024-03-28T14:08:29.097 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:29.095+0000 7fa975334640 1 --2- 172.21.15.104:0/1007227718 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fa97006ef30 0x7fa97013f440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:29.097 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:29.095+0000 7fa975b35640 1 -- 172.21.15.104:0/1007227718 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fa97006ef30 msgr2=0x7fa97013f440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T14:08:29.097 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:29.095+0000 7fa975b35640 1 --2- 172.21.15.104:0/1007227718 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fa97006ef30 0x7fa97013f440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:29.097 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:29.095+0000 7fa975b35640 1 -- 172.21.15.104:0/1007227718 >> v1:172.21.15.104:6789/0 conn(0x7fa9700b5ab0 legacy=0x7fa97013f980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:29.097 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:29.095+0000 7fa975b35640 1 -- 172.21.15.104:0/1007227718 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa970149220 con 0x7fa97006eb50 2024-03-28T14:08:29.097 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:29.095+0000 7fa974b33640 1 -- 172.21.15.104:0/1007227718 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3213057313 0 0) 0x7fa97006fb40 con 0x7fa9700b5ab0 2024-03-28T14:08:29.099 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:29.095+0000 7fa975b35640 1 --2- 172.21.15.104:0/1007227718 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa97006eb50 0x7fa970148ce0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7fa9600097c0 tx=0x7fa96001c380 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ae51f30c5312671d server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:08:29.099 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:29.095+0000 7fa974b33640 1 -- 172.21.15.104:0/1007227718 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9600b0070 con 0x7fa97006eb50 2024-03-28T14:08:29.099 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:29.095+0000 7fa974b33640 1 -- 172.21.15.104:0/1007227718 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa9600ab600 con 0x7fa97006eb50 2024-03-28T14:08:29.099 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:29.095+0000 7fa974b33640 1 -- 172.21.15.104:0/1007227718 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9600ab9d0 con 0x7fa97006eb50 2024-03-28T14:08:29.099 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:29.095+0000 7fa977d99640 1 -- 172.21.15.104:0/1007227718 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa97006eb50 msgr2=0x7fa970148ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:29.099 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:29.095+0000 7fa977d99640 1 --2- 172.21.15.104:0/1007227718 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa97006eb50 0x7fa970148ce0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7fa9600097c0 tx=0x7fa96001c380 comp rx=0 tx=0).stop 2024-03-28T14:08:29.099 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:29.095+0000 7fa977d99640 1 -- 172.21.15.104:0/1007227718 shutdown_connections 2024-03-28T14:08:29.099 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:29.095+0000 7fa977d99640 1 --2- 172.21.15.104:0/1007227718 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fa97006ef30 0x7fa97013f440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:29.099 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:29.095+0000 7fa977d99640 1 --2- 172.21.15.104:0/1007227718 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa97006eb50 0x7fa970148ce0 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:29.100 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:29.095+0000 7fa977d99640 1 -- 172.21.15.104:0/1007227718 >> 172.21.15.104:0/1007227718 conn(0x7fa97002cab0 msgr2=0x7fa9700bb0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T14:08:29.100 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:29.095+0000 7fa977d99640 1 -- 172.21.15.104:0/1007227718 shutdown_connections 2024-03-28T14:08:29.100 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:29.095+0000 7fa977d99640 1 -- 172.21.15.104:0/1007227718 wait complete. 2024-03-28T14:08:29.100 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:29.099+0000 7fa977d99640 1 Processor -- start 2024-03-28T14:08:29.100 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:29.099+0000 7fa977d99640 1 -- start start 2024-03-28T14:08:29.103 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:29.099+0000 7fa977d99640 1 --2- >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fa97006eb50 0x7fa97009f270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:29.103 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:29.099+0000 7fa977d99640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fa97006ef30 0x7fa9701e1430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:29.103 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:29.099+0000 7fa977d99640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa9700b5ab0 0x7fa9701e1970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:29.103 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:29.099+0000 7fa977d99640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7fa970152bc0 con 0x7fa9700b5ab0 2024-03-28T14:08:29.103 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:29.099+0000 7fa977d99640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7fa970152a40 con 0x7fa97006ef30 2024-03-28T14:08:29.104 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:29.099+0000 7fa977d99640 1 -- --> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] -- mon_getmap magic: 0 -- 0x7fa970152d40 con 0x7fa97006eb50 2024-03-28T14:08:29.104 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:29.099+0000 7fa976336640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa9700b5ab0 0x7fa9701e1970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:29.104 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:29.099+0000 7fa976336640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa9700b5ab0 0x7fa9701e1970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:58942/0 (socket says 172.21.15.104:58942) 2024-03-28T14:08:29.104 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:29.099+0000 7fa976336640 1 -- 172.21.15.104:0/3350694860 learned_addr learned my addr 172.21.15.104:0/3350694860 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-03-28T14:08:29.104 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:29.099+0000 7fa975b35640 1 --2- 172.21.15.104:0/3350694860 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fa97006eb50 0x7fa97009f270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:29.104 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:29.099+0000 7fa975334640 1 --2- 172.21.15.104:0/3350694860 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fa97006ef30 0x7fa9701e1430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:29.104 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:29.099+0000 7fa976336640 1 -- 172.21.15.104:0/3350694860 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fa97006eb50 msgr2=0x7fa97009f270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:29.104 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:29.099+0000 7fa976336640 1 --2- 172.21.15.104:0/3350694860 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fa97006eb50 0x7fa97009f270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:29.104 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:29.099+0000 7fa976336640 1 -- 172.21.15.104:0/3350694860 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fa97006ef30 msgr2=0x7fa9701e1430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:29.104 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:29.099+0000 7fa976336640 1 --2- 172.21.15.104:0/3350694860 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fa97006ef30 0x7fa9701e1430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:29.104 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:29.099+0000 7fa976336640 1 -- 172.21.15.104:0/3350694860 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa97009aab0 con 0x7fa9700b5ab0 2024-03-28T14:08:29.104 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:29.099+0000 7fa975b35640 1 --2- 172.21.15.104:0/3350694860 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fa97006eb50 0x7fa97009f270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T14:08:29.104 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:29.099+0000 7fa976336640 1 --2- 172.21.15.104:0/3350694860 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa9700b5ab0 0x7fa9701e1970 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7fa968001570 tx=0x7fa96800cca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:08:29.105 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:29.099+0000 7fa9667fc640 1 -- 172.21.15.104:0/3350694860 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa968018070 con 0x7fa9700b5ab0 2024-03-28T14:08:29.105 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:29.099+0000 7fa9667fc640 1 -- 172.21.15.104:0/3350694860 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa96800a640 con 0x7fa9700b5ab0 2024-03-28T14:08:29.105 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:29.099+0000 7fa977d99640 1 -- 172.21.15.104:0/3350694860 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa97009b940 con 0x7fa9700b5ab0 2024-03-28T14:08:29.105 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:29.099+0000 7fa9667fc640 1 -- 172.21.15.104:0/3350694860 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa96800aa30 con 0x7fa9700b5ab0 2024-03-28T14:08:29.105 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:29.099+0000 7fa977d99640 1 -- 172.21.15.104:0/3350694860 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa97009fe00 con 0x7fa9700b5ab0 2024-03-28T14:08:29.105 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:29.103+0000 7fa977d99640 1 -- 172.21.15.104:0/3350694860 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa9700687c0 con 0x7fa9700b5ab0 2024-03-28T14:08:29.106 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:29.103+0000 7fa9667fc640 1 -- 172.21.15.104:0/3350694860 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) ==== 87803+0+0 (secure 0 0 0) 0x7fa96801b070 con 0x7fa9700b5ab0 2024-03-28T14:08:29.106 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:29.103+0000 7fa9667fc640 1 --2- 172.21.15.104:0/3350694860 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7fa954067fc0 0x7fa95406a480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:29.106 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:29.103+0000 7fa9667fc640 1 -- 172.21.15.104:0/3350694860 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7fa968085ab0 con 0x7fa9700b5ab0 2024-03-28T14:08:29.106 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:29.103+0000 7fa975b35640 1 --2- 172.21.15.104:0/3350694860 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7fa954067fc0 0x7fa95406a480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:29.106 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:29.103+0000 7fa975b35640 1 --2- 172.21.15.104:0/3350694860 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7fa954067fc0 0x7fa95406a480 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fa9600ad000 tx=0x7fa9600a6000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:08:29.108 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:29.107+0000 7fa9667fc640 1 -- 172.21.15.104:0/3350694860 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179117 (secure 0 0 0) 0x7fa9680562f0 con 0x7fa9700b5ab0 2024-03-28T14:08:29.259 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:29.255+0000 7fa977d99640 1 -- 172.21.15.104:0/3350694860 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fa970072fa0 con 0x7fa9700b5ab0 2024-03-28T14:08:29.260 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:29.259+0000 7fa9667fc640 1 -- 172.21.15.104:0/3350694860 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) ==== 74+0+16629 (secure 0 0 0) 0x7fa968055bf0 con 0x7fa9700b5ab0 2024-03-28T14:08:29.261 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-03-28T14:08:29.261 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":21,"fsid":"428a90f2-3351-435f-8fcf-f7e66c78a13e","created":"2024-03-28T14:07:34.862458+0000","modified":"2024-03-28T14:08:00.671587+0000","last_up_change":"2024-03-28T14:07:46.028122+0000","last_in_change":"2024-03-28T14:07:37.157486+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-28T14:07:47.622988+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-28T14:07:57.147450+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-28T14:07:57.990975+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"1cb70540-4e42-402f-a446-aaec4ca83007","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6800","nonce":3745690918},{"type":"v1","addr":"172.21.15.104:6801","nonce":3745690918}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":3745690918},{"type":"v1","addr":"172.21.15.104:6803","nonce":3745690918}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":3745690918},{"type":"v1","addr":"172.21.15.104:6807","nonce":3745690918}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":3745690918},{"type":"v1","addr":"172.21.15.104:6805","nonce":3745690918}]},"public_addr":"172.21.15.104:6801/3745690918","cluster_addr":"172.21.15.104:6803/3745690918","heartbeat_back_addr":"172.21.15.104:6807/3745690918","heartbeat_front_addr":"172.21.15.104:6805/3745690918","state":["exists","up"]},{"osd":1,"uuid":"3ab7afdd-b980-418c-81f2-004e16de3dca","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":19438674},{"type":"v1","addr":"172.21.15.104:6810","nonce":19438674}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":19438674},{"type":"v1","addr":"172.21.15.104:6814","nonce":19438674}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":19438674},{"type":"v1","addr":"172.21.15.104:6823","nonce":19438674}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":19438674},{"type":"v1","addr":"172.21.15.104:6819","nonce":19438674}]},"public_addr":"172.21.15.104:6810/19438674","cluster_addr":"172.21.15.104:6814/19438674","heartbeat_back_addr":"172.21.15.104:6823/19438674","heartbeat_front_addr":"172.21.15.104:6819/19438674","state":["exists","up"]},{"osd":2,"uuid":"3e2651c8-ecae-4a9e-a053-27240a79eac2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":2352012626},{"type":"v1","addr":"172.21.15.104:6825","nonce":2352012626}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6826","nonce":2352012626},{"type":"v1","addr":"172.21.15.104:6827","nonce":2352012626}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":2352012626},{"type":"v1","addr":"172.21.15.104:6831","nonce":2352012626}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6828","nonce":2352012626},{"type":"v1","addr":"172.21.15.104:6829","nonce":2352012626}]},"public_addr":"172.21.15.104:6825/2352012626","cluster_addr":"172.21.15.104:6827/2352012626","heartbeat_back_addr":"172.21.15.104:6831/2352012626","heartbeat_front_addr":"172.21.15.104:6829/2352012626","state":["exists","up"]},{"osd":3,"uuid":"925c7e0a-38ef-47cd-b288-8bec8456fb17","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6809","nonce":3568347381},{"type":"v1","addr":"172.21.15.104:6811","nonce":3568347381}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6813","nonce":3568347381},{"type":"v1","addr":"172.21.15.104:6815","nonce":3568347381}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":3568347381},{"type":"v1","addr":"172.21.15.104:6822","nonce":3568347381}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6817","nonce":3568347381},{"type":"v1","addr":"172.21.15.104:6818","nonce":3568347381}]},"public_addr":"172.21.15.104:6811/3568347381","cluster_addr":"172.21.15.104:6815/3568347381","heartbeat_back_addr":"172.21.15.104:6822/3568347381","heartbeat_front_addr":"172.21.15.104:6818/3568347381","state":["exists","up"]},{"osd":4,"uuid":"4a8fac86-e595-4a0e-96e3-57edb3cc3f57","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6800","nonce":2107001077},{"type":"v1","addr":"172.21.15.171:6801","nonce":2107001077}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6802","nonce":2107001077},{"type":"v1","addr":"172.21.15.171:6803","nonce":2107001077}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6806","nonce":2107001077},{"type":"v1","addr":"172.21.15.171:6807","nonce":2107001077}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6804","nonce":2107001077},{"type":"v1","addr":"172.21.15.171:6805","nonce":2107001077}]},"public_addr":"172.21.15.171:6801/2107001077","cluster_addr":"172.21.15.171:6803/2107001077","heartbeat_back_addr":"172.21.15.171:6807/2107001077","heartbeat_front_addr":"172.21.15.171:6805/2107001077","state":["exists","up"]},{"osd":5,"uuid":"e48d50b0-beae-40ae-b15a-9a84dbfe88ab","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6808","nonce":2934826925},{"type":"v1","addr":"172.21.15.171:6810","nonce":2934826925}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6812","nonce":2934826925},{"type":"v1","addr":"172.21.15.171:6814","nonce":2934826925}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6819","nonce":2934826925},{"type":"v1","addr":"172.21.15.171:6821","nonce":2934826925}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6816","nonce":2934826925},{"type":"v1","addr":"172.21.15.171:6817","nonce":2934826925}]},"public_addr":"172.21.15.171:6810/2934826925","cluster_addr":"172.21.15.171:6814/2934826925","heartbeat_back_addr":"172.21.15.171:6821/2934826925","heartbeat_front_addr":"172.21.15.171:6817/2934826925","state":["exists","up"]},{"osd":6,"uuid":"5e7c3292-9936-4f35-b267-aed9d4e49323","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6824","nonce":436742919},{"type":"v1","addr":"172.21.15.171:6825","nonce":436742919}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6826","nonce":436742919},{"type":"v1","addr":"172.21.15.171:6827","nonce":436742919}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6830","nonce":436742919},{"type":"v1","addr":"172.21.15.171:6831","nonce":436742919}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6828","nonce":436742919},{"type":"v1","addr":"172.21.15.171:6829","nonce":436742919}]},"public_addr":"172.21.15.171:6825/436742919","cluster_addr":"172.21.15.171:6827/436742919","heartbeat_back_addr":"172.21.15.171:6831/436742919","heartbeat_front_addr":"172.21.15.171:6829/436742919","state":["exists","up"]},{"osd":7,"uuid":"339b8414-d7f0-4fb0-a446-efaf635cb7ff","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6809","nonce":2055631175},{"type":"v1","addr":"172.21.15.171:6811","nonce":2055631175}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6813","nonce":2055631175},{"type":"v1","addr":"172.21.15.171:6815","nonce":2055631175}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6822","nonce":2055631175},{"type":"v1","addr":"172.21.15.171:6823","nonce":2055631175}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6818","nonce":2055631175},{"type":"v1","addr":"172.21.15.171:6820","nonce":2055631175}]},"public_addr":"172.21.15.171:6811/2055631175","cluster_addr":"172.21.15.171:6815/2055631175","heartbeat_back_addr":"172.21.15.171:6823/2055631175","heartbeat_front_addr":"172.21.15.171:6820/2055631175","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T14:07:42.668881+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T14:07:42.708397+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T14:07:42.634191+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T14:07:42.733662+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T14:07:42.651541+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T14:07:42.783181+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T14:07:42.766310+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T14:07:42.670797+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-28T14:08:29.263 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:29.259+0000 7fa977d99640 1 -- 172.21.15.104:0/3350694860 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7fa954067fc0 msgr2=0x7fa95406a480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:29.263 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:29.259+0000 7fa977d99640 1 --2- 172.21.15.104:0/3350694860 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7fa954067fc0 0x7fa95406a480 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fa9600ad000 tx=0x7fa9600a6000 comp rx=0 tx=0).stop 2024-03-28T14:08:29.263 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:29.259+0000 7fa977d99640 1 -- 172.21.15.104:0/3350694860 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa9700b5ab0 msgr2=0x7fa9701e1970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:29.263 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:29.259+0000 7fa977d99640 1 --2- 172.21.15.104:0/3350694860 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa9700b5ab0 0x7fa9701e1970 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7fa968001570 tx=0x7fa96800cca0 comp rx=0 tx=0).stop 2024-03-28T14:08:29.263 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:29.259+0000 7fa977d99640 1 -- 172.21.15.104:0/3350694860 shutdown_connections 2024-03-28T14:08:29.263 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:29.259+0000 7fa977d99640 1 --2- 172.21.15.104:0/3350694860 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7fa954067fc0 0x7fa95406a480 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:29.263 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:29.259+0000 7fa977d99640 1 --2- 172.21.15.104:0/3350694860 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa9700b5ab0 0x7fa9701e1970 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:29.263 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:29.259+0000 7fa977d99640 1 --2- 172.21.15.104:0/3350694860 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fa97006ef30 0x7fa9701e1430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:29.263 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:29.259+0000 7fa977d99640 1 --2- 172.21.15.104:0/3350694860 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fa97006eb50 0x7fa97009f270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:29.263 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:29.259+0000 7fa977d99640 1 -- 172.21.15.104:0/3350694860 >> 172.21.15.104:0/3350694860 conn(0x7fa97002cab0 msgr2=0x7fa970146ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T14:08:29.263 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:29.259+0000 7fa977d99640 1 -- 172.21.15.104:0/3350694860 shutdown_connections 2024-03-28T14:08:29.263 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:29.259+0000 7fa977d99640 1 -- 172.21.15.104:0/3350694860 wait complete. 2024-03-28T14:08:29.278 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-03-28T14:08:29.278 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T14:08:29.384 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:29.383+0000 7f73dbeea640 1 Processor -- start 2024-03-28T14:08:29.384 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:29.383+0000 7f73dbeea640 1 -- start start 2024-03-28T14:08:29.386 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:29.383+0000 7f73dbeea640 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f73d406b830 0x7f73d409fa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:29.386 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:29.383+0000 7f73dbeea640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f73d406bc10 con 0x7f73d40b4f70 2024-03-28T14:08:29.386 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:29.383+0000 7f73dbeea640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f73d40b5a10 con 0x7f73d409ff50 2024-03-28T14:08:29.386 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:29.383+0000 7f73dbeea640 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 -- 0x7f73d40b5cf0 con 0x7f73d406b830 2024-03-28T14:08:29.386 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:29.383+0000 7f73d9c86640 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f73d406b830 0x7f73d409fa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:29.386 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:29.383+0000 7f73d9c86640 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f73d406b830 0x7f73d409fa10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:58950/0 (socket says 172.21.15.104:58950) 2024-03-28T14:08:29.386 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:29.383+0000 7f73d9c86640 1 -- 172.21.15.104:0/3185933596 learned_addr learned my addr 172.21.15.104:0/3185933596 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-03-28T14:08:29.386 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:29.383+0000 7f73d9c86640 1 -- 172.21.15.104:0/3185933596 >> v1:172.21.15.171:6789/0 conn(0x7f73d409ff50 legacy=0x7f73d4072ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T14:08:29.386 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:29.383+0000 7f73d9c86640 1 -- 172.21.15.104:0/3185933596 >> v1:172.21.15.104:6789/0 conn(0x7f73d40b4f70 legacy=0x7f73d414ffe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:29.387 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:29.383+0000 7f73d9c86640 1 -- 172.21.15.104:0/3185933596 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f73d40b5fd0 con 0x7f73d406b830 2024-03-28T14:08:29.388 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:29.387+0000 7f73d9c86640 1 --2- 172.21.15.104:0/3185933596 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f73d406b830 0x7f73d409fa10 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f73d0009870 tx=0x7f73d001c400 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ca70cebd3e69d9c6 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:08:29.388 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:29.387+0000 7f73d8c84640 1 -- 172.21.15.104:0/3185933596 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73d00b1070 con 0x7f73d406b830 2024-03-28T14:08:29.388 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:29.387+0000 7f73d8c84640 1 -- 172.21.15.104:0/3185933596 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f73d00ac640 con 0x7f73d406b830 2024-03-28T14:08:29.388 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:29.387+0000 7f73d8c84640 1 -- 172.21.15.104:0/3185933596 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73d00aca10 con 0x7f73d406b830 2024-03-28T14:08:29.389 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:29.387+0000 7f73dbeea640 1 -- 172.21.15.104:0/3185933596 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f73d406b830 msgr2=0x7f73d409fa10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:29.389 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:29.387+0000 7f73dbeea640 1 --2- 172.21.15.104:0/3185933596 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f73d406b830 0x7f73d409fa10 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f73d0009870 tx=0x7f73d001c400 comp rx=0 tx=0).stop 2024-03-28T14:08:29.389 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:29.387+0000 7f73dbeea640 1 -- 172.21.15.104:0/3185933596 shutdown_connections 2024-03-28T14:08:29.389 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:29.387+0000 7f73dbeea640 1 --2- 172.21.15.104:0/3185933596 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f73d406b830 0x7f73d409fa10 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:29.389 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:29.387+0000 7f73dbeea640 1 -- 172.21.15.104:0/3185933596 >> 172.21.15.104:0/3185933596 conn(0x7f73d40ba400 msgr2=0x7f73d40ba810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T14:08:29.389 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:29.387+0000 7f73dbeea640 1 -- 172.21.15.104:0/3185933596 shutdown_connections 2024-03-28T14:08:29.389 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:29.387+0000 7f73dbeea640 1 -- 172.21.15.104:0/3185933596 wait complete. 2024-03-28T14:08:29.390 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:29.387+0000 7f73dbeea640 1 Processor -- start 2024-03-28T14:08:29.390 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:29.387+0000 7f73dbeea640 1 -- start start 2024-03-28T14:08:29.392 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:29.387+0000 7f73dbeea640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f73d406b830 0x7f73d409e490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:29.392 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:29.387+0000 7f73dbeea640 1 --2- >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f73d409ff50 0x7f73d41de980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:29.392 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:29.387+0000 7f73dbeea640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f73d40b4f70 0x7f73d41deec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:29.392 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:29.387+0000 7f73dbeea640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f73d41540a0 con 0x7f73d40b4f70 2024-03-28T14:08:29.392 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:29.387+0000 7f73dbeea640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f73d4153f20 con 0x7f73d406b830 2024-03-28T14:08:29.392 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:29.387+0000 7f73dbeea640 1 -- --> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] -- mon_getmap magic: 0 -- 0x7f73d4154220 con 0x7f73d409ff50 2024-03-28T14:08:29.392 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:29.387+0000 7f73da487640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f73d40b4f70 0x7f73d41deec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:29.393 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:29.387+0000 7f73da487640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f73d40b4f70 0x7f73d41deec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:58960/0 (socket says 172.21.15.104:58960) 2024-03-28T14:08:29.393 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:29.387+0000 7f73da487640 1 -- 172.21.15.104:0/2069059680 learned_addr learned my addr 172.21.15.104:0/2069059680 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-03-28T14:08:29.393 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:29.387+0000 7f73d9c86640 1 --2- 172.21.15.104:0/2069059680 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f73d406b830 0x7f73d409e490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:29.393 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:29.387+0000 7f73d9485640 1 --2- 172.21.15.104:0/2069059680 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f73d409ff50 0x7f73d41de980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:29.393 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:29.387+0000 7f73da487640 1 -- 172.21.15.104:0/2069059680 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f73d409ff50 msgr2=0x7f73d41de980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:29.393 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:29.387+0000 7f73da487640 1 --2- 172.21.15.104:0/2069059680 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f73d409ff50 0x7f73d41de980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:29.393 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:29.387+0000 7f73da487640 1 -- 172.21.15.104:0/2069059680 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f73d406b830 msgr2=0x7f73d409e490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:29.393 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:29.387+0000 7f73da487640 1 --2- 172.21.15.104:0/2069059680 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f73d406b830 0x7f73d409e490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:29.393 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:29.387+0000 7f73da487640 1 -- 172.21.15.104:0/2069059680 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f73d409f230 con 0x7f73d40b4f70 2024-03-28T14:08:29.393 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:29.387+0000 7f73da487640 1 --2- 172.21.15.104:0/2069059680 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f73d40b4f70 0x7f73d41deec0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f73cc00c760 tx=0x7f73cc00cc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:08:29.393 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:29.391+0000 7f73ca7fc640 1 -- 172.21.15.104:0/2069059680 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73cc016020 con 0x7f73d40b4f70 2024-03-28T14:08:29.393 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:29.391+0000 7f73ca7fc640 1 -- 172.21.15.104:0/2069059680 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f73cc006d80 con 0x7f73d40b4f70 2024-03-28T14:08:29.393 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:29.391+0000 7f73dbeea640 1 -- 172.21.15.104:0/2069059680 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f73d40b1860 con 0x7f73d40b4f70 2024-03-28T14:08:29.394 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:29.391+0000 7f73ca7fc640 1 -- 172.21.15.104:0/2069059680 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73cc003bc0 con 0x7f73d40b4f70 2024-03-28T14:08:29.394 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:29.391+0000 7f73dbeea640 1 -- 172.21.15.104:0/2069059680 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f73d409d2d0 con 0x7f73d40b4f70 2024-03-28T14:08:29.394 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:29.391+0000 7f73dbeea640 1 -- 172.21.15.104:0/2069059680 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f73d40687c0 con 0x7f73d40b4f70 2024-03-28T14:08:29.394 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:29.391+0000 7f73ca7fc640 1 -- 172.21.15.104:0/2069059680 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) ==== 87803+0+0 (secure 0 0 0) 0x7f73cc00a7f0 con 0x7f73d40b4f70 2024-03-28T14:08:29.395 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:29.391+0000 7f73ca7fc640 1 --2- 172.21.15.104:0/2069059680 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f73b8067fc0 0x7f73b806a480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:29.395 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:29.391+0000 7f73ca7fc640 1 -- 172.21.15.104:0/2069059680 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f73cc086020 con 0x7f73d40b4f70 2024-03-28T14:08:29.395 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:29.391+0000 7f73d9c86640 1 --2- 172.21.15.104:0/2069059680 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f73b8067fc0 0x7f73b806a480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:29.395 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:29.391+0000 7f73d9c86640 1 --2- 172.21.15.104:0/2069059680 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f73b8067fc0 0x7f73b806a480 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f73d00ae000 tx=0x7f73d00a6070 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:08:29.397 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:29.395+0000 7f73ca7fc640 1 -- 172.21.15.104:0/2069059680 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179117 (secure 0 0 0) 0x7f73cc020020 con 0x7f73d40b4f70 2024-03-28T14:08:29.584 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:29.583+0000 7f73dbeea640 1 -- 172.21.15.104:0/2069059680 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f73d4146c30 con 0x7f73d40b4f70 2024-03-28T14:08:29.584 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:29.583+0000 7f73ca7fc640 1 -- 172.21.15.104:0/2069059680 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 7 v7) ==== 93+0+3533 (secure 0 0 0) 0x7f73cc056860 con 0x7f73d40b4f70 2024-03-28T14:08:29.585 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-03-28T14:08:29.585 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":7,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4229,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.171:6835/1775867467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6834","nonce":1775867467},{"type":"v1","addr":"172.21.15.171:6835","nonce":1775867467}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":7,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T14:07:59.027458+0000","modified":"2024-03-28T14:08:02.057923+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4245},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4245":{"gid":4245,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.104:6833/3184673325","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":3184673325},{"type":"v1","addr":"172.21.15.104:6833","nonce":3184673325}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4245,"qdb_cluster":[4245]},"id":1}]} 2024-03-28T14:08:29.585 INFO:teuthology.orchestra.run.smithi104.stderr:dumped fsmap epoch 7 2024-03-28T14:08:29.586 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:29.583+0000 7f73dbeea640 1 -- 172.21.15.104:0/2069059680 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f73b8067fc0 msgr2=0x7f73b806a480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:29.586 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:29.583+0000 7f73dbeea640 1 --2- 172.21.15.104:0/2069059680 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f73b8067fc0 0x7f73b806a480 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f73d00ae000 tx=0x7f73d00a6070 comp rx=0 tx=0).stop 2024-03-28T14:08:29.586 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:29.583+0000 7f73dbeea640 1 -- 172.21.15.104:0/2069059680 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f73d40b4f70 msgr2=0x7f73d41deec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:29.587 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:29.583+0000 7f73dbeea640 1 --2- 172.21.15.104:0/2069059680 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f73d40b4f70 0x7f73d41deec0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f73cc00c760 tx=0x7f73cc00cc20 comp rx=0 tx=0).stop 2024-03-28T14:08:29.587 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:29.583+0000 7f73dbeea640 1 -- 172.21.15.104:0/2069059680 shutdown_connections 2024-03-28T14:08:29.587 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:29.583+0000 7f73dbeea640 1 --2- 172.21.15.104:0/2069059680 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f73b8067fc0 0x7f73b806a480 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:29.587 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:29.583+0000 7f73dbeea640 1 --2- 172.21.15.104:0/2069059680 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f73d40b4f70 0x7f73d41deec0 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:29.587 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:29.583+0000 7f73dbeea640 1 --2- 172.21.15.104:0/2069059680 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f73d409ff50 0x7f73d41de980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:29.587 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:29.583+0000 7f73dbeea640 1 --2- 172.21.15.104:0/2069059680 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f73d406b830 0x7f73d409e490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:29.587 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:29.583+0000 7f73dbeea640 1 -- 172.21.15.104:0/2069059680 >> 172.21.15.104:0/2069059680 conn(0x7f73d40ba400 msgr2=0x7f73d40b9840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T14:08:29.587 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:29.583+0000 7f73dbeea640 1 -- 172.21.15.104:0/2069059680 shutdown_connections 2024-03-28T14:08:29.587 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:29.583+0000 7f73dbeea640 1 -- 172.21.15.104:0/2069059680 wait complete. 2024-03-28T14:08:29.601 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T14:08:29.723 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:29.719+0000 7feffd17f640 1 Processor -- start 2024-03-28T14:08:29.723 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:29.719+0000 7feffd17f640 1 -- start start 2024-03-28T14:08:29.725 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:29.723+0000 7feffd17f640 1 --2- >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7feff80b58b0 0x7feff80af020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:29.725 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:29.723+0000 7feffd17f640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7feff80b1170 con 0x7feff80b0d90 2024-03-28T14:08:29.726 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:29.723+0000 7feffd17f640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7feff80b51e0 con 0x7feff806e020 2024-03-28T14:08:29.726 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:29.723+0000 7feffd17f640 1 -- --> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] -- mon_getmap magic: 0 -- 0x7feff80af560 con 0x7feff80b58b0 2024-03-28T14:08:29.726 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:29.723+0000 7feff6d9d640 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7feff80b0d90 0x7feff814adb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:41612/0 (socket says 172.21.15.104:41612) 2024-03-28T14:08:29.726 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:29.723+0000 7feff6d9d640 1 -- 172.21.15.104:0/3377559372 learned_addr learned my addr 172.21.15.104:0/3377559372 (peer_addr_for_me v1:172.21.15.104:0/0) 2024-03-28T14:08:29.726 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:29.723+0000 7feff759e640 1 --2- 172.21.15.104:0/3377559372 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7feff80b58b0 0x7feff80af020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:29.726 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:29.723+0000 7feff5d9b640 1 -- 172.21.15.104:0/3377559372 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2341520743 0 0) 0x7feff80b1170 con 0x7feff80b0d90 2024-03-28T14:08:29.726 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:29.723+0000 7feff5d9b640 1 -- 172.21.15.104:0/3377559372 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fefe0003530 con 0x7feff80b0d90 2024-03-28T14:08:29.726 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:29.723+0000 7feff5d9b640 1 -- 172.21.15.104:0/3377559372 <== mon.0 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3371159125 0 0) 0x7fefe0003530 con 0x7feff80b0d90 2024-03-28T14:08:29.726 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:29.723+0000 7feff5d9b640 1 -- 172.21.15.104:0/3377559372 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7feff80b58b0 msgr2=0x7feff80af020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T14:08:29.726 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:29.723+0000 7feff5d9b640 1 --2- 172.21.15.104:0/3377559372 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7feff80b58b0 0x7feff80af020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:29.726 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:29.723+0000 7feff5d9b640 1 -- 172.21.15.104:0/3377559372 >> v1:172.21.15.171:6789/0 conn(0x7feff806e020 legacy=0x7feff8072d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:29.726 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:29.723+0000 7feff5d9b640 1 -- 172.21.15.104:0/3377559372 --> v1:172.21.15.104:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feff80b54c0 con 0x7feff80b0d90 2024-03-28T14:08:29.726 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:29.723+0000 7feff5d9b640 1 -- 172.21.15.104:0/3377559372 <== mon.0 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3745580849 0 0) 0x7feff0001a00 con 0x7feff80b0d90 2024-03-28T14:08:29.727 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:29.723+0000 7feff5d9b640 1 -- 172.21.15.104:0/3377559372 >> v1:172.21.15.104:6789/0 conn(0x7feff80b0d90 legacy=0x7feff814adb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:29.727 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:29.723+0000 7feff5d9b640 1 --2- 172.21.15.104:0/3377559372 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fefe00039f0 0x7fefe0003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:29.727 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:29.723+0000 7feff5d9b640 1 --2- 172.21.15.104:0/3377559372 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fefe0004810 0x7fefe0004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:29.727 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:29.723+0000 7feff5d9b640 1 --2- 172.21.15.104:0/3377559372 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7feff80b58b0 0x7fefe00052a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:29.727 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:29.723+0000 7feff5d9b640 1 -- 172.21.15.104:0/3377559372 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7feff80af560 con 0x7fefe0004810 2024-03-28T14:08:29.727 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:29.723+0000 7feff5d9b640 1 -- 172.21.15.104:0/3377559372 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7fefe000bb90 con 0x7fefe00039f0 2024-03-28T14:08:29.727 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:29.723+0000 7feff5d9b640 1 -- 172.21.15.104:0/3377559372 --> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] -- mon_getmap magic: 0 -- 0x7fefe000bd90 con 0x7feff80b58b0 2024-03-28T14:08:29.727 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:29.723+0000 7feff659c640 1 --2- 172.21.15.104:0/3377559372 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fefe0004810 0x7fefe0004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:29.727 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:29.723+0000 7feff759e640 1 --2- 172.21.15.104:0/3377559372 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7feff80b58b0 0x7fefe00052a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:29.727 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:29.723+0000 7feff6d9d640 1 --2- 172.21.15.104:0/3377559372 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fefe00039f0 0x7fefe0003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:29.727 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:29.723+0000 7feff659c640 1 -- 172.21.15.104:0/3377559372 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7feff80b58b0 msgr2=0x7fefe00052a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T14:08:29.727 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:29.723+0000 7feff659c640 1 --2- 172.21.15.104:0/3377559372 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7feff80b58b0 0x7fefe00052a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:29.727 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:29.723+0000 7feff659c640 1 -- 172.21.15.104:0/3377559372 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fefe00039f0 msgr2=0x7fefe0003e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T14:08:29.728 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:29.723+0000 7feff659c640 1 --2- 172.21.15.104:0/3377559372 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fefe00039f0 0x7fefe0003e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:29.728 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:29.723+0000 7feff659c640 1 -- 172.21.15.104:0/3377559372 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fefe0003530 con 0x7fefe0004810 2024-03-28T14:08:29.728 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:29.723+0000 7feff6d9d640 1 --2- 172.21.15.104:0/3377559372 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fefe00039f0 0x7fefe0003e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T14:08:29.729 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:29.727+0000 7feff659c640 1 --2- 172.21.15.104:0/3377559372 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fefe0004810 0x7fefe0004c90 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7fefec001780 tx=0x7fefec01c420 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fb8feed45ed59b1a server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:08:29.729 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:29.727+0000 7feff5d9b640 1 -- 172.21.15.104:0/3377559372 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fefec0013a0 con 0x7fefe0004810 2024-03-28T14:08:29.729 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:29.727+0000 7feff5d9b640 1 -- 172.21.15.104:0/3377559372 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fefec0052f0 con 0x7fefe0004810 2024-03-28T14:08:29.729 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:29.727+0000 7feff5d9b640 1 -- 172.21.15.104:0/3377559372 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fefec003840 con 0x7fefe0004810 2024-03-28T14:08:29.729 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:29.727+0000 7feffd17f640 1 -- 172.21.15.104:0/3377559372 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fefe0004810 msgr2=0x7fefe0004c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:29.729 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:29.727+0000 7feffd17f640 1 --2- 172.21.15.104:0/3377559372 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fefe0004810 0x7fefe0004c90 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7fefec001780 tx=0x7fefec01c420 comp rx=0 tx=0).stop 2024-03-28T14:08:29.730 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:29.727+0000 7feff759e640 1 -- 172.21.15.104:0/3377559372 reap_dead start 2024-03-28T14:08:29.730 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:29.727+0000 7feffd17f640 1 -- 172.21.15.104:0/3377559372 shutdown_connections 2024-03-28T14:08:29.730 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:29.727+0000 7feffd17f640 1 -- 172.21.15.104:0/3377559372 >> 172.21.15.104:0/3377559372 conn(0x7feff80ba400 msgr2=0x7feff80ba810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T14:08:29.730 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:29.727+0000 7feffd17f640 1 -- 172.21.15.104:0/3377559372 shutdown_connections 2024-03-28T14:08:29.730 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:29.727+0000 7feffd17f640 1 -- 172.21.15.104:0/3377559372 wait complete. 2024-03-28T14:08:29.732 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:29.727+0000 7feffd17f640 1 Processor -- start 2024-03-28T14:08:29.732 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:29.727+0000 7feffd17f640 1 -- start start 2024-03-28T14:08:29.732 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:29.727+0000 7feffd17f640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fefe0004810 0x7feff80b42f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:29.732 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:29.727+0000 7feffd17f640 1 --2- >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7feff81dfe40 0x7feff814cd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:29.733 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:29.727+0000 7feffd17f640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7feff81e0460 0x7feff81e0910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:29.733 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:29.727+0000 7feffd17f640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7feff806db90 con 0x7fefe0004810 2024-03-28T14:08:29.733 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:29.727+0000 7feffd17f640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7feff8073630 con 0x7feff81e0460 2024-03-28T14:08:29.733 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:29.727+0000 7feffd17f640 1 -- --> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] -- mon_getmap magic: 0 -- 0x7feff8153b10 con 0x7feff81dfe40 2024-03-28T14:08:29.733 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:29.727+0000 7feff6d9d640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fefe0004810 0x7feff80b42f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:29.733 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:29.727+0000 7feff6d9d640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fefe0004810 0x7feff80b42f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:58972/0 (socket says 172.21.15.104:58972) 2024-03-28T14:08:29.733 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:29.727+0000 7feff6d9d640 1 -- 172.21.15.104:0/3477673147 learned_addr learned my addr 172.21.15.104:0/3477673147 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-03-28T14:08:29.733 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:29.727+0000 7feff659c640 1 --2- 172.21.15.104:0/3477673147 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7feff81dfe40 0x7feff814cd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:29.733 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:29.727+0000 7feff6d9d640 1 -- 172.21.15.104:0/3477673147 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7feff81dfe40 msgr2=0x7feff814cd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:29.733 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:29.727+0000 7feff6d9d640 1 --2- 172.21.15.104:0/3477673147 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7feff81dfe40 0x7feff814cd80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:29.733 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:29.727+0000 7feff6d9d640 1 -- 172.21.15.104:0/3477673147 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7feff81e0460 msgr2=0x7feff81e0910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:29.733 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:29.727+0000 7feff759e640 1 --2- 172.21.15.104:0/3477673147 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7feff81e0460 0x7feff81e0910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:29.733 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:29.727+0000 7feff6d9d640 1 --2- 172.21.15.104:0/3477673147 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7feff81e0460 0x7feff81e0910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:29.733 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:29.727+0000 7feff6d9d640 1 -- 172.21.15.104:0/3477673147 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feff814e180 con 0x7fefe0004810 2024-03-28T14:08:29.734 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:29.731+0000 7feff659c640 1 --2- 172.21.15.104:0/3477673147 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7feff81dfe40 0x7feff814cd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T14:08:29.734 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:29.731+0000 7feff6d9d640 1 --2- 172.21.15.104:0/3477673147 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fefe0004810 0x7feff80b42f0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7feff0007a40 tx=0x7feff000e530 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:08:29.734 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:29.731+0000 7fefdf7fe640 1 -- 172.21.15.104:0/3477673147 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feff000ee50 con 0x7fefe0004810 2024-03-28T14:08:29.734 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:29.731+0000 7fefdf7fe640 1 -- 172.21.15.104:0/3477673147 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7feff0018070 con 0x7fefe0004810 2024-03-28T14:08:29.734 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:29.731+0000 7feffd17f640 1 -- 172.21.15.104:0/3477673147 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7feff814e460 con 0x7fefe0004810 2024-03-28T14:08:29.734 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:29.731+0000 7fefdf7fe640 1 -- 172.21.15.104:0/3477673147 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feff0013620 con 0x7fefe0004810 2024-03-28T14:08:29.734 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:29.731+0000 7feffd17f640 1 -- 172.21.15.104:0/3477673147 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7feff8150760 con 0x7fefe0004810 2024-03-28T14:08:29.734 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:29.731+0000 7feffd17f640 1 -- 172.21.15.104:0/3477673147 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7feff814c380 con 0x7fefe0004810 2024-03-28T14:08:29.735 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:29.731+0000 7fefdf7fe640 1 -- 172.21.15.104:0/3477673147 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) ==== 87803+0+0 (secure 0 0 0) 0x7feff00137c0 con 0x7fefe0004810 2024-03-28T14:08:29.735 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:29.731+0000 7fefdf7fe640 1 --2- 172.21.15.104:0/3477673147 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7fefe4067fc0 0x7fefe406a480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:29.735 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:29.731+0000 7fefdf7fe640 1 -- 172.21.15.104:0/3477673147 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7feff0086b20 con 0x7fefe0004810 2024-03-28T14:08:29.735 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:29.731+0000 7feff659c640 1 --2- 172.21.15.104:0/3477673147 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7fefe4067fc0 0x7fefe406a480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:29.735 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:29.731+0000 7feff659c640 1 --2- 172.21.15.104:0/3477673147 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7fefe4067fc0 0x7fefe406a480 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fefec001750 tx=0x7fefec0011a0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:08:29.738 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:29.735+0000 7fefdf7fe640 1 -- 172.21.15.104:0/3477673147 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179117 (secure 0 0 0) 0x7feff0057360 con 0x7fefe0004810 2024-03-28T14:08:29.933 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:29.931+0000 7feffd17f640 1 -- 172.21.15.104:0/3477673147 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7feff80687c0 con 0x7fefe0004810 2024-03-28T14:08:29.934 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:29.931+0000 7fefdf7fe640 1 -- 172.21.15.104:0/3477673147 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 7 v7) ==== 93+0+3533 (secure 0 0 0) 0x7feff0056c60 con 0x7fefe0004810 2024-03-28T14:08:29.934 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-03-28T14:08:29.934 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":7,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4229,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.171:6835/1775867467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6834","nonce":1775867467},{"type":"v1","addr":"172.21.15.171:6835","nonce":1775867467}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":7,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T14:07:59.027458+0000","modified":"2024-03-28T14:08:02.057923+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4245},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4245":{"gid":4245,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.104:6833/3184673325","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":3184673325},{"type":"v1","addr":"172.21.15.104:6833","nonce":3184673325}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4245,"qdb_cluster":[4245]},"id":1}]} 2024-03-28T14:08:29.934 INFO:teuthology.orchestra.run.smithi104.stderr:dumped fsmap epoch 7 2024-03-28T14:08:29.936 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:29.935+0000 7feffd17f640 1 -- 172.21.15.104:0/3477673147 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7fefe4067fc0 msgr2=0x7fefe406a480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:29.936 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:29.935+0000 7feffd17f640 1 --2- 172.21.15.104:0/3477673147 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7fefe4067fc0 0x7fefe406a480 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fefec001750 tx=0x7fefec0011a0 comp rx=0 tx=0).stop 2024-03-28T14:08:29.936 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:29.935+0000 7feffd17f640 1 -- 172.21.15.104:0/3477673147 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fefe0004810 msgr2=0x7feff80b42f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:29.936 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:29.935+0000 7feffd17f640 1 --2- 172.21.15.104:0/3477673147 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fefe0004810 0x7feff80b42f0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7feff0007a40 tx=0x7feff000e530 comp rx=0 tx=0).stop 2024-03-28T14:08:29.936 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:29.935+0000 7feffd17f640 1 -- 172.21.15.104:0/3477673147 shutdown_connections 2024-03-28T14:08:29.936 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:29.935+0000 7feffd17f640 1 --2- 172.21.15.104:0/3477673147 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7feff81e0460 0x7feff81e0910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:29.936 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:29.935+0000 7feffd17f640 1 --2- 172.21.15.104:0/3477673147 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7feff81dfe40 0x7feff814cd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:29.936 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:29.935+0000 7feffd17f640 1 --2- 172.21.15.104:0/3477673147 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7fefe4067fc0 0x7fefe406a480 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:29.936 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:29.935+0000 7feffd17f640 1 --2- 172.21.15.104:0/3477673147 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fefe0004810 0x7feff80b42f0 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:29.937 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:29.935+0000 7feffd17f640 1 -- 172.21.15.104:0/3477673147 >> 172.21.15.104:0/3477673147 conn(0x7feff80ba400 msgr2=0x7feff806bd20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T14:08:29.937 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:29.935+0000 7feffd17f640 1 -- 172.21.15.104:0/3477673147 shutdown_connections 2024-03-28T14:08:29.937 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:29.935+0000 7feffd17f640 1 -- 172.21.15.104:0/3477673147 wait complete. 2024-03-28T14:08:29.951 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-28T14:08:30.071 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:30.067+0000 7fa19ba53640 1 Processor -- start 2024-03-28T14:08:30.071 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:30.067+0000 7fa19ba53640 1 -- start start 2024-03-28T14:08:30.076 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:30.071+0000 7fa19ba53640 1 --2- >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fa19413faf0 0x7fa1940b0c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:30.076 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:30.071+0000 7fa19ba53640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa19406dd50 con 0x7fa19406d970 2024-03-28T14:08:30.076 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:30.071+0000 7fa19ba53640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa1940b2d30 con 0x7fa1940732a0 2024-03-28T14:08:30.076 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:30.071+0000 7fa19ba53640 1 -- --> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] -- mon_getmap magic: 0 -- 0x7fa1940b02a0 con 0x7fa19413faf0 2024-03-28T14:08:30.076 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:30.071+0000 7fa1997ef640 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7fa19406d970 0x7fa19413f3e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:41614/0 (socket says 172.21.15.104:41614) 2024-03-28T14:08:30.076 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:30.071+0000 7fa1997ef640 1 -- 172.21.15.104:0/1645940954 learned_addr learned my addr 172.21.15.104:0/1645940954 (peer_addr_for_me v1:172.21.15.104:0/0) 2024-03-28T14:08:30.076 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:30.071+0000 7fa198fee640 1 --2- 172.21.15.104:0/1645940954 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fa19413faf0 0x7fa1940b0c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:30.076 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:30.071+0000 7fa193fff640 1 -- 172.21.15.104:0/1645940954 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4103937086 0 0) 0x7fa19406dd50 con 0x7fa19406d970 2024-03-28T14:08:30.076 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:30.071+0000 7fa193fff640 1 -- 172.21.15.104:0/1645940954 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa180003530 con 0x7fa19406d970 2024-03-28T14:08:30.077 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:30.071+0000 7fa193fff640 1 -- 172.21.15.104:0/1645940954 <== mon.0 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2828809693 0 0) 0x7fa180003530 con 0x7fa19406d970 2024-03-28T14:08:30.077 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:30.071+0000 7fa193fff640 1 -- 172.21.15.104:0/1645940954 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fa19413faf0 msgr2=0x7fa1940b0c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T14:08:30.077 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:30.071+0000 7fa193fff640 1 --2- 172.21.15.104:0/1645940954 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fa19413faf0 0x7fa1940b0c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:30.077 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:30.071+0000 7fa193fff640 1 -- 172.21.15.104:0/1645940954 >> v1:172.21.15.171:6789/0 conn(0x7fa1940732a0 legacy=0x7fa19414fe60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:30.077 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:30.071+0000 7fa193fff640 1 -- 172.21.15.104:0/1645940954 --> v1:172.21.15.104:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa1940b3010 con 0x7fa19406d970 2024-03-28T14:08:30.077 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:30.071+0000 7fa193fff640 1 -- 172.21.15.104:0/1645940954 <== mon.0 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3745580849 0 0) 0x7fa184001680 con 0x7fa19406d970 2024-03-28T14:08:30.077 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:30.071+0000 7fa193fff640 1 -- 172.21.15.104:0/1645940954 >> v1:172.21.15.104:6789/0 conn(0x7fa19406d970 legacy=0x7fa19413f3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:30.077 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:30.071+0000 7fa193fff640 1 --2- 172.21.15.104:0/1645940954 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fa1800039f0 0x7fa180003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:30.077 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:30.071+0000 7fa193fff640 1 --2- 172.21.15.104:0/1645940954 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fa180004810 0x7fa180004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:30.077 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:30.071+0000 7fa193fff640 1 --2- 172.21.15.104:0/1645940954 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa180005220 0x7fa18000b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:30.078 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:30.071+0000 7fa193fff640 1 -- 172.21.15.104:0/1645940954 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7fa1940b02a0 con 0x7fa180005220 2024-03-28T14:08:30.078 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:30.071+0000 7fa193fff640 1 -- 172.21.15.104:0/1645940954 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7fa18000c120 con 0x7fa1800039f0 2024-03-28T14:08:30.078 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:30.071+0000 7fa193fff640 1 -- 172.21.15.104:0/1645940954 --> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] -- mon_getmap magic: 0 -- 0x7fa18000c320 con 0x7fa180004810 2024-03-28T14:08:30.078 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:30.071+0000 7fa198fee640 1 --2- 172.21.15.104:0/1645940954 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fa19413faf0 0x7fa1940b0c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T14:08:30.078 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:30.071+0000 7fa199ff0640 1 --2- 172.21.15.104:0/1645940954 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa180005220 0x7fa18000b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:30.078 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:30.071+0000 7fa1997ef640 1 --2- 172.21.15.104:0/1645940954 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fa1800039f0 0x7fa180003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:30.078 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:30.071+0000 7fa198fee640 1 --2- 172.21.15.104:0/1645940954 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fa180004810 0x7fa180004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:30.078 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:30.071+0000 7fa199ff0640 1 -- 172.21.15.104:0/1645940954 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fa180004810 msgr2=0x7fa180004c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T14:08:30.079 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:30.071+0000 7fa199ff0640 1 --2- 172.21.15.104:0/1645940954 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fa180004810 0x7fa180004c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:30.079 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:30.071+0000 7fa199ff0640 1 -- 172.21.15.104:0/1645940954 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fa1800039f0 msgr2=0x7fa180003e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T14:08:30.079 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:30.071+0000 7fa199ff0640 1 --2- 172.21.15.104:0/1645940954 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fa1800039f0 0x7fa180003e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:30.079 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:30.071+0000 7fa199ff0640 1 -- 172.21.15.104:0/1645940954 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa180003530 con 0x7fa180005220 2024-03-28T14:08:30.079 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:30.071+0000 7fa198fee640 1 --2- 172.21.15.104:0/1645940954 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fa180004810 0x7fa180004c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T14:08:30.079 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:30.071+0000 7fa199ff0640 1 --2- 172.21.15.104:0/1645940954 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa180005220 0x7fa18000b9d0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7fa17c001750 tx=0x7fa17c01c3e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8bb2c944deaed7c9 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:08:30.079 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:30.071+0000 7fa193fff640 1 -- 172.21.15.104:0/1645940954 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa17c0013a0 con 0x7fa180005220 2024-03-28T14:08:30.079 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:30.071+0000 7fa193fff640 1 -- 172.21.15.104:0/1645940954 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa17c0052f0 con 0x7fa180005220 2024-03-28T14:08:30.079 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:30.071+0000 7fa193fff640 1 -- 172.21.15.104:0/1645940954 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa17c003840 con 0x7fa180005220 2024-03-28T14:08:30.079 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:30.075+0000 7fa19ba53640 1 -- 172.21.15.104:0/1645940954 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa180005220 msgr2=0x7fa18000b9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:30.079 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:30.075+0000 7fa19ba53640 1 --2- 172.21.15.104:0/1645940954 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa180005220 0x7fa18000b9d0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7fa17c001750 tx=0x7fa17c01c3e0 comp rx=0 tx=0).stop 2024-03-28T14:08:30.079 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:30.075+0000 7fa199ff0640 1 -- 172.21.15.104:0/1645940954 reap_dead start 2024-03-28T14:08:30.079 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:30.075+0000 7fa19ba53640 1 -- 172.21.15.104:0/1645940954 shutdown_connections 2024-03-28T14:08:30.080 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:30.075+0000 7fa19ba53640 1 -- 172.21.15.104:0/1645940954 >> 172.21.15.104:0/1645940954 conn(0x7fa1940b58a0 msgr2=0x7fa1940b5cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T14:08:30.080 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:30.075+0000 7fa19ba53640 1 -- 172.21.15.104:0/1645940954 shutdown_connections 2024-03-28T14:08:30.080 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:30.075+0000 7fa19ba53640 1 -- 172.21.15.104:0/1645940954 wait complete. 2024-03-28T14:08:30.083 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:30.075+0000 7fa19ba53640 1 Processor -- start 2024-03-28T14:08:30.083 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:30.075+0000 7fa19ba53640 1 -- start start 2024-03-28T14:08:30.083 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:30.075+0000 7fa19ba53640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fa180005220 0x7fa1940b8350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:30.083 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:30.075+0000 7fa19ba53640 1 --2- >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fa1940addc0 0x7fa194130870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:30.083 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:30.075+0000 7fa19ba53640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa194130db0 0x7fa1941e0ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:30.083 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:30.075+0000 7fa19ba53640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7fa1940b5460 con 0x7fa194130db0 2024-03-28T14:08:30.084 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:30.075+0000 7fa19ba53640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7fa19409eef0 con 0x7fa180005220 2024-03-28T14:08:30.084 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:30.075+0000 7fa19ba53640 1 -- --> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] -- mon_getmap magic: 0 -- 0x7fa1940b55e0 con 0x7fa1940addc0 2024-03-28T14:08:30.084 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:30.075+0000 7fa199ff0640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa194130db0 0x7fa1941e0ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:30.084 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:30.075+0000 7fa199ff0640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa194130db0 0x7fa1941e0ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:58986/0 (socket says 172.21.15.104:58986) 2024-03-28T14:08:30.084 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:30.075+0000 7fa199ff0640 1 -- 172.21.15.104:0/875805234 learned_addr learned my addr 172.21.15.104:0/875805234 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-03-28T14:08:30.084 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:30.075+0000 7fa1997ef640 1 --2- 172.21.15.104:0/875805234 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fa180005220 0x7fa1940b8350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:30.084 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:30.075+0000 7fa198fee640 1 --2- 172.21.15.104:0/875805234 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fa1940addc0 0x7fa194130870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:30.084 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:30.075+0000 7fa199ff0640 1 -- 172.21.15.104:0/875805234 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fa1940addc0 msgr2=0x7fa194130870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:30.084 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:30.075+0000 7fa199ff0640 1 --2- 172.21.15.104:0/875805234 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fa1940addc0 0x7fa194130870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:30.084 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:30.075+0000 7fa199ff0640 1 -- 172.21.15.104:0/875805234 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fa180005220 msgr2=0x7fa1940b8350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:30.084 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:30.075+0000 7fa199ff0640 1 --2- 172.21.15.104:0/875805234 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fa180005220 0x7fa1940b8350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:30.084 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:30.075+0000 7fa199ff0640 1 -- 172.21.15.104:0/875805234 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa1940ae7b0 con 0x7fa194130db0 2024-03-28T14:08:30.084 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:30.075+0000 7fa198fee640 1 --2- 172.21.15.104:0/875805234 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fa1940addc0 0x7fa194130870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T14:08:30.084 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:30.075+0000 7fa1997ef640 1 --2- 172.21.15.104:0/875805234 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fa180005220 0x7fa1940b8350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T14:08:30.085 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:30.075+0000 7fa199ff0640 1 --2- 172.21.15.104:0/875805234 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa194130db0 0x7fa1941e0ed0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7fa17c001750 tx=0x7fa180005840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:08:30.085 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:30.075+0000 7fa191ffb640 1 -- 172.21.15.104:0/875805234 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa17c0bc070 con 0x7fa194130db0 2024-03-28T14:08:30.085 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:30.075+0000 7fa191ffb640 1 -- 172.21.15.104:0/875805234 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa17c0054a0 con 0x7fa194130db0 2024-03-28T14:08:30.085 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:30.075+0000 7fa19ba53640 1 -- 172.21.15.104:0/875805234 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa1940aea90 con 0x7fa194130db0 2024-03-28T14:08:30.085 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:30.075+0000 7fa191ffb640 1 -- 172.21.15.104:0/875805234 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa17c0013a0 con 0x7fa194130db0 2024-03-28T14:08:30.085 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:30.075+0000 7fa19ba53640 1 -- 172.21.15.104:0/875805234 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa194136710 con 0x7fa194130db0 2024-03-28T14:08:30.085 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:30.075+0000 7fa19ba53640 1 -- 172.21.15.104:0/875805234 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa1940687c0 con 0x7fa194130db0 2024-03-28T14:08:30.085 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:30.079+0000 7fa191ffb640 1 -- 172.21.15.104:0/875805234 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) ==== 87803+0+0 (secure 0 0 0) 0x7fa17c0b4040 con 0x7fa194130db0 2024-03-28T14:08:30.085 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:30.079+0000 7fa191ffb640 1 --2- 172.21.15.104:0/875805234 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7fa174067ff0 0x7fa17406a4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:30.085 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:30.079+0000 7fa191ffb640 1 -- 172.21.15.104:0/875805234 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7fa17c11fdf0 con 0x7fa194130db0 2024-03-28T14:08:30.085 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:30.079+0000 7fa1997ef640 1 --2- 172.21.15.104:0/875805234 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7fa174067ff0 0x7fa17406a4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:30.085 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:30.079+0000 7fa1997ef640 1 --2- 172.21.15.104:0/875805234 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7fa174067ff0 0x7fa17406a4b0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fa184000c10 tx=0x7fa184009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:08:30.089 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:30.087+0000 7fa191ffb640 1 -- 172.21.15.104:0/875805234 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179117 (secure 0 0 0) 0x7fa17c0160f0 con 0x7fa194130db0 2024-03-28T14:08:30.238 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:30.235+0000 7fa19ba53640 1 -- 172.21.15.104:0/875805234 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fa1941431b0 con 0x7fa194130db0 2024-03-28T14:08:30.241 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:30.239+0000 7fa191ffb640 1 -- 172.21.15.104:0/875805234 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) ==== 74+0+16629 (secure 0 0 0) 0x7fa17c0eff50 con 0x7fa194130db0 2024-03-28T14:08:30.241 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-03-28T14:08:30.241 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":21,"fsid":"428a90f2-3351-435f-8fcf-f7e66c78a13e","created":"2024-03-28T14:07:34.862458+0000","modified":"2024-03-28T14:08:00.671587+0000","last_up_change":"2024-03-28T14:07:46.028122+0000","last_in_change":"2024-03-28T14:07:37.157486+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-28T14:07:47.622988+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-28T14:07:57.147450+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-28T14:07:57.990975+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"1cb70540-4e42-402f-a446-aaec4ca83007","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6800","nonce":3745690918},{"type":"v1","addr":"172.21.15.104:6801","nonce":3745690918}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":3745690918},{"type":"v1","addr":"172.21.15.104:6803","nonce":3745690918}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":3745690918},{"type":"v1","addr":"172.21.15.104:6807","nonce":3745690918}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":3745690918},{"type":"v1","addr":"172.21.15.104:6805","nonce":3745690918}]},"public_addr":"172.21.15.104:6801/3745690918","cluster_addr":"172.21.15.104:6803/3745690918","heartbeat_back_addr":"172.21.15.104:6807/3745690918","heartbeat_front_addr":"172.21.15.104:6805/3745690918","state":["exists","up"]},{"osd":1,"uuid":"3ab7afdd-b980-418c-81f2-004e16de3dca","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":19438674},{"type":"v1","addr":"172.21.15.104:6810","nonce":19438674}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":19438674},{"type":"v1","addr":"172.21.15.104:6814","nonce":19438674}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":19438674},{"type":"v1","addr":"172.21.15.104:6823","nonce":19438674}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":19438674},{"type":"v1","addr":"172.21.15.104:6819","nonce":19438674}]},"public_addr":"172.21.15.104:6810/19438674","cluster_addr":"172.21.15.104:6814/19438674","heartbeat_back_addr":"172.21.15.104:6823/19438674","heartbeat_front_addr":"172.21.15.104:6819/19438674","state":["exists","up"]},{"osd":2,"uuid":"3e2651c8-ecae-4a9e-a053-27240a79eac2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":2352012626},{"type":"v1","addr":"172.21.15.104:6825","nonce":2352012626}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6826","nonce":2352012626},{"type":"v1","addr":"172.21.15.104:6827","nonce":2352012626}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":2352012626},{"type":"v1","addr":"172.21.15.104:6831","nonce":2352012626}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6828","nonce":2352012626},{"type":"v1","addr":"172.21.15.104:6829","nonce":2352012626}]},"public_addr":"172.21.15.104:6825/2352012626","cluster_addr":"172.21.15.104:6827/2352012626","heartbeat_back_addr":"172.21.15.104:6831/2352012626","heartbeat_front_addr":"172.21.15.104:6829/2352012626","state":["exists","up"]},{"osd":3,"uuid":"925c7e0a-38ef-47cd-b288-8bec8456fb17","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6809","nonce":3568347381},{"type":"v1","addr":"172.21.15.104:6811","nonce":3568347381}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6813","nonce":3568347381},{"type":"v1","addr":"172.21.15.104:6815","nonce":3568347381}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":3568347381},{"type":"v1","addr":"172.21.15.104:6822","nonce":3568347381}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6817","nonce":3568347381},{"type":"v1","addr":"172.21.15.104:6818","nonce":3568347381}]},"public_addr":"172.21.15.104:6811/3568347381","cluster_addr":"172.21.15.104:6815/3568347381","heartbeat_back_addr":"172.21.15.104:6822/3568347381","heartbeat_front_addr":"172.21.15.104:6818/3568347381","state":["exists","up"]},{"osd":4,"uuid":"4a8fac86-e595-4a0e-96e3-57edb3cc3f57","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6800","nonce":2107001077},{"type":"v1","addr":"172.21.15.171:6801","nonce":2107001077}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6802","nonce":2107001077},{"type":"v1","addr":"172.21.15.171:6803","nonce":2107001077}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6806","nonce":2107001077},{"type":"v1","addr":"172.21.15.171:6807","nonce":2107001077}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6804","nonce":2107001077},{"type":"v1","addr":"172.21.15.171:6805","nonce":2107001077}]},"public_addr":"172.21.15.171:6801/2107001077","cluster_addr":"172.21.15.171:6803/2107001077","heartbeat_back_addr":"172.21.15.171:6807/2107001077","heartbeat_front_addr":"172.21.15.171:6805/2107001077","state":["exists","up"]},{"osd":5,"uuid":"e48d50b0-beae-40ae-b15a-9a84dbfe88ab","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6808","nonce":2934826925},{"type":"v1","addr":"172.21.15.171:6810","nonce":2934826925}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6812","nonce":2934826925},{"type":"v1","addr":"172.21.15.171:6814","nonce":2934826925}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6819","nonce":2934826925},{"type":"v1","addr":"172.21.15.171:6821","nonce":2934826925}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6816","nonce":2934826925},{"type":"v1","addr":"172.21.15.171:6817","nonce":2934826925}]},"public_addr":"172.21.15.171:6810/2934826925","cluster_addr":"172.21.15.171:6814/2934826925","heartbeat_back_addr":"172.21.15.171:6821/2934826925","heartbeat_front_addr":"172.21.15.171:6817/2934826925","state":["exists","up"]},{"osd":6,"uuid":"5e7c3292-9936-4f35-b267-aed9d4e49323","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6824","nonce":436742919},{"type":"v1","addr":"172.21.15.171:6825","nonce":436742919}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6826","nonce":436742919},{"type":"v1","addr":"172.21.15.171:6827","nonce":436742919}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6830","nonce":436742919},{"type":"v1","addr":"172.21.15.171:6831","nonce":436742919}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6828","nonce":436742919},{"type":"v1","addr":"172.21.15.171:6829","nonce":436742919}]},"public_addr":"172.21.15.171:6825/436742919","cluster_addr":"172.21.15.171:6827/436742919","heartbeat_back_addr":"172.21.15.171:6831/436742919","heartbeat_front_addr":"172.21.15.171:6829/436742919","state":["exists","up"]},{"osd":7,"uuid":"339b8414-d7f0-4fb0-a446-efaf635cb7ff","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6809","nonce":2055631175},{"type":"v1","addr":"172.21.15.171:6811","nonce":2055631175}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6813","nonce":2055631175},{"type":"v1","addr":"172.21.15.171:6815","nonce":2055631175}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6822","nonce":2055631175},{"type":"v1","addr":"172.21.15.171:6823","nonce":2055631175}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6818","nonce":2055631175},{"type":"v1","addr":"172.21.15.171:6820","nonce":2055631175}]},"public_addr":"172.21.15.171:6811/2055631175","cluster_addr":"172.21.15.171:6815/2055631175","heartbeat_back_addr":"172.21.15.171:6823/2055631175","heartbeat_front_addr":"172.21.15.171:6820/2055631175","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T14:07:42.668881+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T14:07:42.708397+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T14:07:42.634191+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T14:07:42.733662+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T14:07:42.651541+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T14:07:42.783181+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T14:07:42.766310+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T14:07:42.670797+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-28T14:08:30.243 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:30.239+0000 7fa19ba53640 1 -- 172.21.15.104:0/875805234 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7fa174067ff0 msgr2=0x7fa17406a4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:30.243 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:30.239+0000 7fa19ba53640 1 --2- 172.21.15.104:0/875805234 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7fa174067ff0 0x7fa17406a4b0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fa184000c10 tx=0x7fa184009040 comp rx=0 tx=0).stop 2024-03-28T14:08:30.244 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:30.239+0000 7fa19ba53640 1 -- 172.21.15.104:0/875805234 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa194130db0 msgr2=0x7fa1941e0ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:30.244 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:30.239+0000 7fa19ba53640 1 --2- 172.21.15.104:0/875805234 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa194130db0 0x7fa1941e0ed0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7fa17c001750 tx=0x7fa180005840 comp rx=0 tx=0).stop 2024-03-28T14:08:30.244 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:30.239+0000 7fa19ba53640 1 -- 172.21.15.104:0/875805234 shutdown_connections 2024-03-28T14:08:30.244 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:30.239+0000 7fa19ba53640 1 --2- 172.21.15.104:0/875805234 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7fa174067ff0 0x7fa17406a4b0 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:30.244 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:30.239+0000 7fa19ba53640 1 --2- 172.21.15.104:0/875805234 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa194130db0 0x7fa1941e0ed0 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:30.245 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:30.239+0000 7fa19ba53640 1 --2- 172.21.15.104:0/875805234 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fa1940addc0 0x7fa194130870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:30.245 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:30.239+0000 7fa19ba53640 1 --2- 172.21.15.104:0/875805234 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fa180005220 0x7fa1940b8350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:30.245 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:30.239+0000 7fa19ba53640 1 -- 172.21.15.104:0/875805234 >> 172.21.15.104:0/875805234 conn(0x7fa1940b58a0 msgr2=0x7fa19413e3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T14:08:30.245 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:30.239+0000 7fa19ba53640 1 -- 172.21.15.104:0/875805234 shutdown_connections 2024-03-28T14:08:30.245 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:30.239+0000 7fa19ba53640 1 -- 172.21.15.104:0/875805234 wait complete. 2024-03-28T14:08:30.259 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-03-28T14:08:30.369 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:30.367+0000 7f5984ffc640 1 Processor -- start 2024-03-28T14:08:30.370 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:30.367+0000 7f5984ffc640 1 -- start start 2024-03-28T14:08:30.371 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:30.367+0000 7f5984ffc640 1 --2- >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f5980071be0 0x7f5980150ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:30.371 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:30.367+0000 7f5984ffc640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5980072030 con 0x7f59800b1e80 2024-03-28T14:08:30.371 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:30.367+0000 7f5984ffc640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5980072fd0 con 0x7f5980072bf0 2024-03-28T14:08:30.371 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:30.367+0000 7f5984ffc640 1 -- --> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] -- mon_getmap magic: 0 -- 0x7f5980073250 con 0x7f5980071be0 2024-03-28T14:08:30.371 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:30.367+0000 7f597dd9b640 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f59800b1e80 0x7f5980145ce0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:41622/0 (socket says 172.21.15.104:41622) 2024-03-28T14:08:30.371 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:30.367+0000 7f597dd9b640 1 -- 172.21.15.104:0/3206410428 learned_addr learned my addr 172.21.15.104:0/3206410428 (peer_addr_for_me v1:172.21.15.104:0/0) 2024-03-28T14:08:30.372 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:30.371+0000 7f597ed9d640 1 --2- 172.21.15.104:0/3206410428 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f5980071be0 0x7f5980150ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:30.372 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:30.371+0000 7f597d59a640 1 -- 172.21.15.104:0/3206410428 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2896740772 0 0) 0x7f5974001120 con 0x7f59800b1e80 2024-03-28T14:08:30.372 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:30.371+0000 7f597d59a640 1 -- 172.21.15.104:0/3206410428 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5968003530 con 0x7f59800b1e80 2024-03-28T14:08:30.373 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:30.371+0000 7f597d59a640 1 -- 172.21.15.104:0/3206410428 <== mon.0 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2151524296 0 0) 0x7f5968003530 con 0x7f59800b1e80 2024-03-28T14:08:30.374 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:30.371+0000 7f597d59a640 1 -- 172.21.15.104:0/3206410428 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f5980071be0 msgr2=0x7f5980150ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T14:08:30.374 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:30.371+0000 7f597d59a640 1 --2- 172.21.15.104:0/3206410428 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f5980071be0 0x7f5980150ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:30.374 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:30.371+0000 7f597d59a640 1 -- 172.21.15.104:0/3206410428 >> v1:172.21.15.171:6789/0 conn(0x7f5980072bf0 legacy=0x7f59800b12f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:30.374 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:30.371+0000 7f597d59a640 1 -- 172.21.15.104:0/3206410428 --> v1:172.21.15.104:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f59800734a0 con 0x7f59800b1e80 2024-03-28T14:08:30.374 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:30.371+0000 7f597d59a640 1 -- 172.21.15.104:0/3206410428 <== mon.0 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3745580849 0 0) 0x7f5974001850 con 0x7f59800b1e80 2024-03-28T14:08:30.374 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:30.371+0000 7f597d59a640 1 -- 172.21.15.104:0/3206410428 >> v1:172.21.15.104:6789/0 conn(0x7f59800b1e80 legacy=0x7f5980145ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:30.374 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:30.371+0000 7f597d59a640 1 --2- 172.21.15.104:0/3206410428 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f59680039f0 0x7f5968003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:30.374 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:30.371+0000 7f597d59a640 1 --2- 172.21.15.104:0/3206410428 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f5980071be0 0x7f5968004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:30.374 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:30.371+0000 7f597d59a640 1 --2- 172.21.15.104:0/3206410428 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f5968004d50 0x7f59680051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:30.374 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:30.371+0000 7f597d59a640 1 -- 172.21.15.104:0/3206410428 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f5980073250 con 0x7f59680039f0 2024-03-28T14:08:30.374 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:30.371+0000 7f597d59a640 1 -- 172.21.15.104:0/3206410428 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f596800bb90 con 0x7f5968004d50 2024-03-28T14:08:30.374 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:30.371+0000 7f597d59a640 1 -- 172.21.15.104:0/3206410428 --> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] -- mon_getmap magic: 0 -- 0x7f596800bd90 con 0x7f5980071be0 2024-03-28T14:08:30.374 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:30.371+0000 7f597e59c640 1 --2- 172.21.15.104:0/3206410428 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f59680039f0 0x7f5968003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:30.375 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:30.371+0000 7f597ed9d640 1 --2- 172.21.15.104:0/3206410428 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f5968004d50 0x7f59680051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:30.375 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:30.371+0000 7f597dd9b640 1 --2- 172.21.15.104:0/3206410428 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f5980071be0 0x7f5968004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:30.375 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:30.371+0000 7f597e59c640 1 -- 172.21.15.104:0/3206410428 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f5980071be0 msgr2=0x7f5968004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T14:08:30.375 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:30.371+0000 7f597e59c640 1 --2- 172.21.15.104:0/3206410428 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f5980071be0 0x7f5968004810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:30.375 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:30.371+0000 7f597e59c640 1 -- 172.21.15.104:0/3206410428 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f5968004d50 msgr2=0x7f59680051d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T14:08:30.375 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:30.371+0000 7f597e59c640 1 --2- 172.21.15.104:0/3206410428 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f5968004d50 0x7f59680051d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:30.375 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:30.371+0000 7f597e59c640 1 -- 172.21.15.104:0/3206410428 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5968003530 con 0x7f59680039f0 2024-03-28T14:08:30.375 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:30.371+0000 7f597dd9b640 1 --2- 172.21.15.104:0/3206410428 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f5980071be0 0x7f5968004810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T14:08:30.379 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:30.375+0000 7f597e59c640 1 --2- 172.21.15.104:0/3206410428 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f59680039f0 0x7f5968003e10 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f5970001780 tx=0x7f597001c420 comp rx=0 tx=0).ready entity=mon.0 client_cookie=693a8d1f61259dca server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:08:30.379 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:30.375+0000 7f597d59a640 1 -- 172.21.15.104:0/3206410428 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59700013a0 con 0x7f59680039f0 2024-03-28T14:08:30.379 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:30.375+0000 7f597d59a640 1 -- 172.21.15.104:0/3206410428 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f59700052f0 con 0x7f59680039f0 2024-03-28T14:08:30.379 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:30.375+0000 7f597d59a640 1 -- 172.21.15.104:0/3206410428 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5970003840 con 0x7f59680039f0 2024-03-28T14:08:30.379 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:30.375+0000 7f5984ffc640 1 -- 172.21.15.104:0/3206410428 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f59680039f0 msgr2=0x7f5968003e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:30.379 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:30.375+0000 7f5984ffc640 1 --2- 172.21.15.104:0/3206410428 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f59680039f0 0x7f5968003e10 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f5970001780 tx=0x7f597001c420 comp rx=0 tx=0).stop 2024-03-28T14:08:30.379 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:30.375+0000 7f597ed9d640 1 -- 172.21.15.104:0/3206410428 reap_dead start 2024-03-28T14:08:30.380 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:30.375+0000 7f5984ffc640 1 -- 172.21.15.104:0/3206410428 shutdown_connections 2024-03-28T14:08:30.380 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:30.375+0000 7f5984ffc640 1 -- 172.21.15.104:0/3206410428 >> 172.21.15.104:0/3206410428 conn(0x7f59800b7b10 msgr2=0x7f59800b7f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T14:08:30.380 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:30.375+0000 7f5984ffc640 1 -- 172.21.15.104:0/3206410428 shutdown_connections 2024-03-28T14:08:30.380 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:30.375+0000 7f5984ffc640 1 -- 172.21.15.104:0/3206410428 wait complete. 2024-03-28T14:08:30.380 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:30.379+0000 7f5984ffc640 1 Processor -- start 2024-03-28T14:08:30.381 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:30.379+0000 7f5984ffc640 1 -- start start 2024-03-28T14:08:30.383 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:30.379+0000 7f5984ffc640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f59680039f0 0x7f5980141ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:30.383 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:30.379+0000 7f5984ffc640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f5980146e60 0x7f59801ddb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:30.384 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:30.379+0000 7f5984ffc640 1 --2- >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f59800b5090 0x7f598013b230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:30.384 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:30.379+0000 7f5984ffc640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f5980146570 con 0x7f59680039f0 2024-03-28T14:08:30.384 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:30.379+0000 7f5984ffc640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f59801463f0 con 0x7f5980146e60 2024-03-28T14:08:30.384 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:30.379+0000 7f5984ffc640 1 -- --> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] -- mon_getmap magic: 0 -- 0x7f59801516b0 con 0x7f59800b5090 2024-03-28T14:08:30.384 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:30.379+0000 7f597e59c640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f59680039f0 0x7f5980141ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:30.384 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:30.379+0000 7f597e59c640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f59680039f0 0x7f5980141ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:59000/0 (socket says 172.21.15.104:59000) 2024-03-28T14:08:30.384 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:30.379+0000 7f597e59c640 1 -- 172.21.15.104:0/3159357605 learned_addr learned my addr 172.21.15.104:0/3159357605 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-03-28T14:08:30.385 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:30.379+0000 7f597ed9d640 1 --2- 172.21.15.104:0/3159357605 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f59800b5090 0x7f598013b230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:30.385 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:30.379+0000 7f597dd9b640 1 --2- 172.21.15.104:0/3159357605 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f5980146e60 0x7f59801ddb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:30.385 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:30.379+0000 7f597e59c640 1 -- 172.21.15.104:0/3159357605 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f59800b5090 msgr2=0x7f598013b230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:30.385 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:30.379+0000 7f597e59c640 1 --2- 172.21.15.104:0/3159357605 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f59800b5090 0x7f598013b230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:30.385 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:30.379+0000 7f597e59c640 1 -- 172.21.15.104:0/3159357605 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f5980146e60 msgr2=0x7f59801ddb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:30.385 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:30.379+0000 7f597e59c640 1 --2- 172.21.15.104:0/3159357605 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f5980146e60 0x7f59801ddb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:30.385 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:30.379+0000 7f597e59c640 1 -- 172.21.15.104:0/3159357605 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5980146990 con 0x7f59680039f0 2024-03-28T14:08:30.386 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:30.379+0000 7f597e59c640 1 --2- 172.21.15.104:0/3159357605 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f59680039f0 0x7f5980141ed0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f5970001780 tx=0x7f5968005840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:08:30.386 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:30.379+0000 7f596effd640 1 -- 172.21.15.104:0/3159357605 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5970007a40 con 0x7f59680039f0 2024-03-28T14:08:30.386 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:30.379+0000 7f596effd640 1 -- 172.21.15.104:0/3159357605 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f59700bb070 con 0x7f59680039f0 2024-03-28T14:08:30.386 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:30.379+0000 7f5984ffc640 1 -- 172.21.15.104:0/3159357605 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f59800b40d0 con 0x7f59680039f0 2024-03-28T14:08:30.386 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:30.379+0000 7f596effd640 1 -- 172.21.15.104:0/3159357605 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59700013a0 con 0x7f59680039f0 2024-03-28T14:08:30.386 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:30.379+0000 7f5984ffc640 1 -- 172.21.15.104:0/3159357605 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f59800b4370 con 0x7f59680039f0 2024-03-28T14:08:30.388 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:30.383+0000 7f5984ffc640 1 -- 172.21.15.104:0/3159357605 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f59800687c0 con 0x7f59680039f0 2024-03-28T14:08:30.388 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:30.383+0000 7f596effd640 1 -- 172.21.15.104:0/3159357605 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) ==== 87803+0+0 (secure 0 0 0) 0x7f59700b4040 con 0x7f59680039f0 2024-03-28T14:08:30.388 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:30.383+0000 7f596effd640 1 --2- 172.21.15.104:0/3159357605 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f5960067fc0 0x7f596006a480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:30.388 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:30.383+0000 7f596effd640 1 -- 172.21.15.104:0/3159357605 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f5970120360 con 0x7f59680039f0 2024-03-28T14:08:30.389 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:30.383+0000 7f597dd9b640 1 --2- 172.21.15.104:0/3159357605 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f5960067fc0 0x7f596006a480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:30.389 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:30.383+0000 7f597dd9b640 1 --2- 172.21.15.104:0/3159357605 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f5960067fc0 0x7f596006a480 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f5980144a70 tx=0x7f5974009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:08:30.389 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:30.387+0000 7f596effd640 1 -- 172.21.15.104:0/3159357605 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179117 (secure 0 0 0) 0x7f5970016130 con 0x7f59680039f0 2024-03-28T14:08:30.570 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:30.567+0000 7f5984ffc640 1 -- 172.21.15.104:0/3159357605 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7f5980147f70 con 0x7f59680039f0 2024-03-28T14:08:30.571 INFO:tasks.ceph.mon.a.smithi104.stderr:2024-03-28T14:08:30.567+0000 7f55f386d640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-28T14:08:30.574 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:30.571+0000 7f596effd640 1 -- 172.21.15.104:0/3159357605 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v8) ==== 161+0+0 (secure 0 0 0) 0x7f59700f0ba0 con 0x7f59680039f0 2024-03-28T14:08:30.574 INFO:teuthology.orchestra.run.smithi104.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-03-28T14:08:30.577 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:30.575+0000 7f5984ffc640 1 -- 172.21.15.104:0/3159357605 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f5960067fc0 msgr2=0x7f596006a480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:30.577 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:30.575+0000 7f5984ffc640 1 --2- 172.21.15.104:0/3159357605 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f5960067fc0 0x7f596006a480 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f5980144a70 tx=0x7f5974009040 comp rx=0 tx=0).stop 2024-03-28T14:08:30.577 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:30.575+0000 7f5984ffc640 1 -- 172.21.15.104:0/3159357605 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f59680039f0 msgr2=0x7f5980141ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:30.578 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:30.575+0000 7f5984ffc640 1 --2- 172.21.15.104:0/3159357605 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f59680039f0 0x7f5980141ed0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f5970001780 tx=0x7f5968005840 comp rx=0 tx=0).stop 2024-03-28T14:08:30.578 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:30.575+0000 7f5984ffc640 1 -- 172.21.15.104:0/3159357605 shutdown_connections 2024-03-28T14:08:30.578 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:30.575+0000 7f5984ffc640 1 --2- 172.21.15.104:0/3159357605 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f59800b5090 0x7f598013b230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:30.578 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:30.575+0000 7f5984ffc640 1 --2- 172.21.15.104:0/3159357605 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f5980146e60 0x7f59801ddb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:30.578 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:30.575+0000 7f5984ffc640 1 --2- 172.21.15.104:0/3159357605 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f5960067fc0 0x7f596006a480 secure :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f5980144a70 tx=0x7f5974009040 comp rx=0 tx=0).stop 2024-03-28T14:08:30.578 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:30.575+0000 7f5984ffc640 1 --2- 172.21.15.104:0/3159357605 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f59680039f0 0x7f5980141ed0 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:30.579 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:30.575+0000 7f5984ffc640 1 -- 172.21.15.104:0/3159357605 >> 172.21.15.104:0/3159357605 conn(0x7f59800b7b10 msgr2=0x7f5980144ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T14:08:30.579 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:30.575+0000 7f5984ffc640 1 -- 172.21.15.104:0/3159357605 shutdown_connections 2024-03-28T14:08:30.579 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:30.575+0000 7f5984ffc640 1 -- 172.21.15.104:0/3159357605 wait complete. 2024-03-28T14:08:30.579 INFO:tasks.ceph.mds.a.smithi104.stderr: -1191> 2024-03-28T14:08:02.771+0000 7fc78ff0f640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-28T14:08:30.595 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-03-28T14:08:30.603 INFO:tasks.ceph.mds.a.smithi104.stdout:starting mds.a at 2024-03-28T14:08:30.707 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:30.703+0000 7f08dc0c8640 1 Processor -- start 2024-03-28T14:08:30.707 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:30.703+0000 7f08dc0c8640 1 -- start start 2024-03-28T14:08:30.709 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:30.707+0000 7f08dc0c8640 1 --2- >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f08d40bd9e0 0x7f08d414af30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:30.709 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:30.707+0000 7f08dc0c8640 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f08d414b470 0x7f08d41413f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:30.709 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:30.707+0000 7f08dc0c8640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f08d40b5050 con 0x7f08d4141930 2024-03-28T14:08:30.709 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:30.707+0000 7f08dc0c8640 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 -- 0x7f08d40b70b0 con 0x7f08d414b470 2024-03-28T14:08:30.710 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:30.707+0000 7f08dc0c8640 1 -- --> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] -- mon_getmap magic: 0 -- 0x7f08d414b8d0 con 0x7f08d40bd9e0 2024-03-28T14:08:30.710 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:30.707+0000 7f08da665640 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f08d4141930 0x7f08d40b79f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:41652/0 (socket says 172.21.15.104:41652) 2024-03-28T14:08:30.710 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:30.707+0000 7f08d9663640 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f08d414b470 0x7f08d41413f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:30.710 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:30.707+0000 7f08da665640 1 -- 172.21.15.104:0/449446685 learned_addr learned my addr 172.21.15.104:0/449446685 (peer_addr_for_me v1:172.21.15.104:0/0) 2024-03-28T14:08:30.710 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:30.707+0000 7f08d9e64640 1 --2- 172.21.15.104:0/449446685 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f08d40bd9e0 0x7f08d414af30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:30.710 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:30.707+0000 7f08d9663640 1 -- 172.21.15.104:0/449446685 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f08d40bd9e0 msgr2=0x7f08d414af30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T14:08:30.710 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:30.707+0000 7f08d9663640 1 --2- 172.21.15.104:0/449446685 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f08d40bd9e0 0x7f08d414af30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:30.710 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:30.707+0000 7f08d9663640 1 -- 172.21.15.104:0/449446685 >> v1:172.21.15.104:6789/0 conn(0x7f08d4141930 legacy=0x7f08d40b79f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:30.710 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:30.707+0000 7f08d9663640 1 -- 172.21.15.104:0/449446685 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f08d40bddc0 con 0x7f08d414b470 2024-03-28T14:08:30.714 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:30.707+0000 7f08d9663640 1 --2- 172.21.15.104:0/449446685 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f08d414b470 0x7f08d41413f0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f08c40097f0 tx=0x7f08c401c360 comp rx=0 tx=0).ready entity=mon.0 client_cookie=78e880b8ed15a4be server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:08:30.715 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:30.707+0000 7f08d8e62640 1 -- 172.21.15.104:0/449446685 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08c40b0070 con 0x7f08d414b470 2024-03-28T14:08:30.715 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:30.707+0000 7f08d8e62640 1 -- 172.21.15.104:0/449446685 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f08c40ab600 con 0x7f08d414b470 2024-03-28T14:08:30.715 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:30.707+0000 7f08d8e62640 1 -- 172.21.15.104:0/449446685 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08c40ab9d0 con 0x7f08d414b470 2024-03-28T14:08:30.715 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:30.707+0000 7f08dc0c8640 1 -- 172.21.15.104:0/449446685 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f08d414b470 msgr2=0x7f08d41413f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:30.715 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:30.707+0000 7f08dc0c8640 1 --2- 172.21.15.104:0/449446685 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f08d414b470 0x7f08d41413f0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f08c40097f0 tx=0x7f08c401c360 comp rx=0 tx=0).stop 2024-03-28T14:08:30.715 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:30.707+0000 7f08dc0c8640 1 -- 172.21.15.104:0/449446685 shutdown_connections 2024-03-28T14:08:30.715 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:30.707+0000 7f08dc0c8640 1 --2- 172.21.15.104:0/449446685 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f08d414b470 0x7f08d41413f0 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:30.715 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:30.707+0000 7f08dc0c8640 1 --2- 172.21.15.104:0/449446685 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f08d40bd9e0 0x7f08d414af30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:30.715 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:30.707+0000 7f08dc0c8640 1 -- 172.21.15.104:0/449446685 >> 172.21.15.104:0/449446685 conn(0x7f08d4073530 msgr2=0x7f08d40bd070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T14:08:30.715 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:30.707+0000 7f08dc0c8640 1 -- 172.21.15.104:0/449446685 shutdown_connections 2024-03-28T14:08:30.715 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:30.707+0000 7f08dc0c8640 1 -- 172.21.15.104:0/449446685 wait complete. 2024-03-28T14:08:30.715 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:30.711+0000 7f08dc0c8640 1 Processor -- start 2024-03-28T14:08:30.715 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:30.711+0000 7f08dc0c8640 1 -- start start 2024-03-28T14:08:30.716 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:30.711+0000 7f08dc0c8640 1 --2- >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f08d40bd9e0 0x7f08d41406d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:30.716 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:30.711+0000 7f08dc0c8640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f08d4141930 0x7f08d41e39c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:30.716 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:30.711+0000 7f08dc0c8640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f08d41e3f00 0x7f08d413b260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:30.716 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:30.711+0000 7f08dc0c8640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f08d40b6140 con 0x7f08d41e3f00 2024-03-28T14:08:30.716 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:30.711+0000 7f08dc0c8640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f08d40b5940 con 0x7f08d4141930 2024-03-28T14:08:30.716 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:30.711+0000 7f08dc0c8640 1 -- --> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] -- mon_getmap magic: 0 -- 0x7f08d40b62c0 con 0x7f08d40bd9e0 2024-03-28T14:08:30.716 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:30.711+0000 7f08da665640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f08d41e3f00 0x7f08d413b260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:30.716 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:30.711+0000 7f08da665640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f08d41e3f00 0x7f08d413b260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:59028/0 (socket says 172.21.15.104:59028) 2024-03-28T14:08:30.716 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:30.711+0000 7f08da665640 1 -- 172.21.15.104:0/3163907604 learned_addr learned my addr 172.21.15.104:0/3163907604 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-03-28T14:08:30.716 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:30.711+0000 7f08d9e64640 1 --2- 172.21.15.104:0/3163907604 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f08d40bd9e0 0x7f08d41406d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:30.716 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:30.711+0000 7f08da665640 1 -- 172.21.15.104:0/3163907604 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f08d40bd9e0 msgr2=0x7f08d41406d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:30.716 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:30.711+0000 7f08da665640 1 --2- 172.21.15.104:0/3163907604 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f08d40bd9e0 0x7f08d41406d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:30.717 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:30.711+0000 7f08da665640 1 -- 172.21.15.104:0/3163907604 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f08d4141930 msgr2=0x7f08d41e39c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:30.717 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:30.711+0000 7f08d9663640 1 --2- 172.21.15.104:0/3163907604 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f08d4141930 0x7f08d41e39c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:30.717 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:30.711+0000 7f08da665640 1 --2- 172.21.15.104:0/3163907604 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f08d4141930 0x7f08d41e39c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:30.717 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:30.711+0000 7f08da665640 1 -- 172.21.15.104:0/3163907604 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f08d414bad0 con 0x7f08d41e3f00 2024-03-28T14:08:30.717 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:30.711+0000 7f08da665640 1 --2- 172.21.15.104:0/3163907604 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f08d41e3f00 0x7f08d413b260 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f08cc00cab0 tx=0x7f08cc00cf70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:08:30.717 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:30.711+0000 7f08ca7fc640 1 -- 172.21.15.104:0/3163907604 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08cc017070 con 0x7f08d41e3f00 2024-03-28T14:08:30.717 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:30.711+0000 7f08ca7fc640 1 -- 172.21.15.104:0/3163907604 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f08cc00ac30 con 0x7f08d41e3f00 2024-03-28T14:08:30.717 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:30.711+0000 7f08dc0c8640 1 -- 172.21.15.104:0/3163907604 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f08d413ee30 con 0x7f08d41e3f00 2024-03-28T14:08:30.717 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:30.711+0000 7f08ca7fc640 1 -- 172.21.15.104:0/3163907604 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08cc012400 con 0x7f08d41e3f00 2024-03-28T14:08:30.717 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:30.711+0000 7f08dc0c8640 1 -- 172.21.15.104:0/3163907604 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f08d4143a50 con 0x7f08d41e3f00 2024-03-28T14:08:30.717 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:30.711+0000 7f08dc0c8640 1 -- 172.21.15.104:0/3163907604 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f08d40687c0 con 0x7f08d41e3f00 2024-03-28T14:08:30.718 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:30.711+0000 7f08ca7fc640 1 -- 172.21.15.104:0/3163907604 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) ==== 87803+0+0 (secure 0 0 0) 0x7f08cc01a070 con 0x7f08d41e3f00 2024-03-28T14:08:30.719 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:30.711+0000 7f08ca7fc640 1 --2- 172.21.15.104:0/3163907604 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f08b8067fc0 0x7f08b806a480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:30.719 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:30.711+0000 7f08ca7fc640 1 -- 172.21.15.104:0/3163907604 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(22..22 src has 1..22) ==== 5795+0+0 (secure 0 0 0) 0x7f08cc086090 con 0x7f08d41e3f00 2024-03-28T14:08:30.719 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:30.715+0000 7f08d9e64640 1 --2- 172.21.15.104:0/3163907604 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f08b8067fc0 0x7f08b806a480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:30.719 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:30.715+0000 7f08d9e64640 1 --2- 172.21.15.104:0/3163907604 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f08b8067fc0 0x7f08b806a480 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f08d4144770 tx=0x7f08d0008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:08:30.720 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:30.719+0000 7f08ca7fc640 1 -- 172.21.15.104:0/3163907604 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179117 (secure 0 0 0) 0x7f08cc056220 con 0x7f08d41e3f00 2024-03-28T14:08:30.908 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:30.907+0000 7f08dc0c8640 1 -- 172.21.15.104:0/3163907604 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) -- 0x7f08d40b6af0 con 0x7f08d41e3f00 2024-03-28T14:08:31.576 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:31.574+0000 7f08ca7fc640 1 -- 172.21.15.104:0/3163907604 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v9) ==== 104+0+0 (secure 0 0 0) 0x7f08cc05a520 con 0x7f08d41e3f00 2024-03-28T14:08:31.579 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:31.574+0000 7f08dc0c8640 1 -- 172.21.15.104:0/3163907604 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f08b8067fc0 msgr2=0x7f08b806a480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:31.580 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:31.574+0000 7f08dc0c8640 1 --2- 172.21.15.104:0/3163907604 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f08b8067fc0 0x7f08b806a480 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f08d4144770 tx=0x7f08d0008040 comp rx=0 tx=0).stop 2024-03-28T14:08:31.580 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:31.574+0000 7f08dc0c8640 1 -- 172.21.15.104:0/3163907604 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f08d41e3f00 msgr2=0x7f08d413b260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:31.580 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:31.574+0000 7f08dc0c8640 1 --2- 172.21.15.104:0/3163907604 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f08d41e3f00 0x7f08d413b260 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f08cc00cab0 tx=0x7f08cc00cf70 comp rx=0 tx=0).stop 2024-03-28T14:08:31.580 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:31.574+0000 7f08dc0c8640 1 -- 172.21.15.104:0/3163907604 shutdown_connections 2024-03-28T14:08:31.580 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:31.574+0000 7f08dc0c8640 1 --2- 172.21.15.104:0/3163907604 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f08b8067fc0 0x7f08b806a480 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:31.580 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:31.574+0000 7f08dc0c8640 1 --2- 172.21.15.104:0/3163907604 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f08d41e3f00 0x7f08d413b260 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:31.580 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:31.574+0000 7f08dc0c8640 1 --2- 172.21.15.104:0/3163907604 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f08d4141930 0x7f08d41e39c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:31.580 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:31.574+0000 7f08dc0c8640 1 --2- 172.21.15.104:0/3163907604 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f08d40bd9e0 0x7f08d41406d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:31.580 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:31.574+0000 7f08dc0c8640 1 -- 172.21.15.104:0/3163907604 >> 172.21.15.104:0/3163907604 conn(0x7f08d4073530 msgr2=0x7f08d4149be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T14:08:31.580 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:31.578+0000 7f08dc0c8640 1 -- 172.21.15.104:0/3163907604 shutdown_connections 2024-03-28T14:08:31.580 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:31.578+0000 7f08dc0c8640 1 -- 172.21.15.104:0/3163907604 wait complete. 2024-03-28T14:08:31.600 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-03-28T14:08:31.773 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:31.770+0000 7f4c38785640 1 Processor -- start 2024-03-28T14:08:31.773 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:31.770+0000 7f4c38785640 1 -- start start 2024-03-28T14:08:31.774 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:31.770+0000 7f4c38785640 1 --2- >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f4c300b4e00 0x7f4c300b7a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:31.774 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:31.770+0000 7f4c38785640 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f4c30141500 0x7f4c30141950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:31.774 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:31.770+0000 7f4c38785640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4c300b51e0 con 0x7f4c3014b840 2024-03-28T14:08:31.774 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:31.770+0000 7f4c38785640 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 -- 0x7f4c30141e90 con 0x7f4c30141500 2024-03-28T14:08:31.775 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:31.770+0000 7f4c38785640 1 -- --> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] -- mon_getmap magic: 0 -- 0x7f4c3014ad60 con 0x7f4c300b4e00 2024-03-28T14:08:31.775 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:31.770+0000 7f4c36d22640 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f4c30141500 0x7f4c30141950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:31.775 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:31.770+0000 7f4c36d22640 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f4c30141500 0x7f4c30141950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:59032/0 (socket says 172.21.15.104:59032) 2024-03-28T14:08:31.775 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:31.770+0000 7f4c36d22640 1 -- 172.21.15.104:0/1468371599 learned_addr learned my addr 172.21.15.104:0/1468371599 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-03-28T14:08:31.775 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:31.770+0000 7f4c36521640 1 --2- 172.21.15.104:0/1468371599 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f4c300b4e00 0x7f4c300b7a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:31.775 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:31.770+0000 7f4c36d22640 1 -- 172.21.15.104:0/1468371599 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f4c300b4e00 msgr2=0x7f4c300b7a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T14:08:31.775 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:31.770+0000 7f4c36d22640 1 --2- 172.21.15.104:0/1468371599 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f4c300b4e00 0x7f4c300b7a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:31.775 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:31.770+0000 7f4c36d22640 1 -- 172.21.15.104:0/1468371599 >> v1:172.21.15.171:6789/0 conn(0x7f4c3014b840 legacy=0x7f4c300b5c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T14:08:31.775 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:31.770+0000 7f4c36d22640 1 -- 172.21.15.104:0/1468371599 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4c3014b040 con 0x7f4c30141500 2024-03-28T14:08:31.777 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:31.774+0000 7f4c36d22640 1 --2- 172.21.15.104:0/1468371599 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4c30141500 0x7f4c30141950 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f4c28009e50 tx=0x7f4c2801c960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8710ea2032b46356 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:08:31.777 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:31.774+0000 7f4c3551f640 1 -- 172.21.15.104:0/1468371599 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c280a56f0 con 0x7f4c30141500 2024-03-28T14:08:31.777 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:31.774+0000 7f4c3551f640 1 -- 172.21.15.104:0/1468371599 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4c280a5890 con 0x7f4c30141500 2024-03-28T14:08:31.777 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:31.774+0000 7f4c3551f640 1 -- 172.21.15.104:0/1468371599 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c280a5c60 con 0x7f4c30141500 2024-03-28T14:08:31.778 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:31.774+0000 7f4c38785640 1 -- 172.21.15.104:0/1468371599 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4c30141500 msgr2=0x7f4c30141950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:31.778 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:31.774+0000 7f4c38785640 1 --2- 172.21.15.104:0/1468371599 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4c30141500 0x7f4c30141950 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f4c28009e50 tx=0x7f4c2801c960 comp rx=0 tx=0).stop 2024-03-28T14:08:31.778 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:31.774+0000 7f4c38785640 1 -- 172.21.15.104:0/1468371599 shutdown_connections 2024-03-28T14:08:31.778 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:31.774+0000 7f4c38785640 1 --2- 172.21.15.104:0/1468371599 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4c30141500 0x7f4c30141950 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:31.778 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:31.774+0000 7f4c38785640 1 --2- 172.21.15.104:0/1468371599 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f4c300b4e00 0x7f4c300b7a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:31.779 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:31.774+0000 7f4c38785640 1 -- 172.21.15.104:0/1468371599 >> 172.21.15.104:0/1468371599 conn(0x7f4c300b9b40 msgr2=0x7f4c300b9f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T14:08:31.779 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:31.774+0000 7f4c38785640 1 -- 172.21.15.104:0/1468371599 shutdown_connections 2024-03-28T14:08:31.779 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:31.774+0000 7f4c38785640 1 -- 172.21.15.104:0/1468371599 wait complete. 2024-03-28T14:08:31.781 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:31.774+0000 7f4c38785640 1 Processor -- start 2024-03-28T14:08:31.781 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:31.774+0000 7f4c38785640 1 -- start start 2024-03-28T14:08:31.782 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:31.774+0000 7f4c38785640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f4c300b4e00 0x7f4c30147d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:31.782 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:31.774+0000 7f4c38785640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4c30141500 0x7f4c3013a9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:31.782 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:31.774+0000 7f4c38785640 1 --2- >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f4c3014b840 0x7f4c30136320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:31.782 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:31.774+0000 7f4c38785640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f4c301549f0 con 0x7f4c30141500 2024-03-28T14:08:31.782 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:31.774+0000 7f4c38785640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f4c30154870 con 0x7f4c300b4e00 2024-03-28T14:08:31.782 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:31.774+0000 7f4c38785640 1 -- --> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] -- mon_getmap magic: 0 -- 0x7f4c30154b70 con 0x7f4c3014b840 2024-03-28T14:08:31.782 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:31.778+0000 7f4c35d20640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4c30141500 0x7f4c3013a9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:31.782 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:31.778+0000 7f4c35d20640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4c30141500 0x7f4c3013a9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:59046/0 (socket says 172.21.15.104:59046) 2024-03-28T14:08:31.782 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:31.778+0000 7f4c35d20640 1 -- 172.21.15.104:0/3502114617 learned_addr learned my addr 172.21.15.104:0/3502114617 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-03-28T14:08:31.783 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:31.778+0000 7f4c36521640 1 --2- 172.21.15.104:0/3502114617 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f4c300b4e00 0x7f4c30147d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:31.783 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:31.778+0000 7f4c36d22640 1 --2- 172.21.15.104:0/3502114617 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f4c3014b840 0x7f4c30136320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:31.783 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:31.778+0000 7f4c35d20640 1 -- 172.21.15.104:0/3502114617 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f4c3014b840 msgr2=0x7f4c30136320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:31.783 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:31.778+0000 7f4c35d20640 1 --2- 172.21.15.104:0/3502114617 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f4c3014b840 0x7f4c30136320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:31.783 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:31.778+0000 7f4c35d20640 1 -- 172.21.15.104:0/3502114617 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f4c300b4e00 msgr2=0x7f4c30147d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:31.783 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:31.778+0000 7f4c35d20640 1 --2- 172.21.15.104:0/3502114617 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f4c300b4e00 0x7f4c30147d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:31.783 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:31.778+0000 7f4c35d20640 1 -- 172.21.15.104:0/3502114617 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4c3013c150 con 0x7f4c30141500 2024-03-28T14:08:31.783 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:31.778+0000 7f4c36521640 1 --2- 172.21.15.104:0/3502114617 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f4c300b4e00 0x7f4c30147d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T14:08:31.783 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:31.778+0000 7f4c35d20640 1 --2- 172.21.15.104:0/3502114617 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4c30141500 0x7f4c3013a9f0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f4c2000b410 tx=0x7f4c2000b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:08:31.783 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:31.778+0000 7f4c26ffd640 1 -- 172.21.15.104:0/3502114617 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c20015030 con 0x7f4c30141500 2024-03-28T14:08:31.783 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:31.778+0000 7f4c26ffd640 1 -- 172.21.15.104:0/3502114617 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4c20004020 con 0x7f4c30141500 2024-03-28T14:08:31.784 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:31.778+0000 7f4c38785640 1 -- 172.21.15.104:0/3502114617 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4c30143ee0 con 0x7f4c30141500 2024-03-28T14:08:31.784 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:31.778+0000 7f4c26ffd640 1 -- 172.21.15.104:0/3502114617 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c20004410 con 0x7f4c30141500 2024-03-28T14:08:31.784 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:31.778+0000 7f4c38785640 1 -- 172.21.15.104:0/3502114617 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4c301431a0 con 0x7f4c30141500 2024-03-28T14:08:31.784 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:31.778+0000 7f4c38785640 1 -- 172.21.15.104:0/3502114617 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4c3002cc50 con 0x7f4c30141500 2024-03-28T14:08:31.784 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:31.778+0000 7f4c26ffd640 1 -- 172.21.15.104:0/3502114617 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) ==== 87803+0+0 (secure 0 0 0) 0x7f4c20018070 con 0x7f4c30141500 2024-03-28T14:08:31.785 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:31.778+0000 7f4c26ffd640 1 --2- 172.21.15.104:0/3502114617 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f4c14067fc0 0x7f4c1406a480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:31.785 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:31.778+0000 7f4c26ffd640 1 -- 172.21.15.104:0/3502114617 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(22..22 src has 1..22) ==== 5795+0+0 (secure 0 0 0) 0x7f4c20086140 con 0x7f4c30141500 2024-03-28T14:08:31.785 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:31.778+0000 7f4c36521640 1 --2- 172.21.15.104:0/3502114617 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f4c14067fc0 0x7f4c1406a480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:31.785 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:31.778+0000 7f4c36521640 1 --2- 172.21.15.104:0/3502114617 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f4c14067fc0 0x7f4c1406a480 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f4c2c009840 tx=0x7f4c2c008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:08:31.786 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:31.782+0000 7f4c26ffd640 1 -- 172.21.15.104:0/3502114617 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179117 (secure 0 0 0) 0x7f4c20056880 con 0x7f4c30141500 2024-03-28T14:08:31.950 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:31.946+0000 7f4c38785640 1 -- 172.21.15.104:0/3502114617 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) -- 0x7f4c300bca20 con 0x7f4c30141500 2024-03-28T14:08:32.860 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:32.858+0000 7f4c26ffd640 1 -- 172.21.15.104:0/3502114617 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v23) ==== 181+0+0 (secure 0 0 0) 0x7f4c20056180 con 0x7f4c30141500 2024-03-28T14:08:32.860 INFO:teuthology.orchestra.run.smithi104.stderr:pool 'cephfs_metadata' removed 2024-03-28T14:08:32.865 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:32.862+0000 7f4c38785640 1 -- 172.21.15.104:0/3502114617 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f4c14067fc0 msgr2=0x7f4c1406a480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:32.865 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:32.862+0000 7f4c38785640 1 --2- 172.21.15.104:0/3502114617 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f4c14067fc0 0x7f4c1406a480 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f4c2c009840 tx=0x7f4c2c008040 comp rx=0 tx=0).stop 2024-03-28T14:08:32.865 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:32.862+0000 7f4c38785640 1 -- 172.21.15.104:0/3502114617 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4c30141500 msgr2=0x7f4c3013a9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:32.865 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:32.862+0000 7f4c38785640 1 --2- 172.21.15.104:0/3502114617 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4c30141500 0x7f4c3013a9f0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f4c2000b410 tx=0x7f4c2000b8d0 comp rx=0 tx=0).stop 2024-03-28T14:08:32.865 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:32.862+0000 7f4c38785640 1 -- 172.21.15.104:0/3502114617 shutdown_connections 2024-03-28T14:08:32.865 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:32.862+0000 7f4c38785640 1 --2- 172.21.15.104:0/3502114617 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f4c3014b840 0x7f4c30136320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:32.865 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:32.862+0000 7f4c38785640 1 --2- 172.21.15.104:0/3502114617 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f4c14067fc0 0x7f4c1406a480 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:32.865 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:32.862+0000 7f4c38785640 1 --2- 172.21.15.104:0/3502114617 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4c30141500 0x7f4c3013a9f0 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:32.865 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:32.862+0000 7f4c38785640 1 --2- 172.21.15.104:0/3502114617 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f4c300b4e00 0x7f4c30147d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:32.866 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:32.862+0000 7f4c38785640 1 -- 172.21.15.104:0/3502114617 >> 172.21.15.104:0/3502114617 conn(0x7f4c300b9b40 msgr2=0x7f4c300bc250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T14:08:32.866 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:32.862+0000 7f4c38785640 1 -- 172.21.15.104:0/3502114617 shutdown_connections 2024-03-28T14:08:32.866 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:32.862+0000 7f4c38785640 1 -- 172.21.15.104:0/3502114617 wait complete. 2024-03-28T14:08:32.886 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-03-28T14:08:33.017 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:33.014+0000 7fd68ba73640 1 Processor -- start 2024-03-28T14:08:33.017 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:33.014+0000 7fd68ba73640 1 -- start start 2024-03-28T14:08:33.018 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:33.014+0000 7fd68ba73640 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fd6840b55f0 0x7fd6840b00d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:33.018 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:33.014+0000 7fd68ba73640 1 --2- >> v2:172.21.15.171:3300/0 conn(0x7fd6840b62a0 0x7fd6840b59d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:33.018 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:33.014+0000 7fd68ba73640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd6840aefe0 con 0x7fd68406b5d0 2024-03-28T14:08:33.018 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:33.014+0000 7fd68ba73640 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 -- 0x7fd6840af2c0 con 0x7fd6840b55f0 2024-03-28T14:08:33.018 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:33.014+0000 7fd68ba73640 1 -- --> v2:172.21.15.171:3300/0 -- mon_getmap magic: 0 -- 0x7fd68406ba60 con 0x7fd6840b62a0 2024-03-28T14:08:33.018 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:33.014+0000 7fd68980f640 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fd6840b55f0 0x7fd6840b00d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:33.019 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:33.014+0000 7fd68980f640 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fd6840b55f0 0x7fd6840b00d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:59050/0 (socket says 172.21.15.104:59050) 2024-03-28T14:08:33.019 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:33.014+0000 7fd68980f640 1 -- 172.21.15.104:0/2241939234 learned_addr learned my addr 172.21.15.104:0/2241939234 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-03-28T14:08:33.019 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:33.014+0000 7fd68900e640 1 --2- 172.21.15.104:0/2241939234 >> v2:172.21.15.171:3300/0 conn(0x7fd6840b62a0 0x7fd6840b59d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:33.019 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:33.014+0000 7fd68980f640 1 -- 172.21.15.104:0/2241939234 >> v2:172.21.15.171:3300/0 conn(0x7fd6840b62a0 msgr2=0x7fd6840b59d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T14:08:33.019 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:33.014+0000 7fd68980f640 1 --2- 172.21.15.104:0/2241939234 >> v2:172.21.15.171:3300/0 conn(0x7fd6840b62a0 0x7fd6840b59d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:33.019 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:33.014+0000 7fd68980f640 1 -- 172.21.15.104:0/2241939234 >> v1:172.21.15.171:6789/0 conn(0x7fd68406b5d0 legacy=0x7fd68409ee20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:33.020 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:33.014+0000 7fd68980f640 1 -- 172.21.15.104:0/2241939234 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd68409f530 con 0x7fd6840b55f0 2024-03-28T14:08:33.020 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:33.014+0000 7fd68900e640 1 --2- 172.21.15.104:0/2241939234 >> v2:172.21.15.171:3300/0 conn(0x7fd6840b62a0 0x7fd6840b59d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T14:08:33.021 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:33.018+0000 7fd68980f640 1 --2- 172.21.15.104:0/2241939234 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd6840b55f0 0x7fd6840b00d0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7fd6840b78c0 tx=0x7fd68001c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eca6b8e5b8d5384 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:08:33.021 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:33.018+0000 7fd68880d640 1 -- 172.21.15.104:0/2241939234 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6800b1070 con 0x7fd6840b55f0 2024-03-28T14:08:33.021 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:33.018+0000 7fd68880d640 1 -- 172.21.15.104:0/2241939234 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd6800ac600 con 0x7fd6840b55f0 2024-03-28T14:08:33.021 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:33.018+0000 7fd68880d640 1 -- 172.21.15.104:0/2241939234 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6800ac9d0 con 0x7fd6840b55f0 2024-03-28T14:08:33.022 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:33.018+0000 7fd68ba73640 1 -- 172.21.15.104:0/2241939234 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd6840b55f0 msgr2=0x7fd6840b00d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:33.022 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:33.018+0000 7fd68ba73640 1 --2- 172.21.15.104:0/2241939234 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd6840b55f0 0x7fd6840b00d0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7fd6840b78c0 tx=0x7fd68001c490 comp rx=0 tx=0).stop 2024-03-28T14:08:33.022 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:33.018+0000 7fd68ba73640 1 -- 172.21.15.104:0/2241939234 shutdown_connections 2024-03-28T14:08:33.022 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:33.018+0000 7fd68ba73640 1 --2- 172.21.15.104:0/2241939234 >> v2:172.21.15.171:3300/0 conn(0x7fd6840b62a0 0x7fd6840b59d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:33.022 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:33.018+0000 7fd68ba73640 1 --2- 172.21.15.104:0/2241939234 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd6840b55f0 0x7fd6840b00d0 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:33.023 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:33.018+0000 7fd68ba73640 1 -- 172.21.15.104:0/2241939234 >> 172.21.15.104:0/2241939234 conn(0x7fd6840b6990 msgr2=0x7fd6840b6da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T14:08:33.023 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:33.018+0000 7fd68ba73640 1 -- 172.21.15.104:0/2241939234 shutdown_connections 2024-03-28T14:08:33.023 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:33.018+0000 7fd68ba73640 1 -- 172.21.15.104:0/2241939234 wait complete. 2024-03-28T14:08:33.025 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:33.018+0000 7fd68ba73640 1 Processor -- start 2024-03-28T14:08:33.025 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:33.018+0000 7fd68ba73640 1 -- start start 2024-03-28T14:08:33.025 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:33.022+0000 7fd68ba73640 1 --2- >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fd68406b5d0 0x7fd684145c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:33.026 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:33.022+0000 7fd68ba73640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd6840b55f0 0x7fd68413fc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:33.026 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:33.022+0000 7fd68ba73640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fd6840b62a0 0x7fd6841406a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:33.026 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:33.022+0000 7fd68ba73640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7fd684154fd0 con 0x7fd6840b55f0 2024-03-28T14:08:33.026 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:33.022+0000 7fd68ba73640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7fd684154e50 con 0x7fd6840b62a0 2024-03-28T14:08:33.026 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:33.022+0000 7fd68ba73640 1 -- --> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] -- mon_getmap magic: 0 -- 0x7fd684155150 con 0x7fd68406b5d0 2024-03-28T14:08:33.026 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:33.022+0000 7fd68900e640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd6840b55f0 0x7fd68413fc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:33.026 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:33.022+0000 7fd68900e640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd6840b55f0 0x7fd68413fc10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:59066/0 (socket says 172.21.15.104:59066) 2024-03-28T14:08:33.026 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:33.022+0000 7fd68900e640 1 -- 172.21.15.104:0/609230793 learned_addr learned my addr 172.21.15.104:0/609230793 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-03-28T14:08:33.026 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:33.022+0000 7fd68980f640 1 --2- 172.21.15.104:0/609230793 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fd68406b5d0 0x7fd684145c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:33.026 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:33.022+0000 7fd68a010640 1 --2- 172.21.15.104:0/609230793 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fd6840b62a0 0x7fd6841406a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:33.026 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:33.022+0000 7fd68900e640 1 -- 172.21.15.104:0/609230793 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fd68406b5d0 msgr2=0x7fd684145c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:33.026 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:33.022+0000 7fd68900e640 1 --2- 172.21.15.104:0/609230793 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fd68406b5d0 0x7fd684145c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:33.026 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:33.022+0000 7fd68900e640 1 -- 172.21.15.104:0/609230793 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fd6840b62a0 msgr2=0x7fd6841406a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:33.027 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:33.022+0000 7fd68900e640 1 --2- 172.21.15.104:0/609230793 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fd6840b62a0 0x7fd6841406a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:33.027 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:33.022+0000 7fd68900e640 1 -- 172.21.15.104:0/609230793 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd684147c00 con 0x7fd6840b55f0 2024-03-28T14:08:33.027 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:33.022+0000 7fd68900e640 1 --2- 172.21.15.104:0/609230793 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd6840b55f0 0x7fd68413fc10 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7fd67400d800 tx=0x7fd67400dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:08:33.027 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:33.022+0000 7fd67a7fc640 1 -- 172.21.15.104:0/609230793 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd67400b6c0 con 0x7fd6840b55f0 2024-03-28T14:08:33.027 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:33.022+0000 7fd67a7fc640 1 -- 172.21.15.104:0/609230793 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd674019070 con 0x7fd6840b55f0 2024-03-28T14:08:33.027 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:33.022+0000 7fd68ba73640 1 -- 172.21.15.104:0/609230793 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd6841e0900 con 0x7fd6840b55f0 2024-03-28T14:08:33.027 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:33.022+0000 7fd67a7fc640 1 -- 172.21.15.104:0/609230793 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd67400baa0 con 0x7fd6840b55f0 2024-03-28T14:08:33.027 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:33.022+0000 7fd68ba73640 1 -- 172.21.15.104:0/609230793 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd684147160 con 0x7fd6840b55f0 2024-03-28T14:08:33.027 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:33.022+0000 7fd68ba73640 1 -- 172.21.15.104:0/609230793 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd6840687c0 con 0x7fd6840b55f0 2024-03-28T14:08:33.028 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:33.022+0000 7fd67a7fc640 1 -- 172.21.15.104:0/609230793 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) ==== 87803+0+0 (secure 0 0 0) 0x7fd67400bc40 con 0x7fd6840b55f0 2024-03-28T14:08:33.028 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:33.022+0000 7fd67a7fc640 1 --2- 172.21.15.104:0/609230793 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7fd668067fc0 0x7fd66806a480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:33.028 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:33.022+0000 7fd67a7fc640 1 -- 172.21.15.104:0/609230793 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5357+0+0 (secure 0 0 0) 0x7fd674086700 con 0x7fd6840b55f0 2024-03-28T14:08:33.028 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:33.022+0000 7fd68980f640 1 --2- 172.21.15.104:0/609230793 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7fd668067fc0 0x7fd66806a480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:33.028 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:33.026+0000 7fd68980f640 1 --2- 172.21.15.104:0/609230793 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7fd668067fc0 0x7fd66806a480 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fd6800ae000 tx=0x7fd680008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:08:33.031 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:33.026+0000 7fd67a7fc640 1 -- 172.21.15.104:0/609230793 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179117 (secure 0 0 0) 0x7fd6740570a0 con 0x7fd6840b55f0 2024-03-28T14:08:33.191 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:33.186+0000 7fd68ba73640 1 -- 172.21.15.104:0/609230793 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) -- 0x7fd684147440 con 0x7fd6840b55f0 2024-03-28T14:08:33.869 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:33.866+0000 7fd67a7fc640 1 -- 172.21.15.104:0/609230793 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v24) ==== 169+0+0 (secure 0 0 0) 0x7fd674086070 con 0x7fd6840b55f0 2024-03-28T14:08:33.870 INFO:teuthology.orchestra.run.smithi104.stderr:pool 'cephfs_data' removed 2024-03-28T14:08:33.874 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:33.870+0000 7fd68ba73640 1 -- 172.21.15.104:0/609230793 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7fd668067fc0 msgr2=0x7fd66806a480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:33.874 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:33.870+0000 7fd68ba73640 1 --2- 172.21.15.104:0/609230793 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7fd668067fc0 0x7fd66806a480 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fd6800ae000 tx=0x7fd680008040 comp rx=0 tx=0).stop 2024-03-28T14:08:33.874 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:33.870+0000 7fd68ba73640 1 -- 172.21.15.104:0/609230793 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd6840b55f0 msgr2=0x7fd68413fc10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:33.874 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:33.870+0000 7fd68ba73640 1 --2- 172.21.15.104:0/609230793 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd6840b55f0 0x7fd68413fc10 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7fd67400d800 tx=0x7fd67400dcc0 comp rx=0 tx=0).stop 2024-03-28T14:08:33.874 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:33.870+0000 7fd68ba73640 1 -- 172.21.15.104:0/609230793 shutdown_connections 2024-03-28T14:08:33.874 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:33.870+0000 7fd68ba73640 1 --2- 172.21.15.104:0/609230793 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fd6840b62a0 0x7fd6841406a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:33.874 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:33.870+0000 7fd68ba73640 1 --2- 172.21.15.104:0/609230793 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7fd668067fc0 0x7fd66806a480 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:33.875 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:33.870+0000 7fd68ba73640 1 --2- 172.21.15.104:0/609230793 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd6840b55f0 0x7fd68413fc10 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:33.875 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:33.870+0000 7fd68ba73640 1 --2- 172.21.15.104:0/609230793 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fd68406b5d0 0x7fd684145c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:33.875 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:33.870+0000 7fd68ba73640 1 -- 172.21.15.104:0/609230793 >> 172.21.15.104:0/609230793 conn(0x7fd6840b6990 msgr2=0x7fd6840b6d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T14:08:33.875 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:33.870+0000 7fd68ba73640 1 -- 172.21.15.104:0/609230793 shutdown_connections 2024-03-28T14:08:33.875 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:33.870+0000 7fd68ba73640 1 -- 172.21.15.104:0/609230793 wait complete. 2024-03-28T14:08:33.895 INFO:tasks.ceph.mds.a:Restarting daemon 2024-03-28T14:08:33.895 INFO:tasks.ceph.mds.a:Stopping old one... 2024-03-28T14:08:33.896 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-03-28T14:08:33.896 INFO:teuthology.orchestra.run:waiting for 300 2024-03-28T14:08:33.896 INFO:tasks.ceph.mds.b:Restarting daemon 2024-03-28T14:08:33.896 INFO:tasks.ceph.mds.b:Stopping old one... 2024-03-28T14:08:33.896 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-03-28T14:08:33.896 INFO:teuthology.orchestra.run:waiting for 300 2024-03-28T14:08:33.908 INFO:tasks.ceph.mds.a:Stopped 2024-03-28T14:08:33.908 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-03-28T14:08:33.911 INFO:tasks.ceph.mds.b:Stopped 2024-03-28T14:08:33.911 DEBUG:teuthology.orchestra.run.smithi171:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-03-28T14:08:33.953 INFO:tasks.ceph.mds.a:Started 2024-03-28T14:08:33.953 DEBUG:teuthology.parallel:result is None 2024-03-28T14:08:33.954 INFO:tasks.ceph.mds.b:Started 2024-03-28T14:08:33.954 DEBUG:teuthology.parallel:result is None 2024-03-28T14:08:33.954 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-03-28T14:08:34.026 INFO:tasks.ceph.mds.a.smithi104.stdout:starting mds.a at 2024-03-28T14:08:34.026 INFO:tasks.ceph.mds.b.smithi171.stdout:starting mds.b at 2024-03-28T14:08:34.077 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:34.074+0000 7f01799ed640 1 Processor -- start 2024-03-28T14:08:34.077 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:34.074+0000 7f01799ed640 1 -- start start 2024-03-28T14:08:34.078 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:34.074+0000 7f01799ed640 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f01740bd9e0 0x7f017414af30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:34.078 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:34.074+0000 7f01799ed640 1 --2- >> v2:172.21.15.171:3300/0 conn(0x7f017414b470 0x7f01741413f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:34.078 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:34.074+0000 7f01799ed640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f017414b900 con 0x7f0174141930 2024-03-28T14:08:34.078 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:34.074+0000 7f01799ed640 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 -- 0x7f01740b70b0 con 0x7f01740bd9e0 2024-03-28T14:08:34.078 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:34.074+0000 7f01799ed640 1 -- --> v2:172.21.15.171:3300/0 -- mon_getmap magic: 0 -- 0x7f01740b5050 con 0x7f017414b470 2024-03-28T14:08:34.078 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:34.074+0000 7f0172ffd640 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f01740bd9e0 0x7f017414af30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:34.078 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:34.074+0000 7f01737fe640 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f0174141930 0x7f01740b79f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:41662/0 (socket says 172.21.15.104:41662) 2024-03-28T14:08:34.079 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:34.074+0000 7f01737fe640 1 -- 172.21.15.104:0/1608685868 learned_addr learned my addr 172.21.15.104:0/1608685868 (peer_addr_for_me v1:172.21.15.104:0/0) 2024-03-28T14:08:34.079 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:34.074+0000 7f0172ffd640 1 -- 172.21.15.104:0/1608685868 >> v2:172.21.15.171:3300/0 conn(0x7f017414b470 msgr2=0x7f01741413f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T14:08:34.079 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:34.074+0000 7f0172ffd640 1 --2- 172.21.15.104:0/1608685868 >> v2:172.21.15.171:3300/0 conn(0x7f017414b470 0x7f01741413f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:34.079 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:34.074+0000 7f0172ffd640 1 -- 172.21.15.104:0/1608685868 >> v1:172.21.15.104:6789/0 conn(0x7f0174141930 legacy=0x7f01740b79f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:34.079 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:34.074+0000 7f0172ffd640 1 -- 172.21.15.104:0/1608685868 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f01740bddc0 con 0x7f01740bd9e0 2024-03-28T14:08:34.080 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:34.074+0000 7f0171ffb640 1 -- 172.21.15.104:0/1608685868 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 631234509 0 0) 0x7f017414b900 con 0x7f0174141930 2024-03-28T14:08:34.081 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:34.078+0000 7f0172ffd640 1 --2- 172.21.15.104:0/1608685868 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f01740bd9e0 0x7f017414af30 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f01740b69a0 tx=0x7f016401c330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=687061d5328df9f1 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:08:34.081 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:34.078+0000 7f0171ffb640 1 -- 172.21.15.104:0/1608685868 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f01640b0070 con 0x7f01740bd9e0 2024-03-28T14:08:34.081 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:34.078+0000 7f0171ffb640 1 -- 172.21.15.104:0/1608685868 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f01640ab600 con 0x7f01740bd9e0 2024-03-28T14:08:34.081 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:34.078+0000 7f0171ffb640 1 -- 172.21.15.104:0/1608685868 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f01640ab9d0 con 0x7f01740bd9e0 2024-03-28T14:08:34.082 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:34.078+0000 7f01799ed640 1 -- 172.21.15.104:0/1608685868 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f01740bd9e0 msgr2=0x7f017414af30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:34.082 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:34.078+0000 7f01799ed640 1 --2- 172.21.15.104:0/1608685868 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f01740bd9e0 0x7f017414af30 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f01740b69a0 tx=0x7f016401c330 comp rx=0 tx=0).stop 2024-03-28T14:08:34.082 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:34.078+0000 7f01799ed640 1 -- 172.21.15.104:0/1608685868 shutdown_connections 2024-03-28T14:08:34.082 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:34.078+0000 7f01799ed640 1 --2- 172.21.15.104:0/1608685868 >> v2:172.21.15.171:3300/0 conn(0x7f017414b470 0x7f01741413f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:34.082 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:34.078+0000 7f01799ed640 1 --2- 172.21.15.104:0/1608685868 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f01740bd9e0 0x7f017414af30 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:34.082 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:34.078+0000 7f01799ed640 1 -- 172.21.15.104:0/1608685868 >> 172.21.15.104:0/1608685868 conn(0x7f0174073530 msgr2=0x7f01740bd070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T14:08:34.082 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:34.078+0000 7f01799ed640 1 -- 172.21.15.104:0/1608685868 shutdown_connections 2024-03-28T14:08:34.082 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:34.078+0000 7f01799ed640 1 -- 172.21.15.104:0/1608685868 wait complete. 2024-03-28T14:08:34.087 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:34.078+0000 7f01799ed640 1 Processor -- start 2024-03-28T14:08:34.087 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:34.078+0000 7f01799ed640 1 -- start start 2024-03-28T14:08:34.087 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:34.078+0000 7f01799ed640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f01740bd9e0 0x7f0174139900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:34.087 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:34.078+0000 7f01799ed640 1 --2- >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f0174141930 0x7f0174144060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:34.087 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:34.078+0000 7f01799ed640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f017414b470 0x7f0174149b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:34.087 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:34.078+0000 7f01799ed640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f01740b6140 con 0x7f01740bd9e0 2024-03-28T14:08:34.087 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:34.078+0000 7f01799ed640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f0174141d40 con 0x7f017414b470 2024-03-28T14:08:34.087 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:34.078+0000 7f01799ed640 1 -- --> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] -- mon_getmap magic: 0 -- 0x7f01740b62c0 con 0x7f0174141930 2024-03-28T14:08:34.087 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:34.078+0000 7f0172ffd640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f01740bd9e0 0x7f0174139900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:34.088 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:34.078+0000 7f0172ffd640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f01740bd9e0 0x7f0174139900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:59100/0 (socket says 172.21.15.104:59100) 2024-03-28T14:08:34.088 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:34.078+0000 7f0172ffd640 1 -- 172.21.15.104:0/2339739382 learned_addr learned my addr 172.21.15.104:0/2339739382 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-03-28T14:08:34.088 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:34.078+0000 7f01727fc640 1 --2- 172.21.15.104:0/2339739382 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f0174141930 0x7f0174144060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:34.088 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:34.078+0000 7f01737fe640 1 --2- 172.21.15.104:0/2339739382 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f017414b470 0x7f0174149b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:34.088 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:34.078+0000 7f0172ffd640 1 -- 172.21.15.104:0/2339739382 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f0174141930 msgr2=0x7f0174144060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:34.088 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:34.078+0000 7f0172ffd640 1 --2- 172.21.15.104:0/2339739382 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f0174141930 0x7f0174144060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:34.088 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:34.078+0000 7f0172ffd640 1 -- 172.21.15.104:0/2339739382 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f017414b470 msgr2=0x7f0174149b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:34.088 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:34.078+0000 7f0172ffd640 1 --2- 172.21.15.104:0/2339739382 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f017414b470 0x7f0174149b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:34.089 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:34.078+0000 7f0172ffd640 1 -- 172.21.15.104:0/2339739382 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0174143b80 con 0x7f01740bd9e0 2024-03-28T14:08:34.089 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:34.078+0000 7f0172ffd640 1 --2- 172.21.15.104:0/2339739382 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f01740bd9e0 0x7f0174139900 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f01640ae040 tx=0x7f016401ab70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:08:34.089 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:34.078+0000 7f015bfff640 1 -- 172.21.15.104:0/2339739382 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f01640b0050 con 0x7f01740bd9e0 2024-03-28T14:08:34.089 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:34.078+0000 7f015bfff640 1 -- 172.21.15.104:0/2339739382 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f01640bc070 con 0x7f01740bd9e0 2024-03-28T14:08:34.089 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:34.078+0000 7f01799ed640 1 -- 172.21.15.104:0/2339739382 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f017413e0d0 con 0x7f01740bd9e0 2024-03-28T14:08:34.089 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:34.078+0000 7f015bfff640 1 -- 172.21.15.104:0/2339739382 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f01640b5a70 con 0x7f01740bd9e0 2024-03-28T14:08:34.090 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:34.082+0000 7f01799ed640 1 -- 172.21.15.104:0/2339739382 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f017413da70 con 0x7f01740bd9e0 2024-03-28T14:08:34.090 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:34.082+0000 7f01799ed640 1 -- 172.21.15.104:0/2339739382 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0174143d50 con 0x7f01740bd9e0 2024-03-28T14:08:34.090 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:34.082+0000 7f015bfff640 1 -- 172.21.15.104:0/2339739382 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) ==== 87803+0+0 (secure 0 0 0) 0x7f01640b5c10 con 0x7f01740bd9e0 2024-03-28T14:08:34.090 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:34.082+0000 7f015bfff640 1 --2- 172.21.15.104:0/2339739382 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f0160067fc0 0x7f016006a480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:34.091 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:34.082+0000 7f015bfff640 1 -- 172.21.15.104:0/2339739382 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 4943+0+0 (secure 0 0 0) 0x7f0164120ac0 con 0x7f01740bd9e0 2024-03-28T14:08:34.091 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:34.082+0000 7f01727fc640 1 --2- 172.21.15.104:0/2339739382 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f0160067fc0 0x7f016006a480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:34.091 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:34.082+0000 7f01727fc640 1 --2- 172.21.15.104:0/2339739382 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f0160067fc0 0x7f016006a480 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f0168006fd0 tx=0x7f0168008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:08:34.091 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:34.086+0000 7f015bfff640 1 -- 172.21.15.104:0/2339739382 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179117 (secure 0 0 0) 0x7f0164016070 con 0x7f01740bd9e0 2024-03-28T14:08:34.266 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:34.262+0000 7f01799ed640 1 -- 172.21.15.104:0/2339739382 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) -- 0x7f01740b6af0 con 0x7f01740bd9e0 2024-03-28T14:08:34.889 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:34.886+0000 7f015bfff640 1 -- 172.21.15.104:0/2339739382 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v26) ==== 97+0+0 (secure 0 0 0) 0x7f01640f1600 con 0x7f01740bd9e0 2024-03-28T14:08:34.889 INFO:teuthology.orchestra.run.smithi104.stderr: removed all blocklist entries 2024-03-28T14:08:34.894 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:34.890+0000 7f01799ed640 1 -- 172.21.15.104:0/2339739382 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f0160067fc0 msgr2=0x7f016006a480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:34.895 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:34.890+0000 7f01799ed640 1 --2- 172.21.15.104:0/2339739382 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f0160067fc0 0x7f016006a480 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f0168006fd0 tx=0x7f0168008040 comp rx=0 tx=0).stop 2024-03-28T14:08:34.895 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:34.890+0000 7f01799ed640 1 -- 172.21.15.104:0/2339739382 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f01740bd9e0 msgr2=0x7f0174139900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:34.895 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:34.890+0000 7f01799ed640 1 --2- 172.21.15.104:0/2339739382 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f01740bd9e0 0x7f0174139900 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f01640ae040 tx=0x7f016401ab70 comp rx=0 tx=0).stop 2024-03-28T14:08:34.895 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:34.890+0000 7f01799ed640 1 -- 172.21.15.104:0/2339739382 shutdown_connections 2024-03-28T14:08:34.895 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:34.890+0000 7f01799ed640 1 --2- 172.21.15.104:0/2339739382 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f017414b470 0x7f0174149b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:34.895 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:34.890+0000 7f01799ed640 1 --2- 172.21.15.104:0/2339739382 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f0174141930 0x7f0174144060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:34.895 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:34.890+0000 7f01799ed640 1 --2- 172.21.15.104:0/2339739382 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f0160067fc0 0x7f016006a480 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:34.895 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:34.890+0000 7f01799ed640 1 --2- 172.21.15.104:0/2339739382 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f01740bd9e0 0x7f0174139900 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:34.895 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:34.890+0000 7f01799ed640 1 -- 172.21.15.104:0/2339739382 >> 172.21.15.104:0/2339739382 conn(0x7f0174073530 msgr2=0x7f0174067330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T14:08:34.895 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:34.890+0000 7f01799ed640 1 -- 172.21.15.104:0/2339739382 shutdown_connections 2024-03-28T14:08:34.895 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:34.890+0000 7f01799ed640 1 -- 172.21.15.104:0/2339739382 wait complete. 2024-03-28T14:08:34.912 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-03-28T14:08:35.035 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:35.030+0000 7f7f6e89d640 1 Processor -- start 2024-03-28T14:08:35.035 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:35.030+0000 7f7f6e89d640 1 -- start start 2024-03-28T14:08:35.036 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:35.034+0000 7f7f6e89d640 1 --2- >> v2:172.21.15.171:3300/0 conn(0x7f7f6813faf0 0x7f7f680b0c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:35.036 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:35.034+0000 7f7f6e89d640 1 --2- >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f7f680732a0 0x7f7f680b1170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:35.036 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:35.034+0000 7f7f6e89d640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7f6806dd50 con 0x7f7f6806d970 2024-03-28T14:08:35.036 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:35.034+0000 7f7f6e89d640 1 -- --> v2:172.21.15.171:3300/0 -- mon_getmap magic: 0 -- 0x7f7f680b02a0 con 0x7f7f6813faf0 2024-03-28T14:08:35.036 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:35.034+0000 7f7f6e89d640 1 -- --> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] -- mon_getmap magic: 0 -- 0x7f7f680b2d30 con 0x7f7f680732a0 2024-03-28T14:08:35.037 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:35.034+0000 7f7f6ce3a640 1 --2- >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f7f680732a0 0x7f7f680b1170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:35.037 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:35.034+0000 7f7f677fe640 1 --2- >> v2:172.21.15.171:3300/0 conn(0x7f7f6813faf0 0x7f7f680b0c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:35.038 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:35.034+0000 7f7f677fe640 1 --2- >> v2:172.21.15.171:3300/0 conn(0x7f7f6813faf0 0x7f7f680b0c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.171:3300/0 says I am v2:172.21.15.104:59542/0 (socket says 172.21.15.104:59542) 2024-03-28T14:08:35.038 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:35.034+0000 7f7f6ce3a640 1 --2- >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f7f680732a0 0x7f7f680b1170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.171:3301/0 says I am v2:172.21.15.104:55016/0 (socket says 172.21.15.104:55016) 2024-03-28T14:08:35.038 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:35.034+0000 7f7f677fe640 1 -- 172.21.15.104:0/2065693068 learned_addr learned my addr 172.21.15.104:0/2065693068 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-03-28T14:08:35.038 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:35.034+0000 7f7f67fff640 1 --1- 172.21.15.104:0/2065693068 >> v1:172.21.15.171:6789/0 conn(0x7f7f6806d970 0x7f7f6813f3e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.104:58470/0 (socket says 172.21.15.104:58470) 2024-03-28T14:08:35.038 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:35.034+0000 7f7f677fe640 1 -- 172.21.15.104:0/2065693068 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f7f680732a0 msgr2=0x7f7f680b1170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T14:08:35.038 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:35.034+0000 7f7f677fe640 1 --2- 172.21.15.104:0/2065693068 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f7f680732a0 0x7f7f680b1170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:35.038 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:35.034+0000 7f7f677fe640 1 -- 172.21.15.104:0/2065693068 >> v1:172.21.15.171:6789/0 conn(0x7f7f6806d970 legacy=0x7f7f6813f3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:35.038 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:35.034+0000 7f7f677fe640 1 -- 172.21.15.104:0/2065693068 --> v2:172.21.15.171:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7f680b3010 con 0x7f7f6813faf0 2024-03-28T14:08:35.039 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:35.038+0000 7f7f677fe640 1 --2- 172.21.15.104:0/2065693068 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f7f6813faf0 0x7f7f680b0c30 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f7f50009870 tx=0x7f7f5001c4d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=dccd8d6202c7774e server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:08:35.040 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:35.038+0000 7f7f66ffd640 1 -- 172.21.15.104:0/2065693068 <== mon.1 v2:172.21.15.171:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f500aa610 con 0x7f7f6813faf0 2024-03-28T14:08:35.040 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:35.038+0000 7f7f66ffd640 1 -- 172.21.15.104:0/2065693068 <== mon.1 v2:172.21.15.171:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7f500af070 con 0x7f7f6813faf0 2024-03-28T14:08:35.040 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:35.038+0000 7f7f66ffd640 1 -- 172.21.15.104:0/2065693068 <== mon.1 v2:172.21.15.171:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f500aa9d0 con 0x7f7f6813faf0 2024-03-28T14:08:35.041 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:35.038+0000 7f7f6e89d640 1 -- 172.21.15.104:0/2065693068 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f7f6813faf0 msgr2=0x7f7f680b0c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:35.041 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:35.038+0000 7f7f6e89d640 1 --2- 172.21.15.104:0/2065693068 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f7f6813faf0 0x7f7f680b0c30 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f7f50009870 tx=0x7f7f5001c4d0 comp rx=0 tx=0).stop 2024-03-28T14:08:35.041 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:35.038+0000 7f7f6e89d640 1 -- 172.21.15.104:0/2065693068 shutdown_connections 2024-03-28T14:08:35.041 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:35.038+0000 7f7f6e89d640 1 --2- 172.21.15.104:0/2065693068 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f7f680732a0 0x7f7f680b1170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:35.041 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:35.038+0000 7f7f6e89d640 1 --2- 172.21.15.104:0/2065693068 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f7f6813faf0 0x7f7f680b0c30 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:35.041 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:35.038+0000 7f7f6e89d640 1 -- 172.21.15.104:0/2065693068 >> 172.21.15.104:0/2065693068 conn(0x7f7f680b58a0 msgr2=0x7f7f680b5cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T14:08:35.041 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:35.038+0000 7f7f6e89d640 1 -- 172.21.15.104:0/2065693068 shutdown_connections 2024-03-28T14:08:35.041 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:35.038+0000 7f7f6e89d640 1 -- 172.21.15.104:0/2065693068 wait complete. 2024-03-28T14:08:35.043 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:35.038+0000 7f7f6e89d640 1 Processor -- start 2024-03-28T14:08:35.044 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:35.038+0000 7f7f6e89d640 1 -- start start 2024-03-28T14:08:35.044 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:35.038+0000 7f7f6e89d640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f7f6806d970 0x7f7f68142760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:35.044 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:35.038+0000 7f7f6e89d640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7f680732a0 0x7f7f68141a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:35.044 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:35.038+0000 7f7f6e89d640 1 --2- >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f7f6813faf0 0x7f7f68072170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:35.044 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:35.038+0000 7f7f6e89d640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f7f68152a10 con 0x7f7f680732a0 2024-03-28T14:08:35.044 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:35.038+0000 7f7f6e89d640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f7f68152890 con 0x7f7f6806d970 2024-03-28T14:08:35.044 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:35.038+0000 7f7f6e89d640 1 -- --> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] -- mon_getmap magic: 0 -- 0x7f7f68152b90 con 0x7f7f6813faf0 2024-03-28T14:08:35.044 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:35.038+0000 7f7f677fe640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7f680732a0 0x7f7f68141a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:35.044 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:35.038+0000 7f7f677fe640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7f680732a0 0x7f7f68141a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:59108/0 (socket says 172.21.15.104:59108) 2024-03-28T14:08:35.045 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:35.038+0000 7f7f677fe640 1 -- 172.21.15.104:0/1387561044 learned_addr learned my addr 172.21.15.104:0/1387561044 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-03-28T14:08:35.045 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:35.038+0000 7f7f6ce3a640 1 --2- 172.21.15.104:0/1387561044 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f7f6813faf0 0x7f7f68072170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:35.045 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:35.038+0000 7f7f67fff640 1 --2- 172.21.15.104:0/1387561044 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f7f6806d970 0x7f7f68142760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:35.045 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:35.038+0000 7f7f677fe640 1 -- 172.21.15.104:0/1387561044 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f7f6813faf0 msgr2=0x7f7f68072170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:35.045 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:35.038+0000 7f7f677fe640 1 --2- 172.21.15.104:0/1387561044 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f7f6813faf0 0x7f7f68072170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:35.045 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:35.038+0000 7f7f677fe640 1 -- 172.21.15.104:0/1387561044 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f7f6806d970 msgr2=0x7f7f68142760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:35.045 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:35.038+0000 7f7f677fe640 1 --2- 172.21.15.104:0/1387561044 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f7f6806d970 0x7f7f68142760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:35.045 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:35.038+0000 7f7f677fe640 1 -- 172.21.15.104:0/1387561044 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7f68142ca0 con 0x7f7f680732a0 2024-03-28T14:08:35.045 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:35.038+0000 7f7f67fff640 1 --2- 172.21.15.104:0/1387561044 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f7f6806d970 0x7f7f68142760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T14:08:35.045 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:35.038+0000 7f7f677fe640 1 --2- 172.21.15.104:0/1387561044 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7f680732a0 0x7f7f68141a30 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f7f500ac040 tx=0x7f7f5001ad10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:08:35.045 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:35.042+0000 7f7f64ff9640 1 -- 172.21.15.104:0/1387561044 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f500bd3f0 con 0x7f7f680732a0 2024-03-28T14:08:35.045 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:35.042+0000 7f7f64ff9640 1 -- 172.21.15.104:0/1387561044 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7f500bd590 con 0x7f7f680732a0 2024-03-28T14:08:35.045 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:35.042+0000 7f7f6e89d640 1 -- 172.21.15.104:0/1387561044 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7f681de110 con 0x7f7f680732a0 2024-03-28T14:08:35.045 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:35.042+0000 7f7f64ff9640 1 -- 172.21.15.104:0/1387561044 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f500bd960 con 0x7f7f680732a0 2024-03-28T14:08:35.046 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:35.042+0000 7f7f6e89d640 1 -- 172.21.15.104:0/1387561044 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7f681de3c0 con 0x7f7f680732a0 2024-03-28T14:08:35.047 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:35.042+0000 7f7f6e89d640 1 -- 172.21.15.104:0/1387561044 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7f680687c0 con 0x7f7f680732a0 2024-03-28T14:08:35.047 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:35.042+0000 7f7f64ff9640 1 -- 172.21.15.104:0/1387561044 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) ==== 87803+0+0 (secure 0 0 0) 0x7f7f500c6440 con 0x7f7f680732a0 2024-03-28T14:08:35.047 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:35.042+0000 7f7f64ff9640 1 --2- 172.21.15.104:0/1387561044 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f7f48067fc0 0x7f7f4806a480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:35.047 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:35.042+0000 7f7f64ff9640 1 -- 172.21.15.104:0/1387561044 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 4857+0+0 (secure 0 0 0) 0x7f7f50120710 con 0x7f7f680732a0 2024-03-28T14:08:35.047 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:35.042+0000 7f7f67fff640 1 --2- 172.21.15.104:0/1387561044 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f7f48067fc0 0x7f7f4806a480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:35.047 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:35.042+0000 7f7f67fff640 1 --2- 172.21.15.104:0/1387561044 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f7f48067fc0 0x7f7f4806a480 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f7f60005fd0 tx=0x7f7f60005b80 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:08:35.050 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:35.046+0000 7f7f64ff9640 1 -- 172.21.15.104:0/1387561044 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179117 (secure 0 0 0) 0x7f7f500a7070 con 0x7f7f680732a0 2024-03-28T14:08:35.266 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:35.262+0000 7f7f6e89d640 1 -- 172.21.15.104:0/1387561044 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) -- 0x7f7f6806c030 con 0x7f7f680732a0 2024-03-28T14:08:35.267 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:35.262+0000 7f7f64ff9640 1 -- 172.21.15.104:0/1387561044 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v4) ==== 80+0+5207 (secure 0 0 0) 0x7f7f50016150 con 0x7f7f680732a0 2024-03-28T14:08:35.267 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-03-28T14:08:35.268 INFO:teuthology.orchestra.run.smithi104.stdout:{ 2024-03-28T14:08:35.268 INFO:teuthology.orchestra.run.smithi104.stdout: "auth_dump": [ 2024-03-28T14:08:35.268 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-03-28T14:08:35.268 INFO:teuthology.orchestra.run.smithi104.stdout: "entity": "mds.a", 2024-03-28T14:08:35.268 INFO:teuthology.orchestra.run.smithi104.stdout: "key": "AQCFeQVmR5QyDhAAi9PSbldc6n0vgJYojFBc7w==", 2024-03-28T14:08:35.268 INFO:teuthology.orchestra.run.smithi104.stdout: "caps": { 2024-03-28T14:08:35.268 INFO:teuthology.orchestra.run.smithi104.stdout: "mds": "allow", 2024-03-28T14:08:35.268 INFO:teuthology.orchestra.run.smithi104.stdout: "mgr": "allow *", 2024-03-28T14:08:35.268 INFO:teuthology.orchestra.run.smithi104.stdout: "mon": "allow *", 2024-03-28T14:08:35.268 INFO:teuthology.orchestra.run.smithi104.stdout: "osd": "allow *" 2024-03-28T14:08:35.268 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-03-28T14:08:35.268 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-03-28T14:08:35.268 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-03-28T14:08:35.269 INFO:teuthology.orchestra.run.smithi104.stdout: "entity": "mds.b", 2024-03-28T14:08:35.269 INFO:teuthology.orchestra.run.smithi104.stdout: "key": "AQCFeQVm9M5TFBAAP3Vf8lLVq8X5bUs8i6Ta0A==", 2024-03-28T14:08:35.269 INFO:teuthology.orchestra.run.smithi104.stdout: "caps": { 2024-03-28T14:08:35.269 INFO:teuthology.orchestra.run.smithi104.stdout: "mds": "allow", 2024-03-28T14:08:35.269 INFO:teuthology.orchestra.run.smithi104.stdout: "mgr": "allow *", 2024-03-28T14:08:35.269 INFO:teuthology.orchestra.run.smithi104.stdout: "mon": "allow *", 2024-03-28T14:08:35.269 INFO:teuthology.orchestra.run.smithi104.stdout: "osd": "allow *" 2024-03-28T14:08:35.269 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-03-28T14:08:35.269 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-03-28T14:08:35.269 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-03-28T14:08:35.269 INFO:teuthology.orchestra.run.smithi104.stdout: "entity": "osd.0", 2024-03-28T14:08:35.269 INFO:teuthology.orchestra.run.smithi104.stdout: "key": "AQCIeQVm8wh8IRAArenUIzBmf88Ujjqo019SbA==", 2024-03-28T14:08:35.269 INFO:teuthology.orchestra.run.smithi104.stdout: "caps": { 2024-03-28T14:08:35.269 INFO:teuthology.orchestra.run.smithi104.stdout: "mgr": "allow profile osd", 2024-03-28T14:08:35.270 INFO:teuthology.orchestra.run.smithi104.stdout: "mon": "allow profile osd", 2024-03-28T14:08:35.270 INFO:teuthology.orchestra.run.smithi104.stdout: "osd": "allow *" 2024-03-28T14:08:35.270 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-03-28T14:08:35.270 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-03-28T14:08:35.270 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-03-28T14:08:35.270 INFO:teuthology.orchestra.run.smithi104.stdout: "entity": "osd.1", 2024-03-28T14:08:35.270 INFO:teuthology.orchestra.run.smithi104.stdout: "key": "AQCLeQVmcj1ZJxAAyxCHsIh7x2ZT1LcQtV4efg==", 2024-03-28T14:08:35.270 INFO:teuthology.orchestra.run.smithi104.stdout: "caps": { 2024-03-28T14:08:35.270 INFO:teuthology.orchestra.run.smithi104.stdout: "mgr": "allow profile osd", 2024-03-28T14:08:35.270 INFO:teuthology.orchestra.run.smithi104.stdout: "mon": "allow profile osd", 2024-03-28T14:08:35.271 INFO:teuthology.orchestra.run.smithi104.stdout: "osd": "allow *" 2024-03-28T14:08:35.271 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-03-28T14:08:35.271 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-03-28T14:08:35.271 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-03-28T14:08:35.271 INFO:teuthology.orchestra.run.smithi104.stdout: "entity": "osd.2", 2024-03-28T14:08:35.271 INFO:teuthology.orchestra.run.smithi104.stdout: "key": "AQCOeQVmf3ixLxAAbuuAonIliKXnzp2IMc8gdg==", 2024-03-28T14:08:35.272 INFO:teuthology.orchestra.run.smithi104.stdout: "caps": { 2024-03-28T14:08:35.272 INFO:teuthology.orchestra.run.smithi104.stdout: "mgr": "allow profile osd", 2024-03-28T14:08:35.272 INFO:teuthology.orchestra.run.smithi104.stdout: "mon": "allow profile osd", 2024-03-28T14:08:35.272 INFO:teuthology.orchestra.run.smithi104.stdout: "osd": "allow *" 2024-03-28T14:08:35.272 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-03-28T14:08:35.272 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-03-28T14:08:35.272 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-03-28T14:08:35.272 INFO:teuthology.orchestra.run.smithi104.stdout: "entity": "osd.3", 2024-03-28T14:08:35.273 INFO:teuthology.orchestra.run.smithi104.stdout: "key": "AQCReQVmB9LPNRAAfUx+PjggnlhW1PIwHt5/Vw==", 2024-03-28T14:08:35.273 INFO:teuthology.orchestra.run.smithi104.stdout: "caps": { 2024-03-28T14:08:35.273 INFO:teuthology.orchestra.run.smithi104.stdout: "mgr": "allow profile osd", 2024-03-28T14:08:35.273 INFO:teuthology.orchestra.run.smithi104.stdout: "mon": "allow profile osd", 2024-03-28T14:08:35.273 INFO:teuthology.orchestra.run.smithi104.stdout: "osd": "allow *" 2024-03-28T14:08:35.273 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-03-28T14:08:35.273 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-03-28T14:08:35.273 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-03-28T14:08:35.273 INFO:teuthology.orchestra.run.smithi104.stdout: "entity": "osd.4", 2024-03-28T14:08:35.274 INFO:teuthology.orchestra.run.smithi104.stdout: "key": "AQCXeQVmd+42OhAAERCUxkaXDdi1vpiHnyL2Rw==", 2024-03-28T14:08:35.274 INFO:teuthology.orchestra.run.smithi104.stdout: "caps": { 2024-03-28T14:08:35.274 INFO:teuthology.orchestra.run.smithi104.stdout: "mgr": "allow profile osd", 2024-03-28T14:08:35.274 INFO:teuthology.orchestra.run.smithi104.stdout: "mon": "allow profile osd", 2024-03-28T14:08:35.274 INFO:teuthology.orchestra.run.smithi104.stdout: "osd": "allow *" 2024-03-28T14:08:35.274 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-03-28T14:08:35.274 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-03-28T14:08:35.274 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-03-28T14:08:35.274 INFO:teuthology.orchestra.run.smithi104.stdout: "entity": "osd.5", 2024-03-28T14:08:35.274 INFO:teuthology.orchestra.run.smithi104.stdout: "key": "AQCbeQVm0eVgAxAAgQIiw5t2P7w6kNIiO0b+SA==", 2024-03-28T14:08:35.274 INFO:teuthology.orchestra.run.smithi104.stdout: "caps": { 2024-03-28T14:08:35.274 INFO:teuthology.orchestra.run.smithi104.stdout: "mgr": "allow profile osd", 2024-03-28T14:08:35.274 INFO:teuthology.orchestra.run.smithi104.stdout: "mon": "allow profile osd", 2024-03-28T14:08:35.275 INFO:teuthology.orchestra.run.smithi104.stdout: "osd": "allow *" 2024-03-28T14:08:35.275 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-03-28T14:08:35.275 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-03-28T14:08:35.275 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-03-28T14:08:35.275 INFO:teuthology.orchestra.run.smithi104.stdout: "entity": "osd.6", 2024-03-28T14:08:35.275 INFO:teuthology.orchestra.run.smithi104.stdout: "key": "AQCeeQVmJX0SCxAARorbp1LQBwG7Ro7YZvDM4Q==", 2024-03-28T14:08:35.275 INFO:teuthology.orchestra.run.smithi104.stdout: "caps": { 2024-03-28T14:08:35.275 INFO:teuthology.orchestra.run.smithi104.stdout: "mgr": "allow profile osd", 2024-03-28T14:08:35.275 INFO:teuthology.orchestra.run.smithi104.stdout: "mon": "allow profile osd", 2024-03-28T14:08:35.275 INFO:teuthology.orchestra.run.smithi104.stdout: "osd": "allow *" 2024-03-28T14:08:35.275 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-03-28T14:08:35.275 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-03-28T14:08:35.275 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-03-28T14:08:35.275 INFO:teuthology.orchestra.run.smithi104.stdout: "entity": "osd.7", 2024-03-28T14:08:35.275 INFO:teuthology.orchestra.run.smithi104.stdout: "key": "AQCheQVmQwxIDxAAKNWebD4FNjtqCSP12WLAKw==", 2024-03-28T14:08:35.276 INFO:teuthology.orchestra.run.smithi104.stdout: "caps": { 2024-03-28T14:08:35.276 INFO:teuthology.orchestra.run.smithi104.stdout: "mgr": "allow profile osd", 2024-03-28T14:08:35.276 INFO:teuthology.orchestra.run.smithi104.stdout: "mon": "allow profile osd", 2024-03-28T14:08:35.276 INFO:teuthology.orchestra.run.smithi104.stdout: "osd": "allow *" 2024-03-28T14:08:35.276 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-03-28T14:08:35.276 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-03-28T14:08:35.276 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-03-28T14:08:35.276 INFO:teuthology.orchestra.run.smithi104.stdout: "entity": "client.0", 2024-03-28T14:08:35.276 INFO:teuthology.orchestra.run.smithi104.stdout: "key": "AQCFeQVmkAgiGhAAUVl6HwWo1xt2L4bwVt1VWw==", 2024-03-28T14:08:35.276 INFO:teuthology.orchestra.run.smithi104.stdout: "caps": { 2024-03-28T14:08:35.276 INFO:teuthology.orchestra.run.smithi104.stdout: "mds": "allow", 2024-03-28T14:08:35.276 INFO:teuthology.orchestra.run.smithi104.stdout: "mgr": "allow r", 2024-03-28T14:08:35.276 INFO:teuthology.orchestra.run.smithi104.stdout: "mon": "allow rw", 2024-03-28T14:08:35.276 INFO:teuthology.orchestra.run.smithi104.stdout: "osd": "allow rwx" 2024-03-28T14:08:35.277 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-03-28T14:08:35.277 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-03-28T14:08:35.277 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-03-28T14:08:35.277 INFO:teuthology.orchestra.run.smithi104.stdout: "entity": "client.admin", 2024-03-28T14:08:35.277 INFO:teuthology.orchestra.run.smithi104.stdout: "key": "AQCEeQVmNLyqLhAAU8xLySSzLuZoOjIK0ZUNBA==", 2024-03-28T14:08:35.277 INFO:teuthology.orchestra.run.smithi104.stdout: "caps": { 2024-03-28T14:08:35.277 INFO:teuthology.orchestra.run.smithi104.stdout: "mds": "allow *", 2024-03-28T14:08:35.277 INFO:teuthology.orchestra.run.smithi104.stdout: "mgr": "allow *", 2024-03-28T14:08:35.277 INFO:teuthology.orchestra.run.smithi104.stdout: "mon": "allow *", 2024-03-28T14:08:35.277 INFO:teuthology.orchestra.run.smithi104.stdout: "osd": "allow *" 2024-03-28T14:08:35.277 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-03-28T14:08:35.277 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-03-28T14:08:35.278 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-03-28T14:08:35.278 INFO:teuthology.orchestra.run.smithi104.stdout: "entity": "client.bootstrap-mds", 2024-03-28T14:08:35.278 INFO:teuthology.orchestra.run.smithi104.stdout: "key": "AQCmeQVmO8F5MxAAprRDuXHElWoQQ6b9eFNrPw==", 2024-03-28T14:08:35.278 INFO:teuthology.orchestra.run.smithi104.stdout: "caps": { 2024-03-28T14:08:35.278 INFO:teuthology.orchestra.run.smithi104.stdout: "mon": "allow profile bootstrap-mds" 2024-03-28T14:08:35.278 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-03-28T14:08:35.278 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-03-28T14:08:35.278 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-03-28T14:08:35.278 INFO:teuthology.orchestra.run.smithi104.stdout: "entity": "client.bootstrap-mgr", 2024-03-28T14:08:35.278 INFO:teuthology.orchestra.run.smithi104.stdout: "key": "AQCmeQVmCtx5MxAASyaiWn0/xaahZ2exDDxtRw==", 2024-03-28T14:08:35.279 INFO:teuthology.orchestra.run.smithi104.stdout: "caps": { 2024-03-28T14:08:35.279 INFO:teuthology.orchestra.run.smithi104.stdout: "mon": "allow profile bootstrap-mgr" 2024-03-28T14:08:35.279 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-03-28T14:08:35.279 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-03-28T14:08:35.279 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-03-28T14:08:35.279 INFO:teuthology.orchestra.run.smithi104.stdout: "entity": "client.bootstrap-osd", 2024-03-28T14:08:35.279 INFO:teuthology.orchestra.run.smithi104.stdout: "key": "AQCmeQVmq/J5MxAAGOjRnGgiAi3NNch2qYcZ1Q==", 2024-03-28T14:08:35.279 INFO:teuthology.orchestra.run.smithi104.stdout: "caps": { 2024-03-28T14:08:35.279 INFO:teuthology.orchestra.run.smithi104.stdout: "mon": "allow profile bootstrap-osd" 2024-03-28T14:08:35.279 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-03-28T14:08:35.279 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-03-28T14:08:35.279 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-03-28T14:08:35.280 INFO:teuthology.orchestra.run.smithi104.stdout: "entity": "client.bootstrap-rbd", 2024-03-28T14:08:35.280 INFO:teuthology.orchestra.run.smithi104.stdout: "key": "AQCmeQVmbQh6MxAABn50HsWADjq+jib9SijWPQ==", 2024-03-28T14:08:35.280 INFO:teuthology.orchestra.run.smithi104.stdout: "caps": { 2024-03-28T14:08:35.280 INFO:teuthology.orchestra.run.smithi104.stdout: "mon": "allow profile bootstrap-rbd" 2024-03-28T14:08:35.280 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-03-28T14:08:35.280 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-03-28T14:08:35.280 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-03-28T14:08:35.280 INFO:teuthology.orchestra.run.smithi104.stdout: "entity": "client.bootstrap-rbd-mirror", 2024-03-28T14:08:35.280 INFO:teuthology.orchestra.run.smithi104.stdout: "key": "AQCmeQVm+x96MxAAEKQbufk9cTTSqMH6UXhTWw==", 2024-03-28T14:08:35.280 INFO:teuthology.orchestra.run.smithi104.stdout: "caps": { 2024-03-28T14:08:35.280 INFO:teuthology.orchestra.run.smithi104.stdout: "mon": "allow profile bootstrap-rbd-mirror" 2024-03-28T14:08:35.281 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-03-28T14:08:35.281 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-03-28T14:08:35.281 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-03-28T14:08:35.281 INFO:teuthology.orchestra.run.smithi104.stdout: "entity": "client.bootstrap-rgw", 2024-03-28T14:08:35.281 INFO:teuthology.orchestra.run.smithi104.stdout: "key": "AQCmeQVm9jZ6MxAAxnoRNHLWvx/tO6pec+nSXw==", 2024-03-28T14:08:35.281 INFO:teuthology.orchestra.run.smithi104.stdout: "caps": { 2024-03-28T14:08:35.281 INFO:teuthology.orchestra.run.smithi104.stdout: "mon": "allow profile bootstrap-rgw" 2024-03-28T14:08:35.281 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-03-28T14:08:35.281 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-03-28T14:08:35.281 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-03-28T14:08:35.281 INFO:teuthology.orchestra.run.smithi104.stdout: "entity": "mgr.x", 2024-03-28T14:08:35.281 INFO:teuthology.orchestra.run.smithi104.stdout: "key": "AQCFeQVmsescCxAA6RH0UHxvAzTjUsxKGN141g==", 2024-03-28T14:08:35.281 INFO:teuthology.orchestra.run.smithi104.stdout: "caps": { 2024-03-28T14:08:35.282 INFO:teuthology.orchestra.run.smithi104.stdout: "mds": "allow *", 2024-03-28T14:08:35.282 INFO:teuthology.orchestra.run.smithi104.stdout: "mon": "allow profile mgr", 2024-03-28T14:08:35.282 INFO:teuthology.orchestra.run.smithi104.stdout: "osd": "allow *" 2024-03-28T14:08:35.282 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-03-28T14:08:35.282 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-03-28T14:08:35.282 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-03-28T14:08:35.282 INFO:teuthology.orchestra.run.smithi104.stdout: "entity": "mgr.y", 2024-03-28T14:08:35.282 INFO:teuthology.orchestra.run.smithi104.stdout: "key": "AQCFeQVmIL12CBAAuJ8i8ChzCtQXwKFxiyzaRg==", 2024-03-28T14:08:35.282 INFO:teuthology.orchestra.run.smithi104.stdout: "caps": { 2024-03-28T14:08:35.282 INFO:teuthology.orchestra.run.smithi104.stdout: "mds": "allow *", 2024-03-28T14:08:35.283 INFO:teuthology.orchestra.run.smithi104.stdout: "mon": "allow profile mgr", 2024-03-28T14:08:35.283 INFO:teuthology.orchestra.run.smithi104.stdout: "osd": "allow *" 2024-03-28T14:08:35.283 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-03-28T14:08:35.283 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-03-28T14:08:35.283 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-03-28T14:08:35.283 INFO:teuthology.orchestra.run.smithi104.stdout:} 2024-03-28T14:08:35.284 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:35.266+0000 7f7f6e89d640 1 -- 172.21.15.104:0/1387561044 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f7f48067fc0 msgr2=0x7f7f4806a480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:35.284 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:35.266+0000 7f7f6e89d640 1 --2- 172.21.15.104:0/1387561044 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f7f48067fc0 0x7f7f4806a480 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f7f60005fd0 tx=0x7f7f60005b80 comp rx=0 tx=0).stop 2024-03-28T14:08:35.285 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:35.266+0000 7f7f6e89d640 1 -- 172.21.15.104:0/1387561044 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7f680732a0 msgr2=0x7f7f68141a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:35.285 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:35.266+0000 7f7f6e89d640 1 --2- 172.21.15.104:0/1387561044 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7f680732a0 0x7f7f68141a30 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f7f500ac040 tx=0x7f7f5001ad10 comp rx=0 tx=0).stop 2024-03-28T14:08:35.285 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:35.266+0000 7f7f6e89d640 1 -- 172.21.15.104:0/1387561044 shutdown_connections 2024-03-28T14:08:35.285 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:35.266+0000 7f7f6e89d640 1 --2- 172.21.15.104:0/1387561044 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f7f6813faf0 0x7f7f68072170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:35.285 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:35.266+0000 7f7f6e89d640 1 --2- 172.21.15.104:0/1387561044 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f7f48067fc0 0x7f7f4806a480 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:35.285 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:35.266+0000 7f7f6e89d640 1 --2- 172.21.15.104:0/1387561044 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7f680732a0 0x7f7f68141a30 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:35.285 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:35.266+0000 7f7f6e89d640 1 --2- 172.21.15.104:0/1387561044 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f7f6806d970 0x7f7f68142760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:35.285 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:35.266+0000 7f7f6e89d640 1 -- 172.21.15.104:0/1387561044 >> 172.21.15.104:0/1387561044 conn(0x7f7f680b58a0 msgr2=0x7f7f680b0e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T14:08:35.285 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:35.266+0000 7f7f6e89d640 1 -- 172.21.15.104:0/1387561044 shutdown_connections 2024-03-28T14:08:35.285 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:35.266+0000 7f7f6e89d640 1 -- 172.21.15.104:0/1387561044 wait complete. 2024-03-28T14:08:35.287 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-mds 2024-03-28T14:08:35.410 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:35.406+0000 7f8f55789640 1 Processor -- start 2024-03-28T14:08:35.410 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:35.406+0000 7f8f55789640 1 -- start start 2024-03-28T14:08:35.411 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:35.406+0000 7f8f55789640 1 --2- >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f8f5006d300 0x7f8f5014bba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:35.412 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:35.406+0000 7f8f55789640 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f8f5014c0e0 0x7f8f5014e4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:35.412 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:35.406+0000 7f8f55789640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8f500af1a0 con 0x7f8f5006cf20 2024-03-28T14:08:35.412 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:35.406+0000 7f8f55789640 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 -- 0x7f8f500afc40 con 0x7f8f5014c0e0 2024-03-28T14:08:35.412 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:35.406+0000 7f8f55789640 1 -- --> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] -- mon_getmap magic: 0 -- 0x7f8f500b0360 con 0x7f8f5006d300 2024-03-28T14:08:35.412 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:35.406+0000 7f8f4effd640 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f8f5006cf20 0x7f8f500aea90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:41668/0 (socket says 172.21.15.104:41668) 2024-03-28T14:08:35.412 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:35.406+0000 7f8f4effd640 1 -- 172.21.15.104:0/335058329 learned_addr learned my addr 172.21.15.104:0/335058329 (peer_addr_for_me v1:172.21.15.104:0/0) 2024-03-28T14:08:35.412 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:35.406+0000 7f8f4f7fe640 1 --2- 172.21.15.104:0/335058329 >> v2:172.21.15.104:3300/0 conn(0x7f8f5014c0e0 0x7f8f5014e4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:35.412 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:35.406+0000 7f8f4e7fc640 1 -- 172.21.15.104:0/335058329 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3451912130 0 0) 0x7f8f500af1a0 con 0x7f8f5006cf20 2024-03-28T14:08:35.412 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:35.406+0000 7f8f4e7fc640 1 -- 172.21.15.104:0/335058329 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8f3c003450 con 0x7f8f5006cf20 2024-03-28T14:08:35.412 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:35.406+0000 7f8f47fff640 1 --2- 172.21.15.104:0/335058329 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f8f5006d300 0x7f8f5014bba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:35.412 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:35.406+0000 7f8f4f7fe640 1 -- 172.21.15.104:0/335058329 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f8f5006d300 msgr2=0x7f8f5014bba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T14:08:35.412 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:35.406+0000 7f8f4f7fe640 1 --2- 172.21.15.104:0/335058329 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f8f5006d300 0x7f8f5014bba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:35.412 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:35.406+0000 7f8f4f7fe640 1 -- 172.21.15.104:0/335058329 >> v1:172.21.15.104:6789/0 conn(0x7f8f5006cf20 legacy=0x7f8f500aea90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:35.413 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:35.406+0000 7f8f4f7fe640 1 -- 172.21.15.104:0/335058329 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8f5009dc50 con 0x7f8f5014c0e0 2024-03-28T14:08:35.413 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:35.406+0000 7f8f47fff640 1 --2- 172.21.15.104:0/335058329 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f8f5006d300 0x7f8f5014bba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T14:08:35.413 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:35.410+0000 7f8f4f7fe640 1 --2- 172.21.15.104:0/335058329 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8f5014c0e0 0x7f8f5014e4d0 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f8f48009d90 tx=0x7f8f4801c820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=534367fa8fb299ea server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:08:35.413 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:35.410+0000 7f8f4e7fc640 1 -- 172.21.15.104:0/335058329 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f480b2070 con 0x7f8f5014c0e0 2024-03-28T14:08:35.413 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:35.410+0000 7f8f4e7fc640 1 -- 172.21.15.104:0/335058329 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8f480a5860 con 0x7f8f5014c0e0 2024-03-28T14:08:35.414 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:35.410+0000 7f8f4e7fc640 1 -- 172.21.15.104:0/335058329 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f480a5c30 con 0x7f8f5014c0e0 2024-03-28T14:08:35.414 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:35.410+0000 7f8f55789640 1 -- 172.21.15.104:0/335058329 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8f5014c0e0 msgr2=0x7f8f5014e4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:35.414 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:35.410+0000 7f8f55789640 1 --2- 172.21.15.104:0/335058329 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8f5014c0e0 0x7f8f5014e4d0 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f8f48009d90 tx=0x7f8f4801c820 comp rx=0 tx=0).stop 2024-03-28T14:08:35.414 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:35.410+0000 7f8f55789640 1 -- 172.21.15.104:0/335058329 shutdown_connections 2024-03-28T14:08:35.415 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:35.410+0000 7f8f55789640 1 --2- 172.21.15.104:0/335058329 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8f5014c0e0 0x7f8f5014e4d0 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:35.415 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:35.410+0000 7f8f55789640 1 --2- 172.21.15.104:0/335058329 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f8f5006d300 0x7f8f5014bba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:35.415 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:35.410+0000 7f8f55789640 1 -- 172.21.15.104:0/335058329 >> 172.21.15.104:0/335058329 conn(0x7f8f500b8f00 msgr2=0x7f8f500b9310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T14:08:35.415 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:35.410+0000 7f8f55789640 1 -- 172.21.15.104:0/335058329 shutdown_connections 2024-03-28T14:08:35.415 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:35.410+0000 7f8f55789640 1 -- 172.21.15.104:0/335058329 wait complete. 2024-03-28T14:08:35.417 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:35.410+0000 7f8f55789640 1 Processor -- start 2024-03-28T14:08:35.417 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:35.410+0000 7f8f55789640 1 -- start start 2024-03-28T14:08:35.417 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:35.414+0000 7f8f55789640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8f5006cf20 0x7f8f5009bcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:35.417 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:35.414+0000 7f8f55789640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f8f5006d300 0x7f8f5009d050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:35.417 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:35.414+0000 7f8f55789640 1 --2- >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f8f5014c0e0 0x7f8f501df840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:35.417 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:35.414+0000 7f8f55789640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f8f500b4c70 con 0x7f8f5006cf20 2024-03-28T14:08:35.417 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:35.414+0000 7f8f55789640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f8f500ae210 con 0x7f8f5006d300 2024-03-28T14:08:35.417 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:35.414+0000 7f8f55789640 1 -- --> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] -- mon_getmap magic: 0 -- 0x7f8f500b4df0 con 0x7f8f5014c0e0 2024-03-28T14:08:35.417 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:35.414+0000 7f8f4effd640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8f5006cf20 0x7f8f5009bcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:35.417 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:35.414+0000 7f8f4effd640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8f5006cf20 0x7f8f5009bcd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:59122/0 (socket says 172.21.15.104:59122) 2024-03-28T14:08:35.418 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:35.414+0000 7f8f4effd640 1 -- 172.21.15.104:0/1799004072 learned_addr learned my addr 172.21.15.104:0/1799004072 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-03-28T14:08:35.418 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:35.414+0000 7f8f4effd640 1 -- 172.21.15.104:0/1799004072 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f8f5014c0e0 msgr2=0x7f8f501df840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:35.418 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:35.414+0000 7f8f4effd640 1 --2- 172.21.15.104:0/1799004072 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f8f5014c0e0 0x7f8f501df840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:35.418 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:35.414+0000 7f8f4effd640 1 -- 172.21.15.104:0/1799004072 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f8f5006d300 msgr2=0x7f8f5009d050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:35.418 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:35.414+0000 7f8f4effd640 1 --2- 172.21.15.104:0/1799004072 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f8f5006d300 0x7f8f5009d050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:35.418 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:35.414+0000 7f8f4effd640 1 -- 172.21.15.104:0/1799004072 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8f5009d590 con 0x7f8f5006cf20 2024-03-28T14:08:35.418 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:35.414+0000 7f8f4effd640 1 --2- 172.21.15.104:0/1799004072 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8f5006cf20 0x7f8f5009bcd0 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f8f4000c400 tx=0x7f8f4000c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:08:35.418 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:35.414+0000 7f8f477fe640 1 -- 172.21.15.104:0/1799004072 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f40017070 con 0x7f8f5006cf20 2024-03-28T14:08:35.418 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:35.414+0000 7f8f477fe640 1 -- 172.21.15.104:0/1799004072 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8f40005d10 con 0x7f8f5006cf20 2024-03-28T14:08:35.418 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:35.414+0000 7f8f55789640 1 -- 172.21.15.104:0/1799004072 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8f5006fa00 con 0x7f8f5006cf20 2024-03-28T14:08:35.418 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:35.414+0000 7f8f477fe640 1 -- 172.21.15.104:0/1799004072 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f40007310 con 0x7f8f5006cf20 2024-03-28T14:08:35.418 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:35.414+0000 7f8f55789640 1 -- 172.21.15.104:0/1799004072 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8f5009b1c0 con 0x7f8f5006cf20 2024-03-28T14:08:35.419 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:35.414+0000 7f8f55789640 1 -- 172.21.15.104:0/1799004072 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8f500687c0 con 0x7f8f5006cf20 2024-03-28T14:08:35.419 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:35.414+0000 7f8f477fe640 1 -- 172.21.15.104:0/1799004072 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) ==== 87803+0+0 (secure 0 0 0) 0x7f8f4001a070 con 0x7f8f5006cf20 2024-03-28T14:08:35.419 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:35.414+0000 7f8f477fe640 1 --2- 172.21.15.104:0/1799004072 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f8f30067fc0 0x7f8f3006a480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:35.419 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:35.414+0000 7f8f477fe640 1 -- 172.21.15.104:0/1799004072 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 4857+0+0 (secure 0 0 0) 0x7f8f40085540 con 0x7f8f5006cf20 2024-03-28T14:08:35.419 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:35.414+0000 7f8f47fff640 1 --2- 172.21.15.104:0/1799004072 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f8f30067fc0 0x7f8f3006a480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:35.420 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:35.418+0000 7f8f47fff640 1 --2- 172.21.15.104:0/1799004072 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f8f30067fc0 0x7f8f3006a480 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f8f50147450 tx=0x7f8f38008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:08:35.423 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:35.418+0000 7f8f477fe640 1 -- 172.21.15.104:0/1799004072 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179117 (secure 0 0 0) 0x7f8f400560d0 con 0x7f8f5006cf20 2024-03-28T14:08:35.646 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:35.642+0000 7f8f55789640 1 -- 172.21.15.104:0/1799004072 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-mds"} v 0) -- 0x7f8f500b7700 con 0x7f8f5006cf20 2024-03-28T14:08:35.650 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:35.646+0000 7f8f477fe640 1 -- 172.21.15.104:0/1799004072 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-mds"}]=0 v5) ==== 90+0+0 (secure 0 0 0) 0x7f8f400559d0 con 0x7f8f5006cf20 2024-03-28T14:08:35.652 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:35.650+0000 7f8f55789640 1 -- 172.21.15.104:0/1799004072 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f8f30067fc0 msgr2=0x7f8f3006a480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:35.652 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:35.650+0000 7f8f55789640 1 --2- 172.21.15.104:0/1799004072 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f8f30067fc0 0x7f8f3006a480 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f8f50147450 tx=0x7f8f38008040 comp rx=0 tx=0).stop 2024-03-28T14:08:35.652 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:35.650+0000 7f8f55789640 1 -- 172.21.15.104:0/1799004072 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8f5006cf20 msgr2=0x7f8f5009bcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:35.652 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:35.650+0000 7f8f55789640 1 --2- 172.21.15.104:0/1799004072 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8f5006cf20 0x7f8f5009bcd0 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f8f4000c400 tx=0x7f8f4000c8c0 comp rx=0 tx=0).stop 2024-03-28T14:08:35.652 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:35.650+0000 7f8f55789640 1 -- 172.21.15.104:0/1799004072 shutdown_connections 2024-03-28T14:08:35.653 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:35.650+0000 7f8f55789640 1 --2- 172.21.15.104:0/1799004072 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f8f5014c0e0 0x7f8f501df840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:35.653 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:35.650+0000 7f8f55789640 1 --2- 172.21.15.104:0/1799004072 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f8f5006d300 0x7f8f5009d050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:35.653 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:35.650+0000 7f8f55789640 1 --2- 172.21.15.104:0/1799004072 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f8f30067fc0 0x7f8f3006a480 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:35.653 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:35.650+0000 7f8f55789640 1 --2- 172.21.15.104:0/1799004072 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8f5006cf20 0x7f8f5009bcd0 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:35.653 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:35.650+0000 7f8f55789640 1 -- 172.21.15.104:0/1799004072 >> 172.21.15.104:0/1799004072 conn(0x7f8f500b8f00 msgr2=0x7f8f500b5f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T14:08:35.653 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:35.650+0000 7f8f55789640 1 -- 172.21.15.104:0/1799004072 shutdown_connections 2024-03-28T14:08:35.653 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:35.650+0000 7f8f55789640 1 -- 172.21.15.104:0/1799004072 wait complete. 2024-03-28T14:08:35.669 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-mgr 2024-03-28T14:08:35.785 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:35.782+0000 7fb6614cc640 1 Processor -- start 2024-03-28T14:08:35.785 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:35.782+0000 7fb6614cc640 1 -- start start 2024-03-28T14:08:35.788 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:35.782+0000 7fb6614cc640 1 --2- >> v2:172.21.15.171:3300/0 conn(0x7fb65c0b6d50 0x7fb65c073370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:35.788 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:35.782+0000 7fb6614cc640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb65c0b5b90 con 0x7fb65c0b66b0 2024-03-28T14:08:35.788 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:35.782+0000 7fb6614cc640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb65c0b6a90 con 0x7fb65c145600 2024-03-28T14:08:35.788 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:35.782+0000 7fb6614cc640 1 -- --> v2:172.21.15.171:3300/0 -- mon_getmap magic: 0 -- 0x7fb65c0b7270 con 0x7fb65c0b6d50 2024-03-28T14:08:35.788 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:35.782+0000 7fb65affd640 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7fb65c0b66b0 0x7fb65c142820 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:41684/0 (socket says 172.21.15.104:41684) 2024-03-28T14:08:35.789 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:35.782+0000 7fb65affd640 1 -- 172.21.15.104:0/1664540459 learned_addr learned my addr 172.21.15.104:0/1664540459 (peer_addr_for_me v1:172.21.15.104:0/0) 2024-03-28T14:08:35.789 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:35.782+0000 7fb659ffb640 1 -- 172.21.15.104:0/1664540459 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 102956984 0 0) 0x7fb65c0b5b90 con 0x7fb65c0b66b0 2024-03-28T14:08:35.789 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:35.782+0000 7fb659ffb640 1 -- 172.21.15.104:0/1664540459 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb648003530 con 0x7fb65c0b66b0 2024-03-28T14:08:35.789 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:35.782+0000 7fb65a7fc640 1 --2- 172.21.15.104:0/1664540459 >> v2:172.21.15.171:3300/0 conn(0x7fb65c0b6d50 0x7fb65c073370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:35.789 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:35.782+0000 7fb659ffb640 1 -- 172.21.15.104:0/1664540459 <== mon.0 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1455438116 0 0) 0x7fb648003530 con 0x7fb65c0b66b0 2024-03-28T14:08:35.789 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:35.782+0000 7fb659ffb640 1 -- 172.21.15.104:0/1664540459 >> v2:172.21.15.171:3300/0 conn(0x7fb65c0b6d50 msgr2=0x7fb65c073370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T14:08:35.789 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:35.782+0000 7fb659ffb640 1 --2- 172.21.15.104:0/1664540459 >> v2:172.21.15.171:3300/0 conn(0x7fb65c0b6d50 0x7fb65c073370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:35.789 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:35.782+0000 7fb659ffb640 1 -- 172.21.15.104:0/1664540459 >> v1:172.21.15.171:6789/0 conn(0x7fb65c145600 legacy=0x7fb65c14fea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T14:08:35.789 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:35.782+0000 7fb659ffb640 1 -- 172.21.15.104:0/1664540459 --> v1:172.21.15.104:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb65c06ed90 con 0x7fb65c0b66b0 2024-03-28T14:08:35.789 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:35.782+0000 7fb659ffb640 1 -- 172.21.15.104:0/1664540459 <== mon.0 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3745580849 0 0) 0x7fb654001680 con 0x7fb65c0b66b0 2024-03-28T14:08:35.789 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:35.782+0000 7fb659ffb640 1 -- 172.21.15.104:0/1664540459 >> v1:172.21.15.104:6789/0 conn(0x7fb65c0b66b0 legacy=0x7fb65c142820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:35.789 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:35.782+0000 7fb659ffb640 1 --2- 172.21.15.104:0/1664540459 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fb6480039c0 0x7fb648003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:35.790 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:35.782+0000 7fb659ffb640 1 --2- 172.21.15.104:0/1664540459 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb648004810 0x7fb648004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:35.790 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:35.782+0000 7fb659ffb640 1 --2- 172.21.15.104:0/1664540459 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fb6480052f0 0x7fb64800b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:35.790 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:35.782+0000 7fb659ffb640 1 -- 172.21.15.104:0/1664540459 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7fb65c0b7270 con 0x7fb648004810 2024-03-28T14:08:35.790 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:35.782+0000 7fb659ffb640 1 -- 172.21.15.104:0/1664540459 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7fb64800c1a0 con 0x7fb6480039c0 2024-03-28T14:08:35.790 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:35.782+0000 7fb659ffb640 1 -- 172.21.15.104:0/1664540459 --> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] -- mon_getmap magic: 0 -- 0x7fb64800c3a0 con 0x7fb6480052f0 2024-03-28T14:08:35.790 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:35.782+0000 7fb65a7fc640 1 --2- 172.21.15.104:0/1664540459 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb648004810 0x7fb648004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:35.790 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:35.782+0000 7fb65affd640 1 --2- 172.21.15.104:0/1664540459 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fb6480039c0 0x7fb648003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:35.790 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:35.782+0000 7fb65b7fe640 1 --2- 172.21.15.104:0/1664540459 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fb6480052f0 0x7fb64800b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:35.791 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:35.782+0000 7fb65a7fc640 1 -- 172.21.15.104:0/1664540459 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fb6480052f0 msgr2=0x7fb64800b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T14:08:35.791 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:35.782+0000 7fb65a7fc640 1 --2- 172.21.15.104:0/1664540459 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fb6480052f0 0x7fb64800b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:35.791 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:35.782+0000 7fb65a7fc640 1 -- 172.21.15.104:0/1664540459 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fb6480039c0 msgr2=0x7fb648003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T14:08:35.791 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:35.782+0000 7fb65a7fc640 1 --2- 172.21.15.104:0/1664540459 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fb6480039c0 0x7fb648003dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:35.791 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:35.782+0000 7fb65a7fc640 1 -- 172.21.15.104:0/1664540459 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb648003530 con 0x7fb648004810 2024-03-28T14:08:35.791 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:35.782+0000 7fb65b7fe640 1 --2- 172.21.15.104:0/1664540459 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fb6480052f0 0x7fb64800b9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T14:08:35.791 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:35.782+0000 7fb65b7fe640 1 -- 172.21.15.104:0/1664540459 reap_dead start 2024-03-28T14:08:35.795 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:35.786+0000 7fb65a7fc640 1 --2- 172.21.15.104:0/1664540459 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb648004810 0x7fb648004c90 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7fb644007330 tx=0x7fb64401ce00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8e7708ab17da601b server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:08:35.795 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:35.786+0000 7fb659ffb640 1 -- 172.21.15.104:0/1664540459 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6440b2070 con 0x7fb648004810 2024-03-28T14:08:35.795 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:35.786+0000 7fb659ffb640 1 -- 172.21.15.104:0/1664540459 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb6440a5ae0 con 0x7fb648004810 2024-03-28T14:08:35.795 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:35.786+0000 7fb659ffb640 1 -- 172.21.15.104:0/1664540459 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb644009be0 con 0x7fb648004810 2024-03-28T14:08:35.795 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:35.786+0000 7fb6614cc640 1 -- 172.21.15.104:0/1664540459 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb648004810 msgr2=0x7fb648004c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:35.795 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:35.786+0000 7fb6614cc640 1 --2- 172.21.15.104:0/1664540459 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb648004810 0x7fb648004c90 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7fb644007330 tx=0x7fb64401ce00 comp rx=0 tx=0).stop 2024-03-28T14:08:35.795 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:35.786+0000 7fb6614cc640 1 -- 172.21.15.104:0/1664540459 shutdown_connections 2024-03-28T14:08:35.796 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:35.786+0000 7fb6614cc640 1 --2- 172.21.15.104:0/1664540459 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb648004810 0x7fb648004c90 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:35.796 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:35.786+0000 7fb6614cc640 1 -- 172.21.15.104:0/1664540459 >> 172.21.15.104:0/1664540459 conn(0x7fb65c02c600 msgr2=0x7fb65c02ca10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T14:08:35.796 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:35.786+0000 7fb6614cc640 1 -- 172.21.15.104:0/1664540459 shutdown_connections 2024-03-28T14:08:35.796 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:35.786+0000 7fb6614cc640 1 -- 172.21.15.104:0/1664540459 wait complete. 2024-03-28T14:08:35.796 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:35.786+0000 7fb6614cc640 1 Processor -- start 2024-03-28T14:08:35.796 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:35.786+0000 7fb6614cc640 1 -- start start 2024-03-28T14:08:35.796 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:35.786+0000 7fb6614cc640 1 --2- >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fb648004810 0x7fb65c147cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:35.796 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:35.786+0000 7fb6614cc640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fb65c140d90 0x7fb65c141210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:35.796 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:35.786+0000 7fb6614cc640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb65c0b8eb0 0x7fb65c0b9330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:35.796 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:35.786+0000 7fb6614cc640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7fb65c148cf0 con 0x7fb65c0b8eb0 2024-03-28T14:08:35.796 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:35.786+0000 7fb6614cc640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7fb65c148b70 con 0x7fb65c140d90 2024-03-28T14:08:35.796 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:35.786+0000 7fb6614cc640 1 -- --> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] -- mon_getmap magic: 0 -- 0x7fb65c148e70 con 0x7fb648004810 2024-03-28T14:08:35.797 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:35.786+0000 7fb65b7fe640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb65c0b8eb0 0x7fb65c0b9330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:35.797 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:35.786+0000 7fb65b7fe640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb65c0b8eb0 0x7fb65c0b9330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:59146/0 (socket says 172.21.15.104:59146) 2024-03-28T14:08:35.797 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:35.786+0000 7fb65b7fe640 1 -- 172.21.15.104:0/1382416903 learned_addr learned my addr 172.21.15.104:0/1382416903 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-03-28T14:08:35.797 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:35.790+0000 7fb65affd640 1 --2- 172.21.15.104:0/1382416903 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fb648004810 0x7fb65c147cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:35.797 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:35.790+0000 7fb65a7fc640 1 --2- 172.21.15.104:0/1382416903 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fb65c140d90 0x7fb65c141210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:35.797 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:35.790+0000 7fb65b7fe640 1 -- 172.21.15.104:0/1382416903 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fb648004810 msgr2=0x7fb65c147cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:35.797 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:35.790+0000 7fb65b7fe640 1 --2- 172.21.15.104:0/1382416903 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fb648004810 0x7fb65c147cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:35.797 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:35.790+0000 7fb65b7fe640 1 -- 172.21.15.104:0/1382416903 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fb65c140d90 msgr2=0x7fb65c141210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:35.797 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:35.790+0000 7fb65b7fe640 1 --2- 172.21.15.104:0/1382416903 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fb65c140d90 0x7fb65c141210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:35.797 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:35.790+0000 7fb65b7fe640 1 -- 172.21.15.104:0/1382416903 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb65c0bad00 con 0x7fb65c0b8eb0 2024-03-28T14:08:35.797 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:35.790+0000 7fb65a7fc640 1 --2- 172.21.15.104:0/1382416903 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fb65c140d90 0x7fb65c141210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T14:08:35.798 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:35.790+0000 7fb65b7fe640 1 --2- 172.21.15.104:0/1382416903 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb65c0b8eb0 0x7fb65c0b9330 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7fb65c073090 tx=0x7fb64c00d660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:08:35.798 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:35.790+0000 7fb6537fe640 1 -- 172.21.15.104:0/1382416903 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb64c016030 con 0x7fb65c0b8eb0 2024-03-28T14:08:35.798 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:35.790+0000 7fb6537fe640 1 -- 172.21.15.104:0/1382416903 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb64c007020 con 0x7fb65c0b8eb0 2024-03-28T14:08:35.798 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:35.790+0000 7fb6614cc640 1 -- 172.21.15.104:0/1382416903 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb65c0ba9d0 con 0x7fb65c0b8eb0 2024-03-28T14:08:35.798 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:35.790+0000 7fb6537fe640 1 -- 172.21.15.104:0/1382416903 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb64c0073f0 con 0x7fb65c0b8eb0 2024-03-28T14:08:35.798 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:35.790+0000 7fb6614cc640 1 -- 172.21.15.104:0/1382416903 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb65c0a1130 con 0x7fb65c0b8eb0 2024-03-28T14:08:35.798 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:35.790+0000 7fb6614cc640 1 -- 172.21.15.104:0/1382416903 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb65c0687c0 con 0x7fb65c0b8eb0 2024-03-28T14:08:35.798 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:35.790+0000 7fb6537fe640 1 -- 172.21.15.104:0/1382416903 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) ==== 87803+0+0 (secure 0 0 0) 0x7fb64c019070 con 0x7fb65c0b8eb0 2024-03-28T14:08:35.798 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:35.790+0000 7fb6537fe640 1 --2- 172.21.15.104:0/1382416903 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7fb63c067fc0 0x7fb63c06a480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:35.798 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:35.790+0000 7fb6537fe640 1 -- 172.21.15.104:0/1382416903 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 4857+0+0 (secure 0 0 0) 0x7fb64c085540 con 0x7fb65c0b8eb0 2024-03-28T14:08:35.798 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:35.790+0000 7fb65affd640 1 --2- 172.21.15.104:0/1382416903 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7fb63c067fc0 0x7fb63c06a480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:35.798 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:35.790+0000 7fb65affd640 1 --2- 172.21.15.104:0/1382416903 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7fb63c067fc0 0x7fb63c06a480 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fb654001460 tx=0x7fb654009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:08:35.799 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:35.794+0000 7fb6537fe640 1 -- 172.21.15.104:0/1382416903 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179117 (secure 0 0 0) 0x7fb64c0560d0 con 0x7fb65c0b8eb0 2024-03-28T14:08:36.020 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:36.018+0000 7fb6614cc640 1 -- 172.21.15.104:0/1382416903 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-mgr"} v 0) -- 0x7fb65c13f580 con 0x7fb65c0b8eb0 2024-03-28T14:08:36.023 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:36.018+0000 7fb6537fe640 1 -- 172.21.15.104:0/1382416903 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-mgr"}]=0 v6) ==== 90+0+0 (secure 0 0 0) 0x7fb64c0559d0 con 0x7fb65c0b8eb0 2024-03-28T14:08:36.026 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:36.022+0000 7fb6614cc640 1 -- 172.21.15.104:0/1382416903 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7fb63c067fc0 msgr2=0x7fb63c06a480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:36.026 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:36.022+0000 7fb6614cc640 1 --2- 172.21.15.104:0/1382416903 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7fb63c067fc0 0x7fb63c06a480 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fb654001460 tx=0x7fb654009040 comp rx=0 tx=0).stop 2024-03-28T14:08:36.026 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:36.022+0000 7fb6614cc640 1 -- 172.21.15.104:0/1382416903 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb65c0b8eb0 msgr2=0x7fb65c0b9330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:36.026 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:36.022+0000 7fb6614cc640 1 --2- 172.21.15.104:0/1382416903 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb65c0b8eb0 0x7fb65c0b9330 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7fb65c073090 tx=0x7fb64c00d660 comp rx=0 tx=0).stop 2024-03-28T14:08:36.026 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:36.022+0000 7fb6614cc640 1 -- 172.21.15.104:0/1382416903 shutdown_connections 2024-03-28T14:08:36.026 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:36.022+0000 7fb6614cc640 1 --2- 172.21.15.104:0/1382416903 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7fb63c067fc0 0x7fb63c06a480 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:36.026 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:36.022+0000 7fb6614cc640 1 --2- 172.21.15.104:0/1382416903 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb65c0b8eb0 0x7fb65c0b9330 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:36.026 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:36.022+0000 7fb6614cc640 1 --2- 172.21.15.104:0/1382416903 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fb65c140d90 0x7fb65c141210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:36.027 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:36.022+0000 7fb6614cc640 1 --2- 172.21.15.104:0/1382416903 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fb648004810 0x7fb65c147cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:36.027 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:36.022+0000 7fb6614cc640 1 -- 172.21.15.104:0/1382416903 >> 172.21.15.104:0/1382416903 conn(0x7fb65c02c600 msgr2=0x7fb65c143bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T14:08:36.027 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:36.022+0000 7fb6614cc640 1 -- 172.21.15.104:0/1382416903 shutdown_connections 2024-03-28T14:08:36.027 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:36.022+0000 7fb6614cc640 1 -- 172.21.15.104:0/1382416903 wait complete. 2024-03-28T14:08:36.043 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-osd 2024-03-28T14:08:36.154 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:36.150+0000 7f7dc15e6640 1 Processor -- start 2024-03-28T14:08:36.154 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:36.150+0000 7f7dc15e6640 1 -- start start 2024-03-28T14:08:36.157 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:36.150+0000 7f7dc15e6640 1 --2- >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f7dbc0b2740 0x7f7dbc0b3730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:36.157 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:36.150+0000 7f7dc15e6640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7dbc06ccd0 con 0x7f7dbc149380 2024-03-28T14:08:36.157 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:36.150+0000 7f7dc15e6640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7dbc0b1980 con 0x7f7dbc06c8f0 2024-03-28T14:08:36.157 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:36.150+0000 7f7dc15e6640 1 -- --> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] -- mon_getmap magic: 0 -- 0x7f7dbc0b0b30 con 0x7f7dbc0b2740 2024-03-28T14:08:36.157 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:36.150+0000 7f7dbb7fe640 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f7dbc149380 0x7f7dbc1501b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:41690/0 (socket says 172.21.15.104:41690) 2024-03-28T14:08:36.157 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:36.150+0000 7f7dbb7fe640 1 -- 172.21.15.104:0/1459053861 learned_addr learned my addr 172.21.15.104:0/1459053861 (peer_addr_for_me v1:172.21.15.104:0/0) 2024-03-28T14:08:36.157 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:36.150+0000 7f7dba7fc640 1 --2- 172.21.15.104:0/1459053861 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f7dbc0b2740 0x7f7dbc0b3730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:36.158 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:36.150+0000 7f7db9ffb640 1 -- 172.21.15.104:0/1459053861 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4255926222 0 0) 0x7f7dbc06ccd0 con 0x7f7dbc149380 2024-03-28T14:08:36.158 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:36.154+0000 7f7db9ffb640 1 -- 172.21.15.104:0/1459053861 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7da4003530 con 0x7f7dbc149380 2024-03-28T14:08:36.158 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:36.154+0000 7f7db9ffb640 1 -- 172.21.15.104:0/1459053861 <== mon.0 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 350067573 0 0) 0x7f7da4003530 con 0x7f7dbc149380 2024-03-28T14:08:36.158 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:36.154+0000 7f7db9ffb640 1 -- 172.21.15.104:0/1459053861 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f7dbc0b2740 msgr2=0x7f7dbc0b3730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T14:08:36.158 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:36.154+0000 7f7db9ffb640 1 --2- 172.21.15.104:0/1459053861 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f7dbc0b2740 0x7f7dbc0b3730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:36.158 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:36.154+0000 7f7db9ffb640 1 -- 172.21.15.104:0/1459053861 >> v1:172.21.15.171:6789/0 conn(0x7f7dbc06c8f0 legacy=0x7f7dbc148c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:36.158 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:36.154+0000 7f7db9ffb640 1 -- 172.21.15.104:0/1459053861 --> v1:172.21.15.104:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7dbc0b0850 con 0x7f7dbc149380 2024-03-28T14:08:36.158 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:36.154+0000 7f7db9ffb640 1 -- 172.21.15.104:0/1459053861 <== mon.0 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3745580849 0 0) 0x7f7db4001680 con 0x7f7dbc149380 2024-03-28T14:08:36.158 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:36.154+0000 7f7db9ffb640 1 -- 172.21.15.104:0/1459053861 >> v1:172.21.15.104:6789/0 conn(0x7f7dbc149380 legacy=0x7f7dbc1501b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:36.158 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:36.154+0000 7f7db9ffb640 1 --2- 172.21.15.104:0/1459053861 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f7da40039f0 0x7f7da4003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:36.158 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:36.154+0000 7f7db9ffb640 1 --2- 172.21.15.104:0/1459053861 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f7dbc0b2740 0x7f7da4004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:36.158 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:36.154+0000 7f7db9ffb640 1 --2- 172.21.15.104:0/1459053861 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7da4004d50 0x7f7da40051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:36.158 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:36.154+0000 7f7db9ffb640 1 -- 172.21.15.104:0/1459053861 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f7dbc0b0b30 con 0x7f7da4004d50 2024-03-28T14:08:36.159 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:36.154+0000 7f7db9ffb640 1 -- 172.21.15.104:0/1459053861 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f7da400bb30 con 0x7f7da40039f0 2024-03-28T14:08:36.159 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:36.154+0000 7f7db9ffb640 1 -- 172.21.15.104:0/1459053861 --> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] -- mon_getmap magic: 0 -- 0x7f7da400bd30 con 0x7f7dbc0b2740 2024-03-28T14:08:36.159 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:36.154+0000 7f7dbb7fe640 1 --2- 172.21.15.104:0/1459053861 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7da4004d50 0x7f7da40051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:36.159 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:36.154+0000 7f7dbaffd640 1 --2- 172.21.15.104:0/1459053861 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f7da40039f0 0x7f7da4003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:36.159 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:36.154+0000 7f7dba7fc640 1 --2- 172.21.15.104:0/1459053861 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f7dbc0b2740 0x7f7da4004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:36.159 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:36.154+0000 7f7dbb7fe640 1 -- 172.21.15.104:0/1459053861 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f7dbc0b2740 msgr2=0x7f7da4004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T14:08:36.159 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:36.154+0000 7f7dbb7fe640 1 --2- 172.21.15.104:0/1459053861 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f7dbc0b2740 0x7f7da4004810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:36.159 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:36.154+0000 7f7dbb7fe640 1 -- 172.21.15.104:0/1459053861 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f7da40039f0 msgr2=0x7f7da4003e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T14:08:36.159 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:36.154+0000 7f7dbb7fe640 1 --2- 172.21.15.104:0/1459053861 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f7da40039f0 0x7f7da4003e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:36.159 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:36.154+0000 7f7dbb7fe640 1 -- 172.21.15.104:0/1459053861 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7da4003530 con 0x7f7da4004d50 2024-03-28T14:08:36.162 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:36.154+0000 7f7dbb7fe640 1 --2- 172.21.15.104:0/1459053861 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7da4004d50 0x7f7da40051d0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f7db400c240 tx=0x7f7db401f3f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=75f5d32ee421d9d4 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:08:36.162 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:36.154+0000 7f7db9ffb640 1 -- 172.21.15.104:0/1459053861 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7db4007d20 con 0x7f7da4004d50 2024-03-28T14:08:36.163 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:36.154+0000 7f7db9ffb640 1 -- 172.21.15.104:0/1459053861 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7db4005ae0 con 0x7f7da4004d50 2024-03-28T14:08:36.163 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:36.154+0000 7f7db9ffb640 1 -- 172.21.15.104:0/1459053861 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7db4007070 con 0x7f7da4004d50 2024-03-28T14:08:36.163 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:36.154+0000 7f7dc15e6640 1 -- 172.21.15.104:0/1459053861 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7da4004d50 msgr2=0x7f7da40051d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:36.163 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:36.154+0000 7f7dc15e6640 1 --2- 172.21.15.104:0/1459053861 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7da4004d50 0x7f7da40051d0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f7db400c240 tx=0x7f7db401f3f0 comp rx=0 tx=0).stop 2024-03-28T14:08:36.163 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:36.154+0000 7f7dbb7fe640 1 -- 172.21.15.104:0/1459053861 reap_dead start 2024-03-28T14:08:36.163 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:36.154+0000 7f7dc15e6640 1 -- 172.21.15.104:0/1459053861 shutdown_connections 2024-03-28T14:08:36.163 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:36.154+0000 7f7dc15e6640 1 -- 172.21.15.104:0/1459053861 >> 172.21.15.104:0/1459053861 conn(0x7f7dbc0b8f00 msgr2=0x7f7dbc0b9310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T14:08:36.163 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:36.154+0000 7f7dc15e6640 1 -- 172.21.15.104:0/1459053861 shutdown_connections 2024-03-28T14:08:36.163 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:36.154+0000 7f7dc15e6640 1 -- 172.21.15.104:0/1459053861 wait complete. 2024-03-28T14:08:36.163 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:36.158+0000 7f7dc15e6640 1 Processor -- start 2024-03-28T14:08:36.163 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:36.158+0000 7f7dc15e6640 1 -- start start 2024-03-28T14:08:36.163 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:36.158+0000 7f7dc15e6640 1 --2- >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f7dbc144310 0x7f7dbc1cf910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:36.163 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:36.158+0000 7f7dc15e6640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7dbc1d0720 0x7f7dbc09ca30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:36.164 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:36.158+0000 7f7dc15e6640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f7dbc1de0f0 0x7f7dbc09cf70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:36.164 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:36.158+0000 7f7dc15e6640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f7dbc0b2b40 con 0x7f7dbc1d0720 2024-03-28T14:08:36.164 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:36.158+0000 7f7dc15e6640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f7dbc0b22b0 con 0x7f7dbc1de0f0 2024-03-28T14:08:36.164 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:36.158+0000 7f7dc15e6640 1 -- --> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] -- mon_getmap magic: 0 -- 0x7f7dbc0b14e0 con 0x7f7dbc144310 2024-03-28T14:08:36.164 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:36.158+0000 7f7dba7fc640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7dbc1d0720 0x7f7dbc09ca30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:36.164 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:36.158+0000 7f7dba7fc640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7dbc1d0720 0x7f7dbc09ca30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:59164/0 (socket says 172.21.15.104:59164) 2024-03-28T14:08:36.164 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:36.158+0000 7f7dba7fc640 1 -- 172.21.15.104:0/3020934810 learned_addr learned my addr 172.21.15.104:0/3020934810 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-03-28T14:08:36.164 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:36.158+0000 7f7dba7fc640 1 -- 172.21.15.104:0/3020934810 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f7dbc144310 msgr2=0x7f7dbc1cf910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:36.164 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:36.158+0000 7f7dbaffd640 1 --2- 172.21.15.104:0/3020934810 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f7dbc144310 0x7f7dbc1cf910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:36.164 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:36.158+0000 7f7dbb7fe640 1 --2- 172.21.15.104:0/3020934810 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f7dbc1de0f0 0x7f7dbc09cf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:36.164 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:36.158+0000 7f7dba7fc640 1 --2- 172.21.15.104:0/3020934810 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f7dbc144310 0x7f7dbc1cf910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:36.164 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:36.158+0000 7f7dba7fc640 1 -- 172.21.15.104:0/3020934810 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f7dbc1de0f0 msgr2=0x7f7dbc09cf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:36.164 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:36.158+0000 7f7dba7fc640 1 --2- 172.21.15.104:0/3020934810 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f7dbc1de0f0 0x7f7dbc09cf70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:36.164 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:36.158+0000 7f7dba7fc640 1 -- 172.21.15.104:0/3020934810 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7dbc09d9d0 con 0x7f7dbc1d0720 2024-03-28T14:08:36.165 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:36.158+0000 7f7dba7fc640 1 --2- 172.21.15.104:0/3020934810 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7dbc1d0720 0x7f7dbc09ca30 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f7db00108c0 tx=0x7f7db0010d80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:08:36.165 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:36.158+0000 7f7da37fe640 1 -- 172.21.15.104:0/3020934810 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7db000f040 con 0x7f7dbc1d0720 2024-03-28T14:08:36.165 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:36.158+0000 7f7da37fe640 1 -- 172.21.15.104:0/3020934810 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7db000eb60 con 0x7f7dbc1d0720 2024-03-28T14:08:36.165 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:36.158+0000 7f7dc15e6640 1 -- 172.21.15.104:0/3020934810 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7dbc1ce980 con 0x7f7dbc1d0720 2024-03-28T14:08:36.165 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:36.158+0000 7f7da37fe640 1 -- 172.21.15.104:0/3020934810 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7db001c400 con 0x7f7dbc1d0720 2024-03-28T14:08:36.165 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:36.158+0000 7f7dc15e6640 1 -- 172.21.15.104:0/3020934810 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7dbc1408f0 con 0x7f7dbc1d0720 2024-03-28T14:08:36.165 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:36.158+0000 7f7dc15e6640 1 -- 172.21.15.104:0/3020934810 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7dbc0687c0 con 0x7f7dbc1d0720 2024-03-28T14:08:36.165 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:36.158+0000 7f7da37fe640 1 -- 172.21.15.104:0/3020934810 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) ==== 87803+0+0 (secure 0 0 0) 0x7f7db001c5a0 con 0x7f7dbc1d0720 2024-03-28T14:08:36.166 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:36.158+0000 7f7da37fe640 1 --2- 172.21.15.104:0/3020934810 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f7da8067ff0 0x7f7da806a4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:36.166 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:36.158+0000 7f7da37fe640 1 -- 172.21.15.104:0/3020934810 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 4857+0+0 (secure 0 0 0) 0x7f7db00867c0 con 0x7f7dbc1d0720 2024-03-28T14:08:36.166 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:36.162+0000 7f7dbaffd640 1 --2- 172.21.15.104:0/3020934810 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f7da8067ff0 0x7f7da806a4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:36.166 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:36.162+0000 7f7dbaffd640 1 --2- 172.21.15.104:0/3020934810 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f7da8067ff0 0x7f7da806a4b0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f7dac001d20 tx=0x7f7dac009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:08:36.167 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:36.162+0000 7f7da37fe640 1 -- 172.21.15.104:0/3020934810 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179117 (secure 0 0 0) 0x7f7db0018030 con 0x7f7dbc1d0720 2024-03-28T14:08:36.386 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:36.382+0000 7f7dc15e6640 1 -- 172.21.15.104:0/3020934810 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-osd"} v 0) -- 0x7f7dbc0b8800 con 0x7f7dbc1d0720 2024-03-28T14:08:36.389 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:36.386+0000 7f7da37fe640 1 -- 172.21.15.104:0/3020934810 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-osd"}]=0 v7) ==== 90+0+0 (secure 0 0 0) 0x7f7db0057350 con 0x7f7dbc1d0720 2024-03-28T14:08:36.392 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:36.390+0000 7f7dc15e6640 1 -- 172.21.15.104:0/3020934810 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f7da8067ff0 msgr2=0x7f7da806a4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:36.393 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:36.390+0000 7f7dc15e6640 1 --2- 172.21.15.104:0/3020934810 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f7da8067ff0 0x7f7da806a4b0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f7dac001d20 tx=0x7f7dac009040 comp rx=0 tx=0).stop 2024-03-28T14:08:36.393 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:36.390+0000 7f7dc15e6640 1 -- 172.21.15.104:0/3020934810 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7dbc1d0720 msgr2=0x7f7dbc09ca30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:36.393 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:36.390+0000 7f7dc15e6640 1 --2- 172.21.15.104:0/3020934810 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7dbc1d0720 0x7f7dbc09ca30 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f7db00108c0 tx=0x7f7db0010d80 comp rx=0 tx=0).stop 2024-03-28T14:08:36.393 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:36.390+0000 7f7dc15e6640 1 -- 172.21.15.104:0/3020934810 shutdown_connections 2024-03-28T14:08:36.393 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:36.390+0000 7f7dc15e6640 1 --2- 172.21.15.104:0/3020934810 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f7dbc1de0f0 0x7f7dbc09cf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:36.393 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:36.390+0000 7f7dc15e6640 1 --2- 172.21.15.104:0/3020934810 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f7da8067ff0 0x7f7da806a4b0 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:36.393 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:36.390+0000 7f7dc15e6640 1 --2- 172.21.15.104:0/3020934810 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7dbc1d0720 0x7f7dbc09ca30 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:36.393 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:36.390+0000 7f7dc15e6640 1 --2- 172.21.15.104:0/3020934810 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f7dbc144310 0x7f7dbc1cf910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:36.393 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:36.390+0000 7f7dc15e6640 1 -- 172.21.15.104:0/3020934810 >> 172.21.15.104:0/3020934810 conn(0x7f7dbc0b8f00 msgr2=0x7f7dbc0b7380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T14:08:36.393 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:36.390+0000 7f7dc15e6640 1 -- 172.21.15.104:0/3020934810 shutdown_connections 2024-03-28T14:08:36.393 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:36.390+0000 7f7dc15e6640 1 -- 172.21.15.104:0/3020934810 wait complete. 2024-03-28T14:08:36.408 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-rbd 2024-03-28T14:08:36.520 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:36.518+0000 7f1edb37d640 1 Processor -- start 2024-03-28T14:08:36.520 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:36.518+0000 7f1edb37d640 1 -- start start 2024-03-28T14:08:36.521 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:36.518+0000 7f1edb37d640 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f1ed402c460 0x7f1ed40b3ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:36.521 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:36.518+0000 7f1edb37d640 1 --2- >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f1ed413fc50 0x7f1ed41456c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:36.522 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:36.518+0000 7f1edb37d640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1ed402c840 con 0x7f1ed40b4530 2024-03-28T14:08:36.522 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:36.518+0000 7f1edb37d640 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 -- 0x7f1ed40b4f20 con 0x7f1ed402c460 2024-03-28T14:08:36.522 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:36.518+0000 7f1edb37d640 1 -- --> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] -- mon_getmap magic: 0 -- 0x7f1ed40b5200 con 0x7f1ed413fc50 2024-03-28T14:08:36.522 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:36.518+0000 7f1ed9119640 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f1ed402c460 0x7f1ed40b3ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:36.522 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:36.518+0000 7f1ed9119640 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f1ed402c460 0x7f1ed40b3ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:59180/0 (socket says 172.21.15.104:59180) 2024-03-28T14:08:36.522 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:36.518+0000 7f1ed9119640 1 -- 172.21.15.104:0/3101950937 learned_addr learned my addr 172.21.15.104:0/3101950937 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-03-28T14:08:36.522 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:36.518+0000 7f1ed991a640 1 --2- 172.21.15.104:0/3101950937 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f1ed413fc50 0x7f1ed41456c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:36.522 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:36.518+0000 7f1ed9119640 1 -- 172.21.15.104:0/3101950937 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f1ed413fc50 msgr2=0x7f1ed41456c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T14:08:36.522 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:36.518+0000 7f1ed9119640 1 --2- 172.21.15.104:0/3101950937 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f1ed413fc50 0x7f1ed41456c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:36.522 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:36.518+0000 7f1ed9119640 1 -- 172.21.15.104:0/3101950937 >> v1:172.21.15.171:6789/0 conn(0x7f1ed40b4530 legacy=0x7f1ed413f540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T14:08:36.522 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:36.518+0000 7f1ed9119640 1 -- 172.21.15.104:0/3101950937 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1ed40b54e0 con 0x7f1ed402c460 2024-03-28T14:08:36.524 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:36.522+0000 7f1ed9119640 1 --2- 172.21.15.104:0/3101950937 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1ed402c460 0x7f1ed40b3ff0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f1ed0009870 tx=0x7f1ed001c4d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fa876fa7b94d11ae server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:08:36.524 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:36.522+0000 7f1ecbfff640 1 -- 172.21.15.104:0/3101950937 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ed00ab610 con 0x7f1ed402c460 2024-03-28T14:08:36.524 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:36.522+0000 7f1ecbfff640 1 -- 172.21.15.104:0/3101950937 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1ed00b0070 con 0x7f1ed402c460 2024-03-28T14:08:36.524 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:36.522+0000 7f1ecbfff640 1 -- 172.21.15.104:0/3101950937 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ed00ab9d0 con 0x7f1ed402c460 2024-03-28T14:08:36.525 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:36.522+0000 7f1edb37d640 1 -- 172.21.15.104:0/3101950937 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1ed402c460 msgr2=0x7f1ed40b3ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:36.525 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:36.522+0000 7f1edb37d640 1 --2- 172.21.15.104:0/3101950937 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1ed402c460 0x7f1ed40b3ff0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f1ed0009870 tx=0x7f1ed001c4d0 comp rx=0 tx=0).stop 2024-03-28T14:08:36.525 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:36.522+0000 7f1edb37d640 1 -- 172.21.15.104:0/3101950937 shutdown_connections 2024-03-28T14:08:36.525 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:36.522+0000 7f1edb37d640 1 --2- 172.21.15.104:0/3101950937 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f1ed413fc50 0x7f1ed41456c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:36.525 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:36.522+0000 7f1edb37d640 1 --2- 172.21.15.104:0/3101950937 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1ed402c460 0x7f1ed40b3ff0 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:36.526 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:36.522+0000 7f1edb37d640 1 -- 172.21.15.104:0/3101950937 >> 172.21.15.104:0/3101950937 conn(0x7f1ed40b95f0 msgr2=0x7f1ed40b99d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T14:08:36.526 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:36.522+0000 7f1edb37d640 1 -- 172.21.15.104:0/3101950937 shutdown_connections 2024-03-28T14:08:36.526 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:36.522+0000 7f1edb37d640 1 -- 172.21.15.104:0/3101950937 wait complete. 2024-03-28T14:08:36.532 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:36.522+0000 7f1edb37d640 1 Processor -- start 2024-03-28T14:08:36.532 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:36.522+0000 7f1edb37d640 1 -- start start 2024-03-28T14:08:36.532 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:36.522+0000 7f1edb37d640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f1ed402c460 0x7f1ed4145390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:36.532 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:36.522+0000 7f1edb37d640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1ed40b4530 0x7f1ed4142090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:36.532 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:36.522+0000 7f1edb37d640 1 --2- >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f1ed413fc50 0x7f1ed40983b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:36.533 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:36.522+0000 7f1edb37d640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f1ed4152b30 con 0x7f1ed40b4530 2024-03-28T14:08:36.533 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:36.522+0000 7f1edb37d640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f1ed41529b0 con 0x7f1ed402c460 2024-03-28T14:08:36.533 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:36.522+0000 7f1edb37d640 1 -- --> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] -- mon_getmap magic: 0 -- 0x7f1ed4152cb0 con 0x7f1ed413fc50 2024-03-28T14:08:36.533 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:36.522+0000 7f1ed8918640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1ed40b4530 0x7f1ed4142090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:36.533 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:36.522+0000 7f1ed8918640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1ed40b4530 0x7f1ed4142090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:59188/0 (socket says 172.21.15.104:59188) 2024-03-28T14:08:36.533 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:36.522+0000 7f1ed8918640 1 -- 172.21.15.104:0/3974971581 learned_addr learned my addr 172.21.15.104:0/3974971581 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-03-28T14:08:36.534 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:36.522+0000 7f1ed991a640 1 --2- 172.21.15.104:0/3974971581 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f1ed413fc50 0x7f1ed40983b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:36.534 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:36.522+0000 7f1ed9119640 1 --2- 172.21.15.104:0/3974971581 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f1ed402c460 0x7f1ed4145390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:36.534 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:36.522+0000 7f1ed8918640 1 -- 172.21.15.104:0/3974971581 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f1ed413fc50 msgr2=0x7f1ed40983b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:36.534 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:36.522+0000 7f1ed8918640 1 --2- 172.21.15.104:0/3974971581 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f1ed413fc50 0x7f1ed40983b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:36.535 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:36.522+0000 7f1ed8918640 1 -- 172.21.15.104:0/3974971581 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f1ed402c460 msgr2=0x7f1ed4145390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:36.535 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:36.522+0000 7f1ed8918640 1 --2- 172.21.15.104:0/3974971581 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f1ed402c460 0x7f1ed4145390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:36.535 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:36.522+0000 7f1ed8918640 1 -- 172.21.15.104:0/3974971581 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1ed413abd0 con 0x7f1ed40b4530 2024-03-28T14:08:36.535 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:36.522+0000 7f1ed9119640 1 --2- 172.21.15.104:0/3974971581 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f1ed402c460 0x7f1ed4145390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T14:08:36.535 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:36.522+0000 7f1ed8918640 1 --2- 172.21.15.104:0/3974971581 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1ed40b4530 0x7f1ed4142090 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f1ec400c4d0 tx=0x7f1ec400c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:08:36.535 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:36.522+0000 7f1ec9ffb640 1 -- 172.21.15.104:0/3974971581 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ec40052d0 con 0x7f1ed40b4530 2024-03-28T14:08:36.535 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:36.522+0000 7f1ec9ffb640 1 -- 172.21.15.104:0/3974971581 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1ec4005470 con 0x7f1ed40b4530 2024-03-28T14:08:36.535 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:36.522+0000 7f1edb37d640 1 -- 172.21.15.104:0/3974971581 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1ed4140a80 con 0x7f1ed40b4530 2024-03-28T14:08:36.535 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:36.522+0000 7f1ec9ffb640 1 -- 172.21.15.104:0/3974971581 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ec4012830 con 0x7f1ed40b4530 2024-03-28T14:08:36.535 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:36.526+0000 7f1edb37d640 1 -- 172.21.15.104:0/3974971581 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1ed413c660 con 0x7f1ed40b4530 2024-03-28T14:08:36.535 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:36.526+0000 7f1edb37d640 1 -- 172.21.15.104:0/3974971581 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1ed40687c0 con 0x7f1ed40b4530 2024-03-28T14:08:36.536 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:36.526+0000 7f1ec9ffb640 1 -- 172.21.15.104:0/3974971581 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) ==== 87803+0+0 (secure 0 0 0) 0x7f1ec40129d0 con 0x7f1ed40b4530 2024-03-28T14:08:36.536 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:36.526+0000 7f1ec9ffb640 1 --2- 172.21.15.104:0/3974971581 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f1eb8067ff0 0x7f1eb806a4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:36.536 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:36.526+0000 7f1ec9ffb640 1 -- 172.21.15.104:0/3974971581 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 4857+0+0 (secure 0 0 0) 0x7f1ec4085d10 con 0x7f1ed40b4530 2024-03-28T14:08:36.536 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:36.526+0000 7f1ed9119640 1 --2- 172.21.15.104:0/3974971581 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f1eb8067ff0 0x7f1eb806a4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:36.536 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:36.526+0000 7f1ed9119640 1 --2- 172.21.15.104:0/3974971581 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f1eb8067ff0 0x7f1eb806a4b0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f1ed00af000 tx=0x7f1ed0008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:08:36.536 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:36.530+0000 7f1ec9ffb640 1 -- 172.21.15.104:0/3974971581 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179117 (secure 0 0 0) 0x7f1ed413abd0 con 0x7f1ed40b4530 2024-03-28T14:08:36.734 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:36.730+0000 7f1edb37d640 1 -- 172.21.15.104:0/3974971581 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-rbd"} v 0) -- 0x7f1ed4143060 con 0x7f1ed40b4530 2024-03-28T14:08:36.736 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:36.734+0000 7f1ec9ffb640 1 -- 172.21.15.104:0/3974971581 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-rbd"}]=0 v8) ==== 90+0+0 (secure 0 0 0) 0x7f1ec40182b0 con 0x7f1ed40b4530 2024-03-28T14:08:36.739 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:36.734+0000 7f1edb37d640 1 -- 172.21.15.104:0/3974971581 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f1eb8067ff0 msgr2=0x7f1eb806a4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:36.739 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:36.734+0000 7f1edb37d640 1 --2- 172.21.15.104:0/3974971581 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f1eb8067ff0 0x7f1eb806a4b0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f1ed00af000 tx=0x7f1ed0008040 comp rx=0 tx=0).stop 2024-03-28T14:08:36.740 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:36.734+0000 7f1edb37d640 1 -- 172.21.15.104:0/3974971581 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1ed40b4530 msgr2=0x7f1ed4142090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:36.740 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:36.734+0000 7f1edb37d640 1 --2- 172.21.15.104:0/3974971581 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1ed40b4530 0x7f1ed4142090 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f1ec400c4d0 tx=0x7f1ec400c990 comp rx=0 tx=0).stop 2024-03-28T14:08:36.740 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:36.734+0000 7f1edb37d640 1 -- 172.21.15.104:0/3974971581 shutdown_connections 2024-03-28T14:08:36.740 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:36.734+0000 7f1edb37d640 1 --2- 172.21.15.104:0/3974971581 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f1ed413fc50 0x7f1ed40983b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:36.740 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:36.734+0000 7f1edb37d640 1 --2- 172.21.15.104:0/3974971581 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f1eb8067ff0 0x7f1eb806a4b0 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:36.740 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:36.734+0000 7f1edb37d640 1 --2- 172.21.15.104:0/3974971581 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1ed40b4530 0x7f1ed4142090 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:36.740 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:36.734+0000 7f1edb37d640 1 --2- 172.21.15.104:0/3974971581 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f1ed402c460 0x7f1ed4145390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:36.740 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:36.734+0000 7f1edb37d640 1 -- 172.21.15.104:0/3974971581 >> 172.21.15.104:0/3974971581 conn(0x7f1ed40b95f0 msgr2=0x7f1ed41356f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T14:08:36.740 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:36.734+0000 7f1edb37d640 1 -- 172.21.15.104:0/3974971581 shutdown_connections 2024-03-28T14:08:36.740 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:36.734+0000 7f1edb37d640 1 -- 172.21.15.104:0/3974971581 wait complete. 2024-03-28T14:08:36.755 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-rbd-mirror 2024-03-28T14:08:36.869 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:36.866+0000 7f0445f8a640 1 Processor -- start 2024-03-28T14:08:36.869 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:36.866+0000 7f0445f8a640 1 -- start start 2024-03-28T14:08:36.871 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:36.866+0000 7f0445f8a640 1 --2- >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f04400bd9e0 0x7f044014af30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:36.872 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:36.866+0000 7f0445f8a640 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f0440141b00 0x7f0440152080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:36.872 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:36.866+0000 7f0445f8a640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f04400b5050 con 0x7f044014b470 2024-03-28T14:08:36.872 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:36.866+0000 7f0445f8a640 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 -- 0x7f04400b70b0 con 0x7f0440141b00 2024-03-28T14:08:36.872 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:36.866+0000 7f0445f8a640 1 -- --> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] -- mon_getmap magic: 0 -- 0x7f044014b8d0 con 0x7f04400bd9e0 2024-03-28T14:08:36.872 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:36.866+0000 7f043ffff640 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f0440141b00 0x7f0440152080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:36.872 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:36.866+0000 7f043ffff640 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f0440141b00 0x7f0440152080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:50226/0 (socket says 172.21.15.104:50226) 2024-03-28T14:08:36.872 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:36.866+0000 7f043ffff640 1 -- 172.21.15.104:0/1157108431 learned_addr learned my addr 172.21.15.104:0/1157108431 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-03-28T14:08:36.872 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:36.866+0000 7f043f7fe640 1 --2- 172.21.15.104:0/1157108431 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f04400bd9e0 0x7f044014af30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:36.872 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:36.866+0000 7f043ffff640 1 -- 172.21.15.104:0/1157108431 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f04400bd9e0 msgr2=0x7f044014af30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T14:08:36.872 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:36.866+0000 7f043ffff640 1 --2- 172.21.15.104:0/1157108431 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f04400bd9e0 0x7f044014af30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:36.872 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:36.866+0000 7f043ffff640 1 -- 172.21.15.104:0/1157108431 >> v1:172.21.15.171:6789/0 conn(0x7f044014b470 legacy=0x7f04401413f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T14:08:36.872 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:36.866+0000 7f043ffff640 1 -- 172.21.15.104:0/1157108431 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f04400bddc0 con 0x7f0440141b00 2024-03-28T14:08:36.872 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:36.866+0000 7f043f7fe640 1 --2- 172.21.15.104:0/1157108431 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f04400bd9e0 0x7f044014af30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T14:08:36.874 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:36.870+0000 7f043ffff640 1 --2- 172.21.15.104:0/1157108431 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0440141b00 0x7f0440152080 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f0430009870 tx=0x7f043001c3d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=65022ae8ce97108f server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:08:36.874 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:36.870+0000 7f043e7fc640 1 -- 172.21.15.104:0/1157108431 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04300b2070 con 0x7f0440141b00 2024-03-28T14:08:36.874 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:36.870+0000 7f043e7fc640 1 -- 172.21.15.104:0/1157108431 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f04300ad640 con 0x7f0440141b00 2024-03-28T14:08:36.874 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:36.870+0000 7f043e7fc640 1 -- 172.21.15.104:0/1157108431 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04300ada10 con 0x7f0440141b00 2024-03-28T14:08:36.874 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:36.870+0000 7f0445f8a640 1 -- 172.21.15.104:0/1157108431 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0440141b00 msgr2=0x7f0440152080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:36.874 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:36.870+0000 7f0445f8a640 1 --2- 172.21.15.104:0/1157108431 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0440141b00 0x7f0440152080 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f0430009870 tx=0x7f043001c3d0 comp rx=0 tx=0).stop 2024-03-28T14:08:36.874 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:36.870+0000 7f0445f8a640 1 -- 172.21.15.104:0/1157108431 shutdown_connections 2024-03-28T14:08:36.875 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:36.870+0000 7f0445f8a640 1 --2- 172.21.15.104:0/1157108431 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0440141b00 0x7f0440152080 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:36.875 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:36.870+0000 7f0445f8a640 1 --2- 172.21.15.104:0/1157108431 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f04400bd9e0 0x7f044014af30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:36.875 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:36.870+0000 7f0445f8a640 1 -- 172.21.15.104:0/1157108431 >> 172.21.15.104:0/1157108431 conn(0x7f0440073530 msgr2=0x7f04400bd070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T14:08:36.875 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:36.870+0000 7f0445f8a640 1 -- 172.21.15.104:0/1157108431 shutdown_connections 2024-03-28T14:08:36.875 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:36.870+0000 7f0445f8a640 1 -- 172.21.15.104:0/1157108431 wait complete. 2024-03-28T14:08:36.877 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:36.870+0000 7f0445f8a640 1 Processor -- start 2024-03-28T14:08:36.877 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:36.870+0000 7f0445f8a640 1 -- start start 2024-03-28T14:08:36.877 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:36.874+0000 7f0445f8a640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f04400bd9e0 0x7f044013b920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:36.877 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:36.874+0000 7f0445f8a640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0440141b00 0x7f0440143f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:36.878 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:36.874+0000 7f0445f8a640 1 --2- >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f044014b470 0x7f04401433a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:36.878 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:36.874+0000 7f0445f8a640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f04400b62c0 con 0x7f0440141b00 2024-03-28T14:08:36.878 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:36.874+0000 7f0445f8a640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f04400b6140 con 0x7f04400bd9e0 2024-03-28T14:08:36.878 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:36.874+0000 7f0445f8a640 1 -- --> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] -- mon_getmap magic: 0 -- 0x7f04400b4940 con 0x7f044014b470 2024-03-28T14:08:36.878 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:36.874+0000 7f043effd640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0440141b00 0x7f0440143f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:36.878 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:36.874+0000 7f043effd640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0440141b00 0x7f0440143f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:50228/0 (socket says 172.21.15.104:50228) 2024-03-28T14:08:36.878 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:36.874+0000 7f043effd640 1 -- 172.21.15.104:0/2666895236 learned_addr learned my addr 172.21.15.104:0/2666895236 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-03-28T14:08:36.878 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:36.874+0000 7f043f7fe640 1 --2- 172.21.15.104:0/2666895236 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f04400bd9e0 0x7f044013b920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:36.878 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:36.874+0000 7f043ffff640 1 --2- 172.21.15.104:0/2666895236 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f044014b470 0x7f04401433a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:36.878 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:36.874+0000 7f043effd640 1 -- 172.21.15.104:0/2666895236 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f044014b470 msgr2=0x7f04401433a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:36.878 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:36.874+0000 7f043effd640 1 --2- 172.21.15.104:0/2666895236 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f044014b470 0x7f04401433a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:36.878 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:36.874+0000 7f043effd640 1 -- 172.21.15.104:0/2666895236 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f04400bd9e0 msgr2=0x7f044013b920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:36.878 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:36.874+0000 7f043effd640 1 --2- 172.21.15.104:0/2666895236 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f04400bd9e0 0x7f044013b920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:36.878 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:36.874+0000 7f043effd640 1 -- 172.21.15.104:0/2666895236 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f04401444b0 con 0x7f0440141b00 2024-03-28T14:08:36.879 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:36.874+0000 7f043effd640 1 --2- 172.21.15.104:0/2666895236 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0440141b00 0x7f0440143f70 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f042800b440 tx=0x7f042800b900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:08:36.879 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:36.874+0000 7f0437fff640 1 -- 172.21.15.104:0/2666895236 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0428015020 con 0x7f0440141b00 2024-03-28T14:08:36.879 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:36.874+0000 7f0437fff640 1 -- 172.21.15.104:0/2666895236 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0428011890 con 0x7f0440141b00 2024-03-28T14:08:36.879 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:36.874+0000 7f0437fff640 1 -- 172.21.15.104:0/2666895236 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0428011c80 con 0x7f0440141b00 2024-03-28T14:08:36.879 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:36.874+0000 7f0445f8a640 1 -- 172.21.15.104:0/2666895236 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f04401438e0 con 0x7f0440141b00 2024-03-28T14:08:36.879 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:36.874+0000 7f0445f8a640 1 -- 172.21.15.104:0/2666895236 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0440142ba0 con 0x7f0440141b00 2024-03-28T14:08:36.879 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:36.874+0000 7f0445f8a640 1 -- 172.21.15.104:0/2666895236 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f04400687c0 con 0x7f0440141b00 2024-03-28T14:08:36.880 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:36.874+0000 7f0437fff640 1 -- 172.21.15.104:0/2666895236 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) ==== 87803+0+0 (secure 0 0 0) 0x7f0428018020 con 0x7f0440141b00 2024-03-28T14:08:36.880 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:36.874+0000 7f0437fff640 1 --2- 172.21.15.104:0/2666895236 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f0420067ff0 0x7f042006a4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:36.880 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:36.874+0000 7f0437fff640 1 -- 172.21.15.104:0/2666895236 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 4857+0+0 (secure 0 0 0) 0x7f0428085f10 con 0x7f0440141b00 2024-03-28T14:08:36.880 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:36.874+0000 7f043f7fe640 1 --2- 172.21.15.104:0/2666895236 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f0420067ff0 0x7f042006a4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:36.880 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:36.878+0000 7f043f7fe640 1 --2- 172.21.15.104:0/2666895236 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f0420067ff0 0x7f042006a4b0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f04380075b0 tx=0x7f0438007540 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:08:36.882 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:36.878+0000 7f0437fff640 1 -- 172.21.15.104:0/2666895236 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179117 (secure 0 0 0) 0x7f042801f030 con 0x7f0440141b00 2024-03-28T14:08:37.089 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:37.086+0000 7f0445f8a640 1 -- 172.21.15.104:0/2666895236 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-rbd-mirror"} v 0) -- 0x7f04400b6af0 con 0x7f0440141b00 2024-03-28T14:08:37.092 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:37.090+0000 7f0437fff640 1 -- 172.21.15.104:0/2666895236 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-rbd-mirror"}]=0 v9) ==== 97+0+0 (secure 0 0 0) 0x7f042800d040 con 0x7f0440141b00 2024-03-28T14:08:37.095 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:37.090+0000 7f0445f8a640 1 -- 172.21.15.104:0/2666895236 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f0420067ff0 msgr2=0x7f042006a4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:37.095 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:37.090+0000 7f0445f8a640 1 --2- 172.21.15.104:0/2666895236 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f0420067ff0 0x7f042006a4b0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f04380075b0 tx=0x7f0438007540 comp rx=0 tx=0).stop 2024-03-28T14:08:37.095 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:37.090+0000 7f0445f8a640 1 -- 172.21.15.104:0/2666895236 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0440141b00 msgr2=0x7f0440143f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:37.096 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:37.090+0000 7f0445f8a640 1 --2- 172.21.15.104:0/2666895236 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0440141b00 0x7f0440143f70 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f042800b440 tx=0x7f042800b900 comp rx=0 tx=0).stop 2024-03-28T14:08:37.096 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:37.090+0000 7f0445f8a640 1 -- 172.21.15.104:0/2666895236 shutdown_connections 2024-03-28T14:08:37.096 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:37.090+0000 7f0445f8a640 1 --2- 172.21.15.104:0/2666895236 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f044014b470 0x7f04401433a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:37.096 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:37.090+0000 7f0445f8a640 1 --2- 172.21.15.104:0/2666895236 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f0420067ff0 0x7f042006a4b0 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:37.097 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:37.090+0000 7f0445f8a640 1 --2- 172.21.15.104:0/2666895236 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0440141b00 0x7f0440143f70 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:37.097 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:37.090+0000 7f0445f8a640 1 --2- 172.21.15.104:0/2666895236 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f04400bd9e0 0x7f044013b920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:37.097 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:37.090+0000 7f0445f8a640 1 -- 172.21.15.104:0/2666895236 >> 172.21.15.104:0/2666895236 conn(0x7f0440073530 msgr2=0x7f0440146910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T14:08:37.097 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:37.090+0000 7f0445f8a640 1 -- 172.21.15.104:0/2666895236 shutdown_connections 2024-03-28T14:08:37.097 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:37.090+0000 7f0445f8a640 1 -- 172.21.15.104:0/2666895236 wait complete. 2024-03-28T14:08:37.110 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-rgw 2024-03-28T14:08:37.225 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:37.222+0000 7fb93b1c5640 1 Processor -- start 2024-03-28T14:08:37.225 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:37.222+0000 7fb93b1c5640 1 -- start start 2024-03-28T14:08:37.227 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:37.222+0000 7fb93b1c5640 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fb9340b2740 0x7fb9340b3730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:37.227 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:37.222+0000 7fb93b1c5640 1 --2- >> v2:172.21.15.171:3300/0 conn(0x7fb934149380 0x7fb9341501b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:37.228 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:37.222+0000 7fb93b1c5640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb93406ccd0 con 0x7fb93406c8f0 2024-03-28T14:08:37.228 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:37.222+0000 7fb93b1c5640 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 -- 0x7fb9340b2b70 con 0x7fb9340b2740 2024-03-28T14:08:37.228 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:37.222+0000 7fb93b1c5640 1 -- --> v2:172.21.15.171:3300/0 -- mon_getmap magic: 0 -- 0x7fb9340b0b30 con 0x7fb934149380 2024-03-28T14:08:37.228 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:37.222+0000 7fb933fff640 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fb9340b2740 0x7fb9340b3730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:37.228 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:37.222+0000 7fb933fff640 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fb9340b2740 0x7fb9340b3730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:50232/0 (socket says 172.21.15.104:50232) 2024-03-28T14:08:37.228 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:37.222+0000 7fb933fff640 1 -- 172.21.15.104:0/4252013895 learned_addr learned my addr 172.21.15.104:0/4252013895 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-03-28T14:08:37.228 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:37.222+0000 7fb933fff640 1 -- 172.21.15.104:0/4252013895 >> v2:172.21.15.171:3300/0 conn(0x7fb934149380 msgr2=0x7fb9341501b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T14:08:37.229 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:37.222+0000 7fb933fff640 1 --2- 172.21.15.104:0/4252013895 >> v2:172.21.15.171:3300/0 conn(0x7fb934149380 0x7fb9341501b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:37.229 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:37.222+0000 7fb933fff640 1 -- 172.21.15.104:0/4252013895 >> v1:172.21.15.171:6789/0 conn(0x7fb93406c8f0 legacy=0x7fb934148c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T14:08:37.229 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:37.222+0000 7fb933fff640 1 -- 172.21.15.104:0/4252013895 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb9340b1980 con 0x7fb9340b2740 2024-03-28T14:08:37.234 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:37.226+0000 7fb933fff640 1 --2- 172.21.15.104:0/4252013895 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb9340b2740 0x7fb9340b3730 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7fb9340b2220 tx=0x7fb92401c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a5ef7af403271dee server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:08:37.235 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:37.226+0000 7fb9337fe640 1 -- 172.21.15.104:0/4252013895 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9240a5660 con 0x7fb9340b2740 2024-03-28T14:08:37.235 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:37.226+0000 7fb9337fe640 1 -- 172.21.15.104:0/4252013895 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb9240a5800 con 0x7fb9340b2740 2024-03-28T14:08:37.235 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:37.226+0000 7fb9337fe640 1 -- 172.21.15.104:0/4252013895 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9240a5bd0 con 0x7fb9340b2740 2024-03-28T14:08:37.235 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:37.226+0000 7fb93b1c5640 1 -- 172.21.15.104:0/4252013895 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb9340b2740 msgr2=0x7fb9340b3730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:37.235 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:37.226+0000 7fb93b1c5640 1 --2- 172.21.15.104:0/4252013895 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb9340b2740 0x7fb9340b3730 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7fb9340b2220 tx=0x7fb92401c920 comp rx=0 tx=0).stop 2024-03-28T14:08:37.235 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:37.226+0000 7fb93b1c5640 1 -- 172.21.15.104:0/4252013895 shutdown_connections 2024-03-28T14:08:37.235 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:37.226+0000 7fb93b1c5640 1 --2- 172.21.15.104:0/4252013895 >> v2:172.21.15.171:3300/0 conn(0x7fb934149380 0x7fb9341501b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:37.235 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:37.226+0000 7fb93b1c5640 1 --2- 172.21.15.104:0/4252013895 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb9340b2740 0x7fb9340b3730 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:37.235 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:37.226+0000 7fb93b1c5640 1 -- 172.21.15.104:0/4252013895 >> 172.21.15.104:0/4252013895 conn(0x7fb9340b8f00 msgr2=0x7fb9340b9310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T14:08:37.235 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:37.226+0000 7fb93b1c5640 1 -- 172.21.15.104:0/4252013895 shutdown_connections 2024-03-28T14:08:37.235 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:37.226+0000 7fb93b1c5640 1 -- 172.21.15.104:0/4252013895 wait complete. 2024-03-28T14:08:37.235 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:37.226+0000 7fb93b1c5640 1 Processor -- start 2024-03-28T14:08:37.235 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:37.226+0000 7fb93b1c5640 1 -- start start 2024-03-28T14:08:37.236 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:37.226+0000 7fb93b1c5640 1 --2- >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fb93406c8f0 0x7fb9340714a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:37.236 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:37.226+0000 7fb93b1c5640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb9340b2740 0x7fb9341de260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:37.236 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:37.226+0000 7fb93b1c5640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fb934149380 0x7fb9341de7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:37.236 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:37.226+0000 7fb93b1c5640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7fb93406ddf0 con 0x7fb9340b2740 2024-03-28T14:08:37.236 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:37.226+0000 7fb93b1c5640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7fb93406dc70 con 0x7fb934149380 2024-03-28T14:08:37.236 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:37.226+0000 7fb93b1c5640 1 -- --> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] -- mon_getmap magic: 0 -- 0x7fb934152f10 con 0x7fb93406c8f0 2024-03-28T14:08:37.236 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:37.226+0000 7fb933fff640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb9340b2740 0x7fb9341de260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:37.236 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:37.226+0000 7fb933fff640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb9340b2740 0x7fb9341de260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:50236/0 (socket says 172.21.15.104:50236) 2024-03-28T14:08:37.236 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:37.226+0000 7fb933fff640 1 -- 172.21.15.104:0/2237612068 learned_addr learned my addr 172.21.15.104:0/2237612068 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-03-28T14:08:37.236 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:37.226+0000 7fb938f61640 1 --2- 172.21.15.104:0/2237612068 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fb93406c8f0 0x7fb9340714a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:37.236 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:37.226+0000 7fb933fff640 1 -- 172.21.15.104:0/2237612068 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fb93406c8f0 msgr2=0x7fb9340714a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:37.236 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:37.226+0000 7fb933fff640 1 --2- 172.21.15.104:0/2237612068 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fb93406c8f0 0x7fb9340714a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:37.236 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:37.226+0000 7fb933fff640 1 -- 172.21.15.104:0/2237612068 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fb934149380 msgr2=0x7fb9341de7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:37.236 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:37.226+0000 7fb933fff640 1 --2- 172.21.15.104:0/2237612068 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fb934149380 0x7fb9341de7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:37.237 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:37.226+0000 7fb933fff640 1 -- 172.21.15.104:0/2237612068 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb934072460 con 0x7fb9340b2740 2024-03-28T14:08:37.237 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:37.226+0000 7fb938f61640 1 --2- 172.21.15.104:0/2237612068 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fb93406c8f0 0x7fb9340714a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T14:08:37.237 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:37.226+0000 7fb933fff640 1 --2- 172.21.15.104:0/2237612068 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb9340b2740 0x7fb9341de260 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7fb9240af040 tx=0x7fb9240196c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:08:37.237 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:37.230+0000 7fb9317fa640 1 -- 172.21.15.104:0/2237612068 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9240bc020 con 0x7fb9340b2740 2024-03-28T14:08:37.237 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:37.230+0000 7fb9317fa640 1 -- 172.21.15.104:0/2237612068 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb9240b8690 con 0x7fb9340b2740 2024-03-28T14:08:37.237 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:37.230+0000 7fb93b1c5640 1 -- 172.21.15.104:0/2237612068 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb934070730 con 0x7fb9340b2740 2024-03-28T14:08:37.237 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:37.230+0000 7fb9317fa640 1 -- 172.21.15.104:0/2237612068 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9240b8a80 con 0x7fb9340b2740 2024-03-28T14:08:37.237 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:37.230+0000 7fb93b1c5640 1 -- 172.21.15.104:0/2237612068 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb9341429b0 con 0x7fb9340b2740 2024-03-28T14:08:37.237 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:37.230+0000 7fb93b1c5640 1 -- 172.21.15.104:0/2237612068 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb934072150 con 0x7fb9340b2740 2024-03-28T14:08:37.237 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:37.230+0000 7fb9317fa640 1 -- 172.21.15.104:0/2237612068 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) ==== 87803+0+0 (secure 0 0 0) 0x7fb9240bf020 con 0x7fb9340b2740 2024-03-28T14:08:37.237 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:37.230+0000 7fb9317fa640 1 --2- 172.21.15.104:0/2237612068 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7fb920067fc0 0x7fb92006a480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:37.237 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:37.230+0000 7fb9317fa640 1 -- 172.21.15.104:0/2237612068 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 4857+0+0 (secure 0 0 0) 0x7fb924121240 con 0x7fb9340b2740 2024-03-28T14:08:37.237 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:37.230+0000 7fb938f61640 1 --2- 172.21.15.104:0/2237612068 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7fb920067fc0 0x7fb92006a480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:37.237 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:37.230+0000 7fb938f61640 1 --2- 172.21.15.104:0/2237612068 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7fb920067fc0 0x7fb92006a480 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fb92c00a830 tx=0x7fb92c008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:08:37.238 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:37.234+0000 7fb9317fa640 1 -- 172.21.15.104:0/2237612068 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179117 (secure 0 0 0) 0x7fb9240c6030 con 0x7fb9340b2740 2024-03-28T14:08:37.455 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:37.450+0000 7fb93b1c5640 1 -- 172.21.15.104:0/2237612068 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-rgw"} v 0) -- 0x7fb9340687c0 con 0x7fb9340b2740 2024-03-28T14:08:37.457 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:37.454+0000 7fb9317fa640 1 -- 172.21.15.104:0/2237612068 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-rgw"}]=0 v10) ==== 90+0+0 (secure 0 0 0) 0x7fb9240165a0 con 0x7fb9340b2740 2024-03-28T14:08:37.460 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:37.458+0000 7fb93b1c5640 1 -- 172.21.15.104:0/2237612068 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7fb920067fc0 msgr2=0x7fb92006a480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:37.460 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:37.458+0000 7fb93b1c5640 1 --2- 172.21.15.104:0/2237612068 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7fb920067fc0 0x7fb92006a480 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fb92c00a830 tx=0x7fb92c008040 comp rx=0 tx=0).stop 2024-03-28T14:08:37.460 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:37.458+0000 7fb93b1c5640 1 -- 172.21.15.104:0/2237612068 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb9340b2740 msgr2=0x7fb9341de260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:37.460 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:37.458+0000 7fb93b1c5640 1 --2- 172.21.15.104:0/2237612068 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb9340b2740 0x7fb9341de260 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7fb9240af040 tx=0x7fb9240196c0 comp rx=0 tx=0).stop 2024-03-28T14:08:37.461 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:37.458+0000 7fb93b1c5640 1 -- 172.21.15.104:0/2237612068 shutdown_connections 2024-03-28T14:08:37.461 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:37.458+0000 7fb93b1c5640 1 --2- 172.21.15.104:0/2237612068 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fb934149380 0x7fb9341de7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:37.461 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:37.458+0000 7fb93b1c5640 1 --2- 172.21.15.104:0/2237612068 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7fb920067fc0 0x7fb92006a480 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:37.461 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:37.458+0000 7fb93b1c5640 1 --2- 172.21.15.104:0/2237612068 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb9340b2740 0x7fb9341de260 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:37.461 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:37.458+0000 7fb93b1c5640 1 --2- 172.21.15.104:0/2237612068 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fb93406c8f0 0x7fb9340714a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:37.461 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:37.458+0000 7fb93b1c5640 1 -- 172.21.15.104:0/2237612068 >> 172.21.15.104:0/2237612068 conn(0x7fb9340b8f00 msgr2=0x7fb934140e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T14:08:37.461 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:37.458+0000 7fb93b1c5640 1 -- 172.21.15.104:0/2237612068 shutdown_connections 2024-03-28T14:08:37.461 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:37.458+0000 7fb93b1c5640 1 -- 172.21.15.104:0/2237612068 wait complete. 2024-03-28T14:08:37.476 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-28T14:08:37.600 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:37.598+0000 7f1109343640 1 Processor -- start 2024-03-28T14:08:37.600 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:37.598+0000 7f1109343640 1 -- start start 2024-03-28T14:08:37.601 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:37.598+0000 7f1109343640 1 --2- >> v2:172.21.15.171:3300/0 conn(0x7f110406d970 0x7f110413f3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:37.601 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:37.598+0000 7f1109343640 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f11040b5460 0x7f110413f920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:37.602 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:37.598+0000 7f1109343640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f110406dd50 con 0x7f11040732a0 2024-03-28T14:08:37.602 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:37.598+0000 7f1109343640 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 -- 0x7f11040b02a0 con 0x7f11040b5460 2024-03-28T14:08:37.602 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:37.598+0000 7f1109343640 1 -- --> v2:172.21.15.171:3300/0 -- mon_getmap magic: 0 -- 0x7f11040b2d30 con 0x7f110406d970 2024-03-28T14:08:37.602 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:37.598+0000 7f11027fc640 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f11040732a0 0x7f11040b0c30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:59760/0 (socket says 172.21.15.104:59760) 2024-03-28T14:08:37.602 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:37.598+0000 7f11027fc640 1 -- 172.21.15.104:0/3261462474 learned_addr learned my addr 172.21.15.104:0/3261462474 (peer_addr_for_me v1:172.21.15.104:0/0) 2024-03-28T14:08:37.602 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:37.598+0000 7f11037fe640 1 --2- 172.21.15.104:0/3261462474 >> v2:172.21.15.104:3300/0 conn(0x7f11040b5460 0x7f110413f920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:37.602 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:37.598+0000 7f1102ffd640 1 --2- 172.21.15.104:0/3261462474 >> v2:172.21.15.171:3300/0 conn(0x7f110406d970 0x7f110413f3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:37.602 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:37.598+0000 7f11037fe640 1 -- 172.21.15.104:0/3261462474 >> v2:172.21.15.171:3300/0 conn(0x7f110406d970 msgr2=0x7f110413f3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T14:08:37.602 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:37.598+0000 7f11037fe640 1 --2- 172.21.15.104:0/3261462474 >> v2:172.21.15.171:3300/0 conn(0x7f110406d970 0x7f110413f3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:37.602 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:37.598+0000 7f11037fe640 1 -- 172.21.15.104:0/3261462474 >> v1:172.21.15.104:6789/0 conn(0x7f11040732a0 legacy=0x7f11040b0c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:37.602 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:37.598+0000 7f11037fe640 1 -- 172.21.15.104:0/3261462474 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f11040b3010 con 0x7f11040b5460 2024-03-28T14:08:37.603 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:37.598+0000 7f11037fe640 1 --2- 172.21.15.104:0/3261462474 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f11040b5460 0x7f110413f920 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f11040b0420 tx=0x7f10fc01c870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=72e2bfc63dafc0f8 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:08:37.603 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:37.598+0000 7f1101ffb640 1 -- 172.21.15.104:0/3261462474 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10fc0b2070 con 0x7f11040b5460 2024-03-28T14:08:37.603 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:37.598+0000 7f1101ffb640 1 -- 172.21.15.104:0/3261462474 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f10fc0a5910 con 0x7f11040b5460 2024-03-28T14:08:37.603 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:37.598+0000 7f1101ffb640 1 -- 172.21.15.104:0/3261462474 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10fc0a5ce0 con 0x7f11040b5460 2024-03-28T14:08:37.604 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:37.602+0000 7f1109343640 1 -- 172.21.15.104:0/3261462474 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f11040b5460 msgr2=0x7f110413f920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:37.604 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:37.602+0000 7f1109343640 1 --2- 172.21.15.104:0/3261462474 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f11040b5460 0x7f110413f920 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f11040b0420 tx=0x7f10fc01c870 comp rx=0 tx=0).stop 2024-03-28T14:08:37.604 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:37.602+0000 7f1109343640 1 -- 172.21.15.104:0/3261462474 shutdown_connections 2024-03-28T14:08:37.604 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:37.602+0000 7f1109343640 1 --2- 172.21.15.104:0/3261462474 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f11040b5460 0x7f110413f920 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:37.604 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:37.602+0000 7f1109343640 1 --2- 172.21.15.104:0/3261462474 >> v2:172.21.15.171:3300/0 conn(0x7f110406d970 0x7f110413f3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:37.604 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:37.602+0000 7f1109343640 1 -- 172.21.15.104:0/3261462474 >> 172.21.15.104:0/3261462474 conn(0x7f11040b58a0 msgr2=0x7f11040b5cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T14:08:37.604 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:37.602+0000 7f1109343640 1 -- 172.21.15.104:0/3261462474 shutdown_connections 2024-03-28T14:08:37.604 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:37.602+0000 7f1109343640 1 -- 172.21.15.104:0/3261462474 wait complete. 2024-03-28T14:08:37.605 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:37.602+0000 7f1109343640 1 Processor -- start 2024-03-28T14:08:37.605 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:37.602+0000 7f1109343640 1 -- start start 2024-03-28T14:08:37.607 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:37.602+0000 7f1109343640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f110406d970 0x7f1104141db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:37.607 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:37.602+0000 7f1109343640 1 --2- >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f11040732a0 0x7f110409e000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:37.607 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:37.602+0000 7f1109343640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f11040b5460 0x7f11041408e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:37.607 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:37.602+0000 7f1109343640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f1104152a10 con 0x7f110406d970 2024-03-28T14:08:37.607 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:37.602+0000 7f1109343640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f1104152890 con 0x7f11040b5460 2024-03-28T14:08:37.607 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:37.602+0000 7f1109343640 1 -- --> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] -- mon_getmap magic: 0 -- 0x7f1104152b90 con 0x7f11040732a0 2024-03-28T14:08:37.608 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:37.602+0000 7f1102ffd640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f110406d970 0x7f1104141db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:37.608 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:37.602+0000 7f1102ffd640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f110406d970 0x7f1104141db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:50244/0 (socket says 172.21.15.104:50244) 2024-03-28T14:08:37.608 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:37.602+0000 7f1102ffd640 1 -- 172.21.15.104:0/439252591 learned_addr learned my addr 172.21.15.104:0/439252591 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-03-28T14:08:37.608 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:37.602+0000 7f11027fc640 1 --2- 172.21.15.104:0/439252591 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f11040732a0 0x7f110409e000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:37.608 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:37.602+0000 7f1102ffd640 1 -- 172.21.15.104:0/439252591 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f11040732a0 msgr2=0x7f110409e000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:37.608 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:37.602+0000 7f11037fe640 1 --2- 172.21.15.104:0/439252591 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f11040b5460 0x7f11041408e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:37.608 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:37.602+0000 7f1102ffd640 1 --2- 172.21.15.104:0/439252591 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f11040732a0 0x7f110409e000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:37.608 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:37.602+0000 7f1102ffd640 1 -- 172.21.15.104:0/439252591 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f11040b5460 msgr2=0x7f11041408e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:37.608 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:37.602+0000 7f1102ffd640 1 --2- 172.21.15.104:0/439252591 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f11040b5460 0x7f11041408e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:37.608 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:37.602+0000 7f1102ffd640 1 -- 172.21.15.104:0/439252591 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f110409aab0 con 0x7f110406d970 2024-03-28T14:08:37.609 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:37.602+0000 7f1102ffd640 1 --2- 172.21.15.104:0/439252591 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f110406d970 0x7f1104141db0 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f10ec00d800 tx=0x7f10ec00dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:08:37.609 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:37.602+0000 7f10fb7fe640 1 -- 172.21.15.104:0/439252591 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10ec00b660 con 0x7f110406d970 2024-03-28T14:08:37.609 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:37.602+0000 7f10fb7fe640 1 -- 172.21.15.104:0/439252591 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f10ec019070 con 0x7f110406d970 2024-03-28T14:08:37.609 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:37.602+0000 7f1109343640 1 -- 172.21.15.104:0/439252591 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1104072520 con 0x7f110406d970 2024-03-28T14:08:37.609 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:37.602+0000 7f10fb7fe640 1 -- 172.21.15.104:0/439252591 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10ec00ba00 con 0x7f110406d970 2024-03-28T14:08:37.609 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:37.602+0000 7f1109343640 1 -- 172.21.15.104:0/439252591 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f11041dead0 con 0x7f110406d970 2024-03-28T14:08:37.610 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:37.606+0000 7f1109343640 1 -- 172.21.15.104:0/439252591 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1104140520 con 0x7f110406d970 2024-03-28T14:08:37.610 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:37.606+0000 7f10fb7fe640 1 -- 172.21.15.104:0/439252591 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) ==== 87803+0+0 (secure 0 0 0) 0x7f10ec00bba0 con 0x7f110406d970 2024-03-28T14:08:37.610 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:37.606+0000 7f10fb7fe640 1 --2- 172.21.15.104:0/439252591 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f10e4067fc0 0x7f10e406a480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:37.611 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:37.606+0000 7f10fb7fe640 1 -- 172.21.15.104:0/439252591 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 4857+0+0 (secure 0 0 0) 0x7f10ec086570 con 0x7f110406d970 2024-03-28T14:08:37.611 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:37.606+0000 7f11027fc640 1 --2- 172.21.15.104:0/439252591 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f10e4067fc0 0x7f10e406a480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:37.611 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:37.606+0000 7f11027fc640 1 --2- 172.21.15.104:0/439252591 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f10e4067fc0 0x7f10e406a480 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f110413a880 tx=0x7f10f4009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:08:37.613 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:37.610+0000 7f10fb7fe640 1 -- 172.21.15.104:0/439252591 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179117 (secure 0 0 0) 0x7f10ec057100 con 0x7f110406d970 2024-03-28T14:08:37.769 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:37.766+0000 7f1109343640 1 -- 172.21.15.104:0/439252591 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f11040687c0 con 0x7f110406d970 2024-03-28T14:08:37.771 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:37.766+0000 7f10fb7fe640 1 -- 172.21.15.104:0/439252591 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v26) ==== 74+0+12915 (secure 0 0 0) 0x7f10ec056a00 con 0x7f110406d970 2024-03-28T14:08:37.771 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-03-28T14:08:37.771 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":26,"fsid":"428a90f2-3351-435f-8fcf-f7e66c78a13e","created":"2024-03-28T14:07:34.862458+0000","modified":"2024-03-28T14:08:34.887005+0000","last_up_change":"2024-03-28T14:07:46.028122+0000","last_in_change":"2024-03-28T14:07:37.157486+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-28T14:07:47.622988+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"1cb70540-4e42-402f-a446-aaec4ca83007","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6800","nonce":3745690918},{"type":"v1","addr":"172.21.15.104:6801","nonce":3745690918}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":3745690918},{"type":"v1","addr":"172.21.15.104:6803","nonce":3745690918}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":3745690918},{"type":"v1","addr":"172.21.15.104:6807","nonce":3745690918}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":3745690918},{"type":"v1","addr":"172.21.15.104:6805","nonce":3745690918}]},"public_addr":"172.21.15.104:6801/3745690918","cluster_addr":"172.21.15.104:6803/3745690918","heartbeat_back_addr":"172.21.15.104:6807/3745690918","heartbeat_front_addr":"172.21.15.104:6805/3745690918","state":["exists","up"]},{"osd":1,"uuid":"3ab7afdd-b980-418c-81f2-004e16de3dca","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":19438674},{"type":"v1","addr":"172.21.15.104:6810","nonce":19438674}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":19438674},{"type":"v1","addr":"172.21.15.104:6814","nonce":19438674}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":19438674},{"type":"v1","addr":"172.21.15.104:6823","nonce":19438674}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":19438674},{"type":"v1","addr":"172.21.15.104:6819","nonce":19438674}]},"public_addr":"172.21.15.104:6810/19438674","cluster_addr":"172.21.15.104:6814/19438674","heartbeat_back_addr":"172.21.15.104:6823/19438674","heartbeat_front_addr":"172.21.15.104:6819/19438674","state":["exists","up"]},{"osd":2,"uuid":"3e2651c8-ecae-4a9e-a053-27240a79eac2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":2352012626},{"type":"v1","addr":"172.21.15.104:6825","nonce":2352012626}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6826","nonce":2352012626},{"type":"v1","addr":"172.21.15.104:6827","nonce":2352012626}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":2352012626},{"type":"v1","addr":"172.21.15.104:6831","nonce":2352012626}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6828","nonce":2352012626},{"type":"v1","addr":"172.21.15.104:6829","nonce":2352012626}]},"public_addr":"172.21.15.104:6825/2352012626","cluster_addr":"172.21.15.104:6827/2352012626","heartbeat_back_addr":"172.21.15.104:6831/2352012626","heartbeat_front_addr":"172.21.15.104:6829/2352012626","state":["exists","up"]},{"osd":3,"uuid":"925c7e0a-38ef-47cd-b288-8bec8456fb17","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6809","nonce":3568347381},{"type":"v1","addr":"172.21.15.104:6811","nonce":3568347381}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6813","nonce":3568347381},{"type":"v1","addr":"172.21.15.104:6815","nonce":3568347381}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":3568347381},{"type":"v1","addr":"172.21.15.104:6822","nonce":3568347381}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6817","nonce":3568347381},{"type":"v1","addr":"172.21.15.104:6818","nonce":3568347381}]},"public_addr":"172.21.15.104:6811/3568347381","cluster_addr":"172.21.15.104:6815/3568347381","heartbeat_back_addr":"172.21.15.104:6822/3568347381","heartbeat_front_addr":"172.21.15.104:6818/3568347381","state":["exists","up"]},{"osd":4,"uuid":"4a8fac86-e595-4a0e-96e3-57edb3cc3f57","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6800","nonce":2107001077},{"type":"v1","addr":"172.21.15.171:6801","nonce":2107001077}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6802","nonce":2107001077},{"type":"v1","addr":"172.21.15.171:6803","nonce":2107001077}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6806","nonce":2107001077},{"type":"v1","addr":"172.21.15.171:6807","nonce":2107001077}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6804","nonce":2107001077},{"type":"v1","addr":"172.21.15.171:6805","nonce":2107001077}]},"public_addr":"172.21.15.171:6801/2107001077","cluster_addr":"172.21.15.171:6803/2107001077","heartbeat_back_addr":"172.21.15.171:6807/2107001077","heartbeat_front_addr":"172.21.15.171:6805/2107001077","state":["exists","up"]},{"osd":5,"uuid":"e48d50b0-beae-40ae-b15a-9a84dbfe88ab","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6808","nonce":2934826925},{"type":"v1","addr":"172.21.15.171:6810","nonce":2934826925}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6812","nonce":2934826925},{"type":"v1","addr":"172.21.15.171:6814","nonce":2934826925}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6819","nonce":2934826925},{"type":"v1","addr":"172.21.15.171:6821","nonce":2934826925}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6816","nonce":2934826925},{"type":"v1","addr":"172.21.15.171:6817","nonce":2934826925}]},"public_addr":"172.21.15.171:6810/2934826925","cluster_addr":"172.21.15.171:6814/2934826925","heartbeat_back_addr":"172.21.15.171:6821/2934826925","heartbeat_front_addr":"172.21.15.171:6817/2934826925","state":["exists","up"]},{"osd":6,"uuid":"5e7c3292-9936-4f35-b267-aed9d4e49323","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6824","nonce":436742919},{"type":"v1","addr":"172.21.15.171:6825","nonce":436742919}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6826","nonce":436742919},{"type":"v1","addr":"172.21.15.171:6827","nonce":436742919}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6830","nonce":436742919},{"type":"v1","addr":"172.21.15.171:6831","nonce":436742919}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6828","nonce":436742919},{"type":"v1","addr":"172.21.15.171:6829","nonce":436742919}]},"public_addr":"172.21.15.171:6825/436742919","cluster_addr":"172.21.15.171:6827/436742919","heartbeat_back_addr":"172.21.15.171:6831/436742919","heartbeat_front_addr":"172.21.15.171:6829/436742919","state":["exists","up"]},{"osd":7,"uuid":"339b8414-d7f0-4fb0-a446-efaf635cb7ff","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6809","nonce":2055631175},{"type":"v1","addr":"172.21.15.171:6811","nonce":2055631175}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6813","nonce":2055631175},{"type":"v1","addr":"172.21.15.171:6815","nonce":2055631175}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6822","nonce":2055631175},{"type":"v1","addr":"172.21.15.171:6823","nonce":2055631175}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6818","nonce":2055631175},{"type":"v1","addr":"172.21.15.171:6820","nonce":2055631175}]},"public_addr":"172.21.15.171:6811/2055631175","cluster_addr":"172.21.15.171:6815/2055631175","heartbeat_back_addr":"172.21.15.171:6823/2055631175","heartbeat_front_addr":"172.21.15.171:6820/2055631175","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T14:07:42.668881+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T14:07:42.708397+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T14:07:42.634191+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T14:07:42.733662+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T14:07:42.651541+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T14:07:42.783181+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T14:07:42.766310+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T14:07:42.670797+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-28T14:08:37.773 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:37.770+0000 7f1109343640 1 -- 172.21.15.104:0/439252591 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f10e4067fc0 msgr2=0x7f10e406a480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:37.773 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:37.770+0000 7f1109343640 1 --2- 172.21.15.104:0/439252591 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f10e4067fc0 0x7f10e406a480 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f110413a880 tx=0x7f10f4009040 comp rx=0 tx=0).stop 2024-03-28T14:08:37.773 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:37.770+0000 7f1109343640 1 -- 172.21.15.104:0/439252591 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f110406d970 msgr2=0x7f1104141db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:37.773 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:37.770+0000 7f1109343640 1 --2- 172.21.15.104:0/439252591 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f110406d970 0x7f1104141db0 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f10ec00d800 tx=0x7f10ec00dcc0 comp rx=0 tx=0).stop 2024-03-28T14:08:37.773 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:37.770+0000 7f1109343640 1 -- 172.21.15.104:0/439252591 shutdown_connections 2024-03-28T14:08:37.774 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:37.770+0000 7f1109343640 1 --2- 172.21.15.104:0/439252591 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f11040b5460 0x7f11041408e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:37.774 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:37.770+0000 7f1109343640 1 --2- 172.21.15.104:0/439252591 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f11040732a0 0x7f110409e000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:37.774 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:37.770+0000 7f1109343640 1 --2- 172.21.15.104:0/439252591 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f10e4067fc0 0x7f10e406a480 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:37.774 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:37.770+0000 7f1109343640 1 --2- 172.21.15.104:0/439252591 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f110406d970 0x7f1104141db0 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:37.774 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:37.770+0000 7f1109343640 1 -- 172.21.15.104:0/439252591 >> 172.21.15.104:0/439252591 conn(0x7f11040b58a0 msgr2=0x7f110413ce20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T14:08:37.774 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:37.770+0000 7f1109343640 1 -- 172.21.15.104:0/439252591 shutdown_connections 2024-03-28T14:08:37.774 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:37.770+0000 7f1109343640 1 -- 172.21.15.104:0/439252591 wait complete. 2024-03-28T14:08:37.789 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-28T14:07:47.622988+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-28T14:08:37.789 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-28T14:08:37.903 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:37.898+0000 7f5f6ce55640 1 Processor -- start 2024-03-28T14:08:37.903 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:37.898+0000 7f5f6ce55640 1 -- start start 2024-03-28T14:08:37.906 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:37.902+0000 7f5f6ce55640 1 --2- >> v2:172.21.15.171:3300/0 conn(0x7f5f680b51e0 0x7f5f6809c450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:37.906 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:37.902+0000 7f5f6ce55640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5f6809ae40 con 0x7f5f6806dd10 2024-03-28T14:08:37.906 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:37.902+0000 7f5f6ce55640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5f6806e0f0 con 0x7f5f6809aa10 2024-03-28T14:08:37.906 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:37.902+0000 7f5f6ce55640 1 -- --> v2:172.21.15.171:3300/0 -- mon_getmap magic: 0 -- 0x7f5f680b6370 con 0x7f5f680b51e0 2024-03-28T14:08:37.906 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:37.902+0000 7f5f6659c640 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f5f6806dd10 0x7f5f6813f6e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:59776/0 (socket says 172.21.15.104:59776) 2024-03-28T14:08:37.906 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:37.902+0000 7f5f6659c640 1 -- 172.21.15.104:0/63621990 learned_addr learned my addr 172.21.15.104:0/63621990 (peer_addr_for_me v1:172.21.15.104:0/0) 2024-03-28T14:08:37.906 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:37.902+0000 7f5f65d9b640 1 --2- 172.21.15.104:0/63621990 >> v2:172.21.15.171:3300/0 conn(0x7f5f680b51e0 0x7f5f6809c450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:37.907 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:37.902+0000 7f5f6559a640 1 -- 172.21.15.104:0/63621990 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3556096289 0 0) 0x7f5f6809ae40 con 0x7f5f6806dd10 2024-03-28T14:08:37.907 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:37.902+0000 7f5f6559a640 1 -- 172.21.15.104:0/63621990 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5f50003530 con 0x7f5f6806dd10 2024-03-28T14:08:37.907 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:37.902+0000 7f5f6559a640 1 -- 172.21.15.104:0/63621990 <== mon.0 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3974844595 0 0) 0x7f5f50003530 con 0x7f5f6806dd10 2024-03-28T14:08:37.907 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:37.902+0000 7f5f6559a640 1 -- 172.21.15.104:0/63621990 >> v2:172.21.15.171:3300/0 conn(0x7f5f680b51e0 msgr2=0x7f5f6809c450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T14:08:37.907 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:37.902+0000 7f5f6559a640 1 --2- 172.21.15.104:0/63621990 >> v2:172.21.15.171:3300/0 conn(0x7f5f680b51e0 0x7f5f6809c450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:37.907 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:37.902+0000 7f5f6559a640 1 -- 172.21.15.104:0/63621990 >> v1:172.21.15.171:6789/0 conn(0x7f5f6809aa10 legacy=0x7f5f681500e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:37.907 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:37.902+0000 7f5f6559a640 1 -- 172.21.15.104:0/63621990 --> v1:172.21.15.104:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5f680b5d20 con 0x7f5f6806dd10 2024-03-28T14:08:37.907 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:37.902+0000 7f5f6559a640 1 -- 172.21.15.104:0/63621990 <== mon.0 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3745580849 0 0) 0x7f5f58001680 con 0x7f5f6806dd10 2024-03-28T14:08:37.907 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:37.902+0000 7f5f6559a640 1 -- 172.21.15.104:0/63621990 >> v1:172.21.15.104:6789/0 conn(0x7f5f6806dd10 legacy=0x7f5f6813f6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:37.907 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:37.902+0000 7f5f6559a640 1 --2- 172.21.15.104:0/63621990 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5f500039c0 0x7f5f50003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:37.908 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:37.902+0000 7f5f6559a640 1 --2- 172.21.15.104:0/63621990 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f5f50004810 0x7f5f50004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:37.908 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:37.902+0000 7f5f6559a640 1 --2- 172.21.15.104:0/63621990 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f5f50005220 0x7f5f5000b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:37.908 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:37.902+0000 7f5f6559a640 1 -- 172.21.15.104:0/63621990 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f5f680b6370 con 0x7f5f500039c0 2024-03-28T14:08:37.908 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:37.902+0000 7f5f6559a640 1 -- 172.21.15.104:0/63621990 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f5f5000c150 con 0x7f5f50005220 2024-03-28T14:08:37.908 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:37.902+0000 7f5f6559a640 1 -- 172.21.15.104:0/63621990 --> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] -- mon_getmap magic: 0 -- 0x7f5f5000c350 con 0x7f5f50004810 2024-03-28T14:08:37.908 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:37.902+0000 7f5f6659c640 1 --2- 172.21.15.104:0/63621990 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5f500039c0 0x7f5f50003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:37.908 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:37.902+0000 7f5f65d9b640 1 --2- 172.21.15.104:0/63621990 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f5f50004810 0x7f5f50004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:37.908 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:37.902+0000 7f5f6659c640 1 -- 172.21.15.104:0/63621990 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f5f50004810 msgr2=0x7f5f50004c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T14:08:37.908 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:37.902+0000 7f5f6659c640 1 --2- 172.21.15.104:0/63621990 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f5f50004810 0x7f5f50004c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:37.908 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:37.902+0000 7f5f66d9d640 1 --2- 172.21.15.104:0/63621990 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f5f50005220 0x7f5f5000b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:37.909 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:37.902+0000 7f5f6659c640 1 -- 172.21.15.104:0/63621990 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f5f50005220 msgr2=0x7f5f5000b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T14:08:37.909 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:37.902+0000 7f5f6659c640 1 --2- 172.21.15.104:0/63621990 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f5f50005220 0x7f5f5000b9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:37.909 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:37.902+0000 7f5f66d9d640 1 -- 172.21.15.104:0/63621990 reap_dead start 2024-03-28T14:08:37.909 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:37.902+0000 7f5f6659c640 1 -- 172.21.15.104:0/63621990 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5f50003530 con 0x7f5f500039c0 2024-03-28T14:08:37.913 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:37.902+0000 7f5f6659c640 1 --2- 172.21.15.104:0/63621990 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5f500039c0 0x7f5f50003e00 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f5f5800c240 tx=0x7f5f5801cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=99660dbf908b60bf server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:08:37.913 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:37.902+0000 7f5f6559a640 1 -- 172.21.15.104:0/63621990 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f58007d20 con 0x7f5f500039c0 2024-03-28T14:08:37.913 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:37.902+0000 7f5f6559a640 1 -- 172.21.15.104:0/63621990 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5f580a9070 con 0x7f5f500039c0 2024-03-28T14:08:37.913 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:37.902+0000 7f5f6559a640 1 -- 172.21.15.104:0/63621990 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f58005d80 con 0x7f5f500039c0 2024-03-28T14:08:37.913 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:37.906+0000 7f5f6ce55640 1 -- 172.21.15.104:0/63621990 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5f500039c0 msgr2=0x7f5f50003e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:37.913 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:37.906+0000 7f5f6ce55640 1 --2- 172.21.15.104:0/63621990 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5f500039c0 0x7f5f50003e00 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f5f5800c240 tx=0x7f5f5801cc80 comp rx=0 tx=0).stop 2024-03-28T14:08:37.914 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:37.906+0000 7f5f6ce55640 1 -- 172.21.15.104:0/63621990 shutdown_connections 2024-03-28T14:08:37.914 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:37.906+0000 7f5f6ce55640 1 --2- 172.21.15.104:0/63621990 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5f500039c0 0x7f5f50003e00 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:37.914 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:37.906+0000 7f5f6ce55640 1 -- 172.21.15.104:0/63621990 >> 172.21.15.104:0/63621990 conn(0x7f5f680ba390 msgr2=0x7f5f680ba7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T14:08:37.914 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:37.906+0000 7f5f6ce55640 1 -- 172.21.15.104:0/63621990 shutdown_connections 2024-03-28T14:08:37.914 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:37.906+0000 7f5f6ce55640 1 -- 172.21.15.104:0/63621990 wait complete. 2024-03-28T14:08:37.914 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:37.906+0000 7f5f6ce55640 1 Processor -- start 2024-03-28T14:08:37.914 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:37.906+0000 7f5f6ce55640 1 -- start start 2024-03-28T14:08:37.914 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:37.906+0000 7f5f6ce55640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5f500039c0 0x7f5f681436b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:37.914 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:37.906+0000 7f5f6ce55640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f5f681d42b0 0x7f5f6809d350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:37.914 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:37.906+0000 7f5f6ce55640 1 --2- >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f5f6809d890 0x7f5f6813a240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:37.914 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:37.906+0000 7f5f6ce55640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f5f6806b750 con 0x7f5f500039c0 2024-03-28T14:08:37.915 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:37.906+0000 7f5f6ce55640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f5f6806b5d0 con 0x7f5f681d42b0 2024-03-28T14:08:37.915 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:37.906+0000 7f5f6ce55640 1 -- --> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] -- mon_getmap magic: 0 -- 0x7f5f6806b8d0 con 0x7f5f6809d890 2024-03-28T14:08:37.915 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:37.906+0000 7f5f6659c640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5f500039c0 0x7f5f681436b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:37.915 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:37.906+0000 7f5f6659c640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5f500039c0 0x7f5f681436b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:50254/0 (socket says 172.21.15.104:50254) 2024-03-28T14:08:37.915 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:37.906+0000 7f5f6659c640 1 -- 172.21.15.104:0/2408604775 learned_addr learned my addr 172.21.15.104:0/2408604775 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-03-28T14:08:37.915 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:37.906+0000 7f5f65d9b640 1 --2- 172.21.15.104:0/2408604775 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f5f681d42b0 0x7f5f6809d350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:37.915 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:37.906+0000 7f5f6659c640 1 -- 172.21.15.104:0/2408604775 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f5f6809d890 msgr2=0x7f5f6813a240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:37.915 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:37.906+0000 7f5f6659c640 1 --2- 172.21.15.104:0/2408604775 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f5f6809d890 0x7f5f6813a240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:37.915 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:37.906+0000 7f5f6659c640 1 -- 172.21.15.104:0/2408604775 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f5f681d42b0 msgr2=0x7f5f6809d350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:37.915 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:37.906+0000 7f5f6659c640 1 --2- 172.21.15.104:0/2408604775 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f5f681d42b0 0x7f5f6809d350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:37.916 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:37.906+0000 7f5f6659c640 1 -- 172.21.15.104:0/2408604775 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5f68144950 con 0x7f5f500039c0 2024-03-28T14:08:37.916 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:37.906+0000 7f5f6659c640 1 --2- 172.21.15.104:0/2408604775 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5f500039c0 0x7f5f681436b0 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f5f5801cab0 tx=0x7f5f50005840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:08:37.916 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:37.906+0000 7f5f56ffd640 1 -- 172.21.15.104:0/2408604775 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f58007d20 con 0x7f5f500039c0 2024-03-28T14:08:37.916 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:37.906+0000 7f5f56ffd640 1 -- 172.21.15.104:0/2408604775 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5f58007390 con 0x7f5f500039c0 2024-03-28T14:08:37.916 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:37.906+0000 7f5f6ce55640 1 -- 172.21.15.104:0/2408604775 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5f6809ff80 con 0x7f5f500039c0 2024-03-28T14:08:37.916 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:37.906+0000 7f5f56ffd640 1 -- 172.21.15.104:0/2408604775 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f58007740 con 0x7f5f500039c0 2024-03-28T14:08:37.916 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:37.906+0000 7f5f6ce55640 1 -- 172.21.15.104:0/2408604775 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5f6809f930 con 0x7f5f500039c0 2024-03-28T14:08:37.916 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:37.906+0000 7f5f6ce55640 1 -- 172.21.15.104:0/2408604775 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5f680687c0 con 0x7f5f500039c0 2024-03-28T14:08:37.916 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:37.906+0000 7f5f56ffd640 1 -- 172.21.15.104:0/2408604775 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) ==== 87803+0+0 (secure 0 0 0) 0x7f5f580a5870 con 0x7f5f500039c0 2024-03-28T14:08:37.917 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:37.910+0000 7f5f56ffd640 1 --2- 172.21.15.104:0/2408604775 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f5f48067fc0 0x7f5f4806a480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:37.917 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:37.910+0000 7f5f56ffd640 1 -- 172.21.15.104:0/2408604775 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 4857+0+0 (secure 0 0 0) 0x7f5f5811f9c0 con 0x7f5f500039c0 2024-03-28T14:08:37.917 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:37.910+0000 7f5f65d9b640 1 --2- 172.21.15.104:0/2408604775 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f5f48067fc0 0x7f5f4806a480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:37.917 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:37.910+0000 7f5f65d9b640 1 --2- 172.21.15.104:0/2408604775 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f5f48067fc0 0x7f5f4806a480 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f5f6809e170 tx=0x7f5f5c008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:08:37.917 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:37.914+0000 7f5f56ffd640 1 -- 172.21.15.104:0/2408604775 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179117 (secure 0 0 0) 0x7f5f58121e20 con 0x7f5f500039c0 2024-03-28T14:08:38.062 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:38.058+0000 7f5f6ce55640 1 -- 172.21.15.104:0/2408604775 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f5f680b4fa0 con 0x7f5f500039c0 2024-03-28T14:08:38.062 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:38.058+0000 7f5f56ffd640 1 -- 172.21.15.104:0/2408604775 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v26) ==== 93+0+10 (secure 0 0 0) 0x7f5f58016990 con 0x7f5f500039c0 2024-03-28T14:08:38.063 INFO:teuthology.orchestra.run.smithi104.stdout:pg_num: 1 2024-03-28T14:08:38.065 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:38.062+0000 7f5f6ce55640 1 -- 172.21.15.104:0/2408604775 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f5f48067fc0 msgr2=0x7f5f4806a480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:38.065 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:38.062+0000 7f5f6ce55640 1 --2- 172.21.15.104:0/2408604775 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f5f48067fc0 0x7f5f4806a480 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f5f6809e170 tx=0x7f5f5c008040 comp rx=0 tx=0).stop 2024-03-28T14:08:38.065 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:38.062+0000 7f5f6ce55640 1 -- 172.21.15.104:0/2408604775 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5f500039c0 msgr2=0x7f5f681436b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:38.065 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:38.062+0000 7f5f6ce55640 1 --2- 172.21.15.104:0/2408604775 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5f500039c0 0x7f5f681436b0 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f5f5801cab0 tx=0x7f5f50005840 comp rx=0 tx=0).stop 2024-03-28T14:08:38.065 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:38.062+0000 7f5f6ce55640 1 -- 172.21.15.104:0/2408604775 shutdown_connections 2024-03-28T14:08:38.065 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:38.062+0000 7f5f6ce55640 1 --2- 172.21.15.104:0/2408604775 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f5f6809d890 0x7f5f6813a240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:38.065 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:38.062+0000 7f5f6ce55640 1 --2- 172.21.15.104:0/2408604775 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f5f681d42b0 0x7f5f6809d350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:38.065 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:38.062+0000 7f5f6ce55640 1 --2- 172.21.15.104:0/2408604775 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f5f48067fc0 0x7f5f4806a480 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:38.065 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:38.062+0000 7f5f6ce55640 1 --2- 172.21.15.104:0/2408604775 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5f500039c0 0x7f5f681436b0 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:38.066 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:38.062+0000 7f5f6ce55640 1 -- 172.21.15.104:0/2408604775 >> 172.21.15.104:0/2408604775 conn(0x7f5f680ba390 msgr2=0x7f5f680b8a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T14:08:38.066 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:38.062+0000 7f5f6ce55640 1 -- 172.21.15.104:0/2408604775 shutdown_connections 2024-03-28T14:08:38.066 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:38.062+0000 7f5f6ce55640 1 -- 172.21.15.104:0/2408604775 wait complete. 2024-03-28T14:08:38.080 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-03-28T14:08:38.198 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:38.194+0000 7fa53b251640 1 Processor -- start 2024-03-28T14:08:38.198 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:38.194+0000 7fa53b251640 1 -- start start 2024-03-28T14:08:38.200 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:38.194+0000 7fa53b251640 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fa53414b640 0x7fa5341413f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:38.200 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:38.194+0000 7fa53b251640 1 --2- >> v2:172.21.15.171:3300/0 conn(0x7fa534141930 0x7fa534152080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:38.200 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:38.194+0000 7fa53b251640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa5340b5050 con 0x7fa5340bd9e0 2024-03-28T14:08:38.200 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:38.194+0000 7fa53b251640 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 -- 0x7fa5340b70b0 con 0x7fa53414b640 2024-03-28T14:08:38.200 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:38.194+0000 7fa53b251640 1 -- --> v2:172.21.15.171:3300/0 -- mon_getmap magic: 0 -- 0x7fa5340bddc0 con 0x7fa534141930 2024-03-28T14:08:38.200 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:38.194+0000 7fa533fff640 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fa53414b640 0x7fa5341413f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:38.200 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:38.194+0000 7fa538fed640 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7fa5340bd9e0 0x7fa53414af30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:59788/0 (socket says 172.21.15.104:59788) 2024-03-28T14:08:38.200 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:38.194+0000 7fa538fed640 1 -- 172.21.15.104:0/1369569362 learned_addr learned my addr 172.21.15.104:0/1369569362 (peer_addr_for_me v1:172.21.15.104:0/0) 2024-03-28T14:08:38.200 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:38.198+0000 7fa5397ee640 1 --2- 172.21.15.104:0/1369569362 >> v2:172.21.15.171:3300/0 conn(0x7fa534141930 0x7fa534152080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:38.200 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:38.198+0000 7fa533fff640 1 -- 172.21.15.104:0/1369569362 >> v2:172.21.15.171:3300/0 conn(0x7fa534141930 msgr2=0x7fa534152080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T14:08:38.200 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:38.198+0000 7fa533fff640 1 --2- 172.21.15.104:0/1369569362 >> v2:172.21.15.171:3300/0 conn(0x7fa534141930 0x7fa534152080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:38.201 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:38.198+0000 7fa533fff640 1 -- 172.21.15.104:0/1369569362 >> v1:172.21.15.104:6789/0 conn(0x7fa5340bd9e0 legacy=0x7fa53414af30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:38.201 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:38.198+0000 7fa533fff640 1 -- 172.21.15.104:0/1369569362 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa5340b52f0 con 0x7fa53414b640 2024-03-28T14:08:38.201 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:38.198+0000 7fa5337fe640 1 -- 172.21.15.104:0/1369569362 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1899946003 0 0) 0x7fa5340b5050 con 0x7fa5340bd9e0 2024-03-28T14:08:38.201 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:38.198+0000 7fa5397ee640 1 --2- 172.21.15.104:0/1369569362 >> v2:172.21.15.171:3300/0 conn(0x7fa534141930 0x7fa534152080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T14:08:38.202 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:38.198+0000 7fa533fff640 1 --2- 172.21.15.104:0/1369569362 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa53414b640 0x7fa5341413f0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7fa5340b4eb0 tx=0x7fa52801c330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9aa72f5cc192b210 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:08:38.202 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:38.198+0000 7fa5337fe640 1 -- 172.21.15.104:0/1369569362 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5280b0070 con 0x7fa53414b640 2024-03-28T14:08:38.202 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:38.198+0000 7fa5337fe640 1 -- 172.21.15.104:0/1369569362 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa5280ab600 con 0x7fa53414b640 2024-03-28T14:08:38.202 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:38.198+0000 7fa5337fe640 1 -- 172.21.15.104:0/1369569362 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5280ab9d0 con 0x7fa53414b640 2024-03-28T14:08:38.203 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:38.198+0000 7fa53b251640 1 -- 172.21.15.104:0/1369569362 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa53414b640 msgr2=0x7fa5341413f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:38.203 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:38.198+0000 7fa53b251640 1 --2- 172.21.15.104:0/1369569362 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa53414b640 0x7fa5341413f0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7fa5340b4eb0 tx=0x7fa52801c330 comp rx=0 tx=0).stop 2024-03-28T14:08:38.203 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:38.198+0000 7fa53b251640 1 -- 172.21.15.104:0/1369569362 shutdown_connections 2024-03-28T14:08:38.203 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:38.198+0000 7fa53b251640 1 --2- 172.21.15.104:0/1369569362 >> v2:172.21.15.171:3300/0 conn(0x7fa534141930 0x7fa534152080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:38.203 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:38.198+0000 7fa53b251640 1 --2- 172.21.15.104:0/1369569362 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa53414b640 0x7fa5341413f0 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:38.203 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:38.198+0000 7fa53b251640 1 -- 172.21.15.104:0/1369569362 >> 172.21.15.104:0/1369569362 conn(0x7fa534073530 msgr2=0x7fa5340bd070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T14:08:38.203 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:38.198+0000 7fa53b251640 1 -- 172.21.15.104:0/1369569362 shutdown_connections 2024-03-28T14:08:38.203 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:38.198+0000 7fa53b251640 1 -- 172.21.15.104:0/1369569362 wait complete. 2024-03-28T14:08:38.205 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:38.202+0000 7fa53b251640 1 Processor -- start 2024-03-28T14:08:38.205 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:38.202+0000 7fa53b251640 1 -- start start 2024-03-28T14:08:38.205 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:38.202+0000 7fa53b251640 1 --2- >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fa5340bd9e0 0x7fa534140a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:38.205 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:38.202+0000 7fa53b251640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fa534141930 0x7fa5341dff40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:38.205 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:38.202+0000 7fa53b251640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa53414b640 0x7fa5341342f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:38.205 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:38.202+0000 7fa53b251640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7fa5340b62c0 con 0x7fa53414b640 2024-03-28T14:08:38.206 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:38.202+0000 7fa53b251640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7fa5340b6140 con 0x7fa534141930 2024-03-28T14:08:38.206 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:38.202+0000 7fa53b251640 1 -- --> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] -- mon_getmap magic: 0 -- 0x7fa5340b4940 con 0x7fa5340bd9e0 2024-03-28T14:08:38.206 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:38.202+0000 7fa5397ee640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa53414b640 0x7fa5341342f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:38.206 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:38.202+0000 7fa5397ee640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa53414b640 0x7fa5341342f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:50276/0 (socket says 172.21.15.104:50276) 2024-03-28T14:08:38.206 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:38.202+0000 7fa5397ee640 1 -- 172.21.15.104:0/1092691576 learned_addr learned my addr 172.21.15.104:0/1092691576 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-03-28T14:08:38.206 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:38.202+0000 7fa538fed640 1 --2- 172.21.15.104:0/1092691576 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fa5340bd9e0 0x7fa534140a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:38.206 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:38.202+0000 7fa533fff640 1 --2- 172.21.15.104:0/1092691576 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fa534141930 0x7fa5341dff40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:38.206 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:38.202+0000 7fa5397ee640 1 -- 172.21.15.104:0/1092691576 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fa5340bd9e0 msgr2=0x7fa534140a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:38.206 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:38.202+0000 7fa5397ee640 1 --2- 172.21.15.104:0/1092691576 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fa5340bd9e0 0x7fa534140a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:38.206 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:38.202+0000 7fa5397ee640 1 -- 172.21.15.104:0/1092691576 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fa534141930 msgr2=0x7fa5341dff40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:38.206 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:38.202+0000 7fa5397ee640 1 --2- 172.21.15.104:0/1092691576 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fa534141930 0x7fa5341dff40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:38.206 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:38.202+0000 7fa5397ee640 1 -- 172.21.15.104:0/1092691576 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa534142960 con 0x7fa53414b640 2024-03-28T14:08:38.206 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:38.202+0000 7fa5397ee640 1 --2- 172.21.15.104:0/1092691576 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa53414b640 0x7fa5341342f0 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7fa52c00c7f0 tx=0x7fa52c00ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:08:38.207 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:38.202+0000 7fa5317fa640 1 -- 172.21.15.104:0/1092691576 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa52c015020 con 0x7fa53414b640 2024-03-28T14:08:38.207 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:38.202+0000 7fa5317fa640 1 -- 172.21.15.104:0/1092691576 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa52c004250 con 0x7fa53414b640 2024-03-28T14:08:38.207 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:38.202+0000 7fa53b251640 1 -- 172.21.15.104:0/1092691576 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa5341446b0 con 0x7fa53414b640 2024-03-28T14:08:38.207 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:38.202+0000 7fa5317fa640 1 -- 172.21.15.104:0/1092691576 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa52c003620 con 0x7fa53414b640 2024-03-28T14:08:38.207 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:38.202+0000 7fa53b251640 1 -- 172.21.15.104:0/1092691576 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa5341434b0 con 0x7fa53414b640 2024-03-28T14:08:38.208 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:38.202+0000 7fa53b251640 1 -- 172.21.15.104:0/1092691576 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa5340687c0 con 0x7fa53414b640 2024-03-28T14:08:38.208 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:38.202+0000 7fa5317fa640 1 -- 172.21.15.104:0/1092691576 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) ==== 87803+0+0 (secure 0 0 0) 0x7fa52c018020 con 0x7fa53414b640 2024-03-28T14:08:38.208 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:38.202+0000 7fa5317fa640 1 --2- 172.21.15.104:0/1092691576 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7fa520067fc0 0x7fa52006a480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:38.208 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:38.202+0000 7fa5317fa640 1 -- 172.21.15.104:0/1092691576 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 4857+0+0 (secure 0 0 0) 0x7fa52c086390 con 0x7fa53414b640 2024-03-28T14:08:38.208 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:38.202+0000 7fa538fed640 1 --2- 172.21.15.104:0/1092691576 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7fa520067fc0 0x7fa52006a480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:38.208 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:38.206+0000 7fa538fed640 1 --2- 172.21.15.104:0/1092691576 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7fa520067fc0 0x7fa52006a480 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fa524007fd0 tx=0x7fa524009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:08:38.211 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:38.206+0000 7fa5317fa640 1 -- 172.21.15.104:0/1092691576 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179117 (secure 0 0 0) 0x7fa52c01f020 con 0x7fa53414b640 2024-03-28T14:08:38.360 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:38.358+0000 7fa53b251640 1 -- 172.21.15.104:0/1092691576 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) -- 0x7fa5340b5be0 con 0x7fa53414b640 2024-03-28T14:08:38.361 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:38.358+0000 7fa5317fa640 1 -- 172.21.15.104:0/1092691576 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v26) ==== 84+0+65 (secure 0 0 0) 0x7fa52c056f20 con 0x7fa53414b640 2024-03-28T14:08:38.361 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-03-28T14:08:38.361 INFO:teuthology.orchestra.run.smithi104.stdout:[ 2024-03-28T14:08:38.361 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-03-28T14:08:38.361 INFO:teuthology.orchestra.run.smithi104.stdout: "poolnum": 1, 2024-03-28T14:08:38.361 INFO:teuthology.orchestra.run.smithi104.stdout: "poolname": ".mgr" 2024-03-28T14:08:38.362 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-03-28T14:08:38.362 INFO:teuthology.orchestra.run.smithi104.stdout:] 2024-03-28T14:08:38.363 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:38.358+0000 7fa53b251640 1 -- 172.21.15.104:0/1092691576 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7fa520067fc0 msgr2=0x7fa52006a480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:38.363 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:38.358+0000 7fa53b251640 1 --2- 172.21.15.104:0/1092691576 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7fa520067fc0 0x7fa52006a480 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fa524007fd0 tx=0x7fa524009040 comp rx=0 tx=0).stop 2024-03-28T14:08:38.363 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:38.358+0000 7fa53b251640 1 -- 172.21.15.104:0/1092691576 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa53414b640 msgr2=0x7fa5341342f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:38.363 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:38.358+0000 7fa53b251640 1 --2- 172.21.15.104:0/1092691576 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa53414b640 0x7fa5341342f0 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7fa52c00c7f0 tx=0x7fa52c00ccb0 comp rx=0 tx=0).stop 2024-03-28T14:08:38.363 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:38.358+0000 7fa53b251640 1 -- 172.21.15.104:0/1092691576 shutdown_connections 2024-03-28T14:08:38.364 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:38.358+0000 7fa53b251640 1 --2- 172.21.15.104:0/1092691576 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7fa520067fc0 0x7fa52006a480 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:38.364 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:38.358+0000 7fa53b251640 1 --2- 172.21.15.104:0/1092691576 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa53414b640 0x7fa5341342f0 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:38.364 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:38.358+0000 7fa53b251640 1 --2- 172.21.15.104:0/1092691576 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fa534141930 0x7fa5341dff40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:38.364 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:38.358+0000 7fa53b251640 1 --2- 172.21.15.104:0/1092691576 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fa5340bd9e0 0x7fa534140a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:38.364 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:38.358+0000 7fa53b251640 1 -- 172.21.15.104:0/1092691576 >> 172.21.15.104:0/1092691576 conn(0x7fa534073530 msgr2=0x7fa534067330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T14:08:38.364 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:38.358+0000 7fa53b251640 1 -- 172.21.15.104:0/1092691576 shutdown_connections 2024-03-28T14:08:38.364 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:38.358+0000 7fa53b251640 1 -- 172.21.15.104:0/1092691576 wait complete. 2024-03-28T14:08:38.378 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-03-28T14:08:38.379 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-03-28T14:08:38.502 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:38.498+0000 7fec5d5a5640 1 Processor -- start 2024-03-28T14:08:38.502 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:38.498+0000 7fec5d5a5640 1 -- start start 2024-03-28T14:08:38.504 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:38.498+0000 7fec5d5a5640 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fec5806c8f0 0x7fec580b2740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:38.504 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:38.498+0000 7fec5d5a5640 1 --2- >> v2:172.21.15.171:3300/0 conn(0x7fec58149380 0x7fec581501b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:38.504 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:38.498+0000 7fec5d5a5640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fec5806ccd0 con 0x7fec580b3730 2024-03-28T14:08:38.504 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:38.498+0000 7fec5d5a5640 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 -- 0x7fec580b0b30 con 0x7fec5806c8f0 2024-03-28T14:08:38.504 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:38.498+0000 7fec5d5a5640 1 -- --> v2:172.21.15.171:3300/0 -- mon_getmap magic: 0 -- 0x7fec580b22b0 con 0x7fec58149380 2024-03-28T14:08:38.504 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:38.498+0000 7fec567fc640 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7fec580b3730 0x7fec58148c70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:59802/0 (socket says 172.21.15.104:59802) 2024-03-28T14:08:38.504 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:38.498+0000 7fec567fc640 1 -- 172.21.15.104:0/99147857 learned_addr learned my addr 172.21.15.104:0/99147857 (peer_addr_for_me v1:172.21.15.104:0/0) 2024-03-28T14:08:38.504 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:38.498+0000 7fec56ffd640 1 --2- 172.21.15.104:0/99147857 >> v2:172.21.15.104:3300/0 conn(0x7fec5806c8f0 0x7fec580b2740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:38.504 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:38.498+0000 7fec577fe640 1 --2- 172.21.15.104:0/99147857 >> v2:172.21.15.171:3300/0 conn(0x7fec58149380 0x7fec581501b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:38.505 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:38.498+0000 7fec55ffb640 1 -- 172.21.15.104:0/99147857 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2049404792 0 0) 0x7fec5806ccd0 con 0x7fec580b3730 2024-03-28T14:08:38.505 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:38.498+0000 7fec55ffb640 1 -- 172.21.15.104:0/99147857 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fec44003530 con 0x7fec580b3730 2024-03-28T14:08:38.505 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:38.502+0000 7fec56ffd640 1 -- 172.21.15.104:0/99147857 >> v2:172.21.15.171:3300/0 conn(0x7fec58149380 msgr2=0x7fec581501b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T14:08:38.505 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:38.502+0000 7fec56ffd640 1 --2- 172.21.15.104:0/99147857 >> v2:172.21.15.171:3300/0 conn(0x7fec58149380 0x7fec581501b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:38.505 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:38.502+0000 7fec56ffd640 1 -- 172.21.15.104:0/99147857 >> v1:172.21.15.104:6789/0 conn(0x7fec580b3730 legacy=0x7fec58148c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:38.505 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:38.502+0000 7fec56ffd640 1 -- 172.21.15.104:0/99147857 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fec580b1980 con 0x7fec5806c8f0 2024-03-28T14:08:38.506 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:38.502+0000 7fec56ffd640 1 --2- 172.21.15.104:0/99147857 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fec5806c8f0 0x7fec580b2740 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7fec580b2220 tx=0x7fec4001c3f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=23b6a33ac852c657 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:08:38.506 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:38.502+0000 7fec55ffb640 1 -- 172.21.15.104:0/99147857 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec400b0070 con 0x7fec5806c8f0 2024-03-28T14:08:38.507 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:38.502+0000 7fec55ffb640 1 -- 172.21.15.104:0/99147857 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fec400ab600 con 0x7fec5806c8f0 2024-03-28T14:08:38.507 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:38.502+0000 7fec55ffb640 1 -- 172.21.15.104:0/99147857 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec400ab9d0 con 0x7fec5806c8f0 2024-03-28T14:08:38.508 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:38.502+0000 7fec5d5a5640 1 -- 172.21.15.104:0/99147857 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fec5806c8f0 msgr2=0x7fec580b2740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:38.508 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:38.502+0000 7fec5d5a5640 1 --2- 172.21.15.104:0/99147857 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fec5806c8f0 0x7fec580b2740 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7fec580b2220 tx=0x7fec4001c3f0 comp rx=0 tx=0).stop 2024-03-28T14:08:38.508 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:38.502+0000 7fec5d5a5640 1 -- 172.21.15.104:0/99147857 shutdown_connections 2024-03-28T14:08:38.508 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:38.502+0000 7fec5d5a5640 1 --2- 172.21.15.104:0/99147857 >> v2:172.21.15.171:3300/0 conn(0x7fec58149380 0x7fec581501b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:38.508 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:38.502+0000 7fec5d5a5640 1 --2- 172.21.15.104:0/99147857 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fec5806c8f0 0x7fec580b2740 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:38.508 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:38.502+0000 7fec5d5a5640 1 -- 172.21.15.104:0/99147857 >> 172.21.15.104:0/99147857 conn(0x7fec580b8f00 msgr2=0x7fec580b9310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T14:08:38.508 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:38.502+0000 7fec5d5a5640 1 -- 172.21.15.104:0/99147857 shutdown_connections 2024-03-28T14:08:38.508 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:38.502+0000 7fec5d5a5640 1 -- 172.21.15.104:0/99147857 wait complete. 2024-03-28T14:08:38.510 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:38.506+0000 7fec5d5a5640 1 Processor -- start 2024-03-28T14:08:38.510 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:38.506+0000 7fec5d5a5640 1 -- start start 2024-03-28T14:08:38.511 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:38.506+0000 7fec5d5a5640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fec5806c8f0 0x7fec58144350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:38.511 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:38.506+0000 7fec5d5a5640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fec580b3730 0x7fec58143510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:38.511 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:38.506+0000 7fec5d5a5640 1 --2- >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fec58149380 0x7fec58141e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:38.511 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:38.506+0000 7fec5d5a5640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7fec5806ddf0 con 0x7fec5806c8f0 2024-03-28T14:08:38.511 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:38.506+0000 7fec5d5a5640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7fec5806dc70 con 0x7fec580b3730 2024-03-28T14:08:38.511 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:38.506+0000 7fec5d5a5640 1 -- --> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] -- mon_getmap magic: 0 -- 0x7fec58152f10 con 0x7fec58149380 2024-03-28T14:08:38.511 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:38.506+0000 7fec56ffd640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fec5806c8f0 0x7fec58144350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:38.511 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:38.506+0000 7fec56ffd640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fec5806c8f0 0x7fec58144350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:50288/0 (socket says 172.21.15.104:50288) 2024-03-28T14:08:38.511 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:38.506+0000 7fec56ffd640 1 -- 172.21.15.104:0/881510870 learned_addr learned my addr 172.21.15.104:0/881510870 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-03-28T14:08:38.511 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:38.506+0000 7fec577fe640 1 --2- 172.21.15.104:0/881510870 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fec58149380 0x7fec58141e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:38.511 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:38.506+0000 7fec567fc640 1 --2- 172.21.15.104:0/881510870 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fec580b3730 0x7fec58143510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:38.511 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:38.506+0000 7fec56ffd640 1 -- 172.21.15.104:0/881510870 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fec58149380 msgr2=0x7fec58141e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:38.511 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:38.506+0000 7fec56ffd640 1 --2- 172.21.15.104:0/881510870 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fec58149380 0x7fec58141e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:38.511 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:38.506+0000 7fec56ffd640 1 -- 172.21.15.104:0/881510870 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fec580b3730 msgr2=0x7fec58143510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:38.512 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:38.506+0000 7fec56ffd640 1 --2- 172.21.15.104:0/881510870 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fec580b3730 0x7fec58143510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:38.512 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:38.506+0000 7fec56ffd640 1 -- 172.21.15.104:0/881510870 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fec581de520 con 0x7fec5806c8f0 2024-03-28T14:08:38.512 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:38.506+0000 7fec567fc640 1 --2- 172.21.15.104:0/881510870 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fec580b3730 0x7fec58143510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T14:08:38.512 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:38.506+0000 7fec56ffd640 1 --2- 172.21.15.104:0/881510870 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fec5806c8f0 0x7fec58144350 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7fec4001a020 tx=0x7fec40019190 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:08:38.512 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:38.506+0000 7fec4f7fe640 1 -- 172.21.15.104:0/881510870 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec400b0040 con 0x7fec5806c8f0 2024-03-28T14:08:38.512 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:38.506+0000 7fec4f7fe640 1 -- 172.21.15.104:0/881510870 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fec400b9070 con 0x7fec5806c8f0 2024-03-28T14:08:38.512 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:38.506+0000 7fec5d5a5640 1 -- 172.21.15.104:0/881510870 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fec58143c40 con 0x7fec5806c8f0 2024-03-28T14:08:38.512 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:38.506+0000 7fec4f7fe640 1 -- 172.21.15.104:0/881510870 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec400b4650 con 0x7fec5806c8f0 2024-03-28T14:08:38.512 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:38.506+0000 7fec5d5a5640 1 -- 172.21.15.104:0/881510870 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fec581449e0 con 0x7fec5806c8f0 2024-03-28T14:08:38.513 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:38.506+0000 7fec5d5a5640 1 -- 172.21.15.104:0/881510870 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fec58143a50 con 0x7fec5806c8f0 2024-03-28T14:08:38.513 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:38.506+0000 7fec4f7fe640 1 -- 172.21.15.104:0/881510870 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) ==== 87803+0+0 (secure 0 0 0) 0x7fec400b47f0 con 0x7fec5806c8f0 2024-03-28T14:08:38.513 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:38.510+0000 7fec4f7fe640 1 --2- 172.21.15.104:0/881510870 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7fec38067fc0 0x7fec3806a480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:38.513 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:38.510+0000 7fec4f7fe640 1 -- 172.21.15.104:0/881510870 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 4857+0+0 (secure 0 0 0) 0x7fec4011fdd0 con 0x7fec5806c8f0 2024-03-28T14:08:38.513 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:38.510+0000 7fec567fc640 1 --2- 172.21.15.104:0/881510870 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7fec38067fc0 0x7fec3806a480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:38.513 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:38.510+0000 7fec567fc640 1 --2- 172.21.15.104:0/881510870 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7fec38067fc0 0x7fec3806a480 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fec58143070 tx=0x7fec48009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:08:38.516 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:38.514+0000 7fec4f7fe640 1 -- 172.21.15.104:0/881510870 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179117 (secure 0 0 0) 0x7fec40016070 con 0x7fec5806c8f0 2024-03-28T14:08:38.679 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:38.674+0000 7fec5d5a5640 1 -- 172.21.15.104:0/881510870 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) -- 0x7fec580687c0 con 0x7fec5806c8f0 2024-03-28T14:08:39.463 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:39.458+0000 7fec4f7fe640 1 -- 172.21.15.104:0/881510870 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v27) ==== 138+0+0 (secure 0 0 0) 0x7fec400f0960 con 0x7fec5806c8f0 2024-03-28T14:08:39.463 INFO:teuthology.orchestra.run.smithi104.stderr:pool 'cephfs_metadata' created 2024-03-28T14:08:39.466 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:39.462+0000 7fec5d5a5640 1 -- 172.21.15.104:0/881510870 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7fec38067fc0 msgr2=0x7fec3806a480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:39.467 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:39.462+0000 7fec5d5a5640 1 --2- 172.21.15.104:0/881510870 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7fec38067fc0 0x7fec3806a480 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fec58143070 tx=0x7fec48009040 comp rx=0 tx=0).stop 2024-03-28T14:08:39.467 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:39.462+0000 7fec5d5a5640 1 -- 172.21.15.104:0/881510870 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fec5806c8f0 msgr2=0x7fec58144350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:39.467 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:39.462+0000 7fec5d5a5640 1 --2- 172.21.15.104:0/881510870 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fec5806c8f0 0x7fec58144350 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7fec4001a020 tx=0x7fec40019190 comp rx=0 tx=0).stop 2024-03-28T14:08:39.467 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:39.462+0000 7fec5d5a5640 1 -- 172.21.15.104:0/881510870 shutdown_connections 2024-03-28T14:08:39.467 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:39.462+0000 7fec5d5a5640 1 --2- 172.21.15.104:0/881510870 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fec58149380 0x7fec58141e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:39.467 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:39.462+0000 7fec5d5a5640 1 --2- 172.21.15.104:0/881510870 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fec580b3730 0x7fec58143510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:39.467 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:39.462+0000 7fec5d5a5640 1 --2- 172.21.15.104:0/881510870 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7fec38067fc0 0x7fec3806a480 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:39.467 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:39.462+0000 7fec5d5a5640 1 --2- 172.21.15.104:0/881510870 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fec5806c8f0 0x7fec58144350 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:39.467 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:39.462+0000 7fec5d5a5640 1 -- 172.21.15.104:0/881510870 >> 172.21.15.104:0/881510870 conn(0x7fec580b8f00 msgr2=0x7fec580b4850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T14:08:39.467 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:39.462+0000 7fec5d5a5640 1 -- 172.21.15.104:0/881510870 shutdown_connections 2024-03-28T14:08:39.467 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:39.462+0000 7fec5d5a5640 1 -- 172.21.15.104:0/881510870 wait complete. 2024-03-28T14:08:39.491 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-03-28T14:08:39.621 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:39.618+0000 7fba0619f640 1 Processor -- start 2024-03-28T14:08:39.621 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:39.618+0000 7fba0619f640 1 -- start start 2024-03-28T14:08:39.622 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:39.618+0000 7fba0619f640 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fba0006ebe0 0x7fba0006d660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:39.622 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:39.618+0000 7fba0619f640 1 --2- >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fba0014b270 0x7fba0014b700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:39.622 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:39.618+0000 7fba0619f640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fba0014bc40 con 0x7fba000b59a0 2024-03-28T14:08:39.622 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:39.618+0000 7fba0619f640 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 -- 0x7fba0014ade0 con 0x7fba0006ebe0 2024-03-28T14:08:39.623 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:39.618+0000 7fba0619f640 1 -- --> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] -- mon_getmap magic: 0 -- 0x7fba0006efc0 con 0x7fba0014b270 2024-03-28T14:08:39.623 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:39.618+0000 7fb9feffd640 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7fba000b59a0 0x7fba00144a00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:59808/0 (socket says 172.21.15.104:59808) 2024-03-28T14:08:39.623 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:39.618+0000 7fb9feffd640 1 -- 172.21.15.104:0/1812331151 learned_addr learned my addr 172.21.15.104:0/1812331151 (peer_addr_for_me v1:172.21.15.104:0/0) 2024-03-28T14:08:39.623 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:39.618+0000 7fb9ff7fe640 1 --2- 172.21.15.104:0/1812331151 >> v2:172.21.15.104:3300/0 conn(0x7fba0006ebe0 0x7fba0006d660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:39.623 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:39.618+0000 7fb9fffff640 1 --2- 172.21.15.104:0/1812331151 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fba0014b270 0x7fba0014b700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:39.623 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:39.618+0000 7fb9ff7fe640 1 -- 172.21.15.104:0/1812331151 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fba0014b270 msgr2=0x7fba0014b700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T14:08:39.623 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:39.618+0000 7fb9ff7fe640 1 --2- 172.21.15.104:0/1812331151 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fba0014b270 0x7fba0014b700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:39.623 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:39.618+0000 7fb9ff7fe640 1 -- 172.21.15.104:0/1812331151 >> v1:172.21.15.104:6789/0 conn(0x7fba000b59a0 legacy=0x7fba00144a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:39.623 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:39.618+0000 7fb9ff7fe640 1 -- 172.21.15.104:0/1812331151 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fba0006dd10 con 0x7fba0006ebe0 2024-03-28T14:08:39.623 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:39.618+0000 7fb9fe7fc640 1 -- 172.21.15.104:0/1812331151 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 456910666 0 0) 0x7fba0014bc40 con 0x7fba000b59a0 2024-03-28T14:08:39.624 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:39.622+0000 7fb9ff7fe640 1 --2- 172.21.15.104:0/1812331151 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fba0006ebe0 0x7fba0006d660 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7fb9f40097c0 tx=0x7fb9f401c410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6ac1f7db9de8adb3 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:08:39.624 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:39.622+0000 7fb9fe7fc640 1 -- 172.21.15.104:0/1812331151 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9f40b0070 con 0x7fba0006ebe0 2024-03-28T14:08:39.624 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:39.622+0000 7fb9fe7fc640 1 -- 172.21.15.104:0/1812331151 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb9f40ab600 con 0x7fba0006ebe0 2024-03-28T14:08:39.625 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:39.622+0000 7fb9fe7fc640 1 -- 172.21.15.104:0/1812331151 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9f40ab9d0 con 0x7fba0006ebe0 2024-03-28T14:08:39.625 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:39.622+0000 7fba0619f640 1 -- 172.21.15.104:0/1812331151 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fba0006ebe0 msgr2=0x7fba0006d660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:39.625 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:39.622+0000 7fba0619f640 1 --2- 172.21.15.104:0/1812331151 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fba0006ebe0 0x7fba0006d660 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7fb9f40097c0 tx=0x7fb9f401c410 comp rx=0 tx=0).stop 2024-03-28T14:08:39.626 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:39.622+0000 7fba0619f640 1 -- 172.21.15.104:0/1812331151 shutdown_connections 2024-03-28T14:08:39.626 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:39.622+0000 7fba0619f640 1 --2- 172.21.15.104:0/1812331151 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fba0014b270 0x7fba0014b700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:39.626 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:39.622+0000 7fba0619f640 1 --2- 172.21.15.104:0/1812331151 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fba0006ebe0 0x7fba0006d660 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:39.626 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:39.622+0000 7fba0619f640 1 -- 172.21.15.104:0/1812331151 >> 172.21.15.104:0/1812331151 conn(0x7fba000bb170 msgr2=0x7fba000bb580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T14:08:39.626 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:39.622+0000 7fba0619f640 1 -- 172.21.15.104:0/1812331151 shutdown_connections 2024-03-28T14:08:39.626 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:39.622+0000 7fba0619f640 1 -- 172.21.15.104:0/1812331151 wait complete. 2024-03-28T14:08:39.628 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:39.622+0000 7fba0619f640 1 Processor -- start 2024-03-28T14:08:39.628 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:39.622+0000 7fba0619f640 1 -- start start 2024-03-28T14:08:39.628 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:39.622+0000 7fba0619f640 1 --2- >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fba0006ebe0 0x7fba00145940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:39.628 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:39.622+0000 7fba0619f640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fba000b59a0 0x7fba001e01d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:39.628 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:39.622+0000 7fba0619f640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fba0014b270 0x7fba001e0710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:39.629 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:39.622+0000 7fba0619f640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7fba00154ab0 con 0x7fba0014b270 2024-03-28T14:08:39.629 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:39.622+0000 7fba0619f640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7fba00154930 con 0x7fba000b59a0 2024-03-28T14:08:39.629 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:39.622+0000 7fba0619f640 1 -- --> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] -- mon_getmap magic: 0 -- 0x7fba00154c30 con 0x7fba0006ebe0 2024-03-28T14:08:39.629 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:39.622+0000 7fb9fffff640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fba0014b270 0x7fba001e0710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:39.629 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:39.622+0000 7fb9fffff640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fba0014b270 0x7fba001e0710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:50308/0 (socket says 172.21.15.104:50308) 2024-03-28T14:08:39.629 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:39.622+0000 7fb9fffff640 1 -- 172.21.15.104:0/4265360291 learned_addr learned my addr 172.21.15.104:0/4265360291 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-03-28T14:08:39.629 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:39.622+0000 7fb9ff7fe640 1 --2- 172.21.15.104:0/4265360291 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fba0006ebe0 0x7fba00145940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:39.629 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:39.622+0000 7fb9fffff640 1 -- 172.21.15.104:0/4265360291 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fba0006ebe0 msgr2=0x7fba00145940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:39.629 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:39.622+0000 7fb9fffff640 1 --2- 172.21.15.104:0/4265360291 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fba0006ebe0 0x7fba00145940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:39.629 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:39.622+0000 7fb9feffd640 1 --2- 172.21.15.104:0/4265360291 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fba000b59a0 0x7fba001e01d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:39.629 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:39.622+0000 7fb9fffff640 1 -- 172.21.15.104:0/4265360291 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fba000b59a0 msgr2=0x7fba001e01d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:39.630 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:39.622+0000 7fb9fffff640 1 --2- 172.21.15.104:0/4265360291 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fba000b59a0 0x7fba001e01d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:39.630 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:39.626+0000 7fb9fffff640 1 -- 172.21.15.104:0/4265360291 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fba00146960 con 0x7fba0014b270 2024-03-28T14:08:39.630 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:39.626+0000 7fb9fffff640 1 --2- 172.21.15.104:0/4265360291 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fba0014b270 0x7fba001e0710 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7fb9f8009d30 tx=0x7fb9f800c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:08:39.630 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:39.626+0000 7fb9e7fff640 1 -- 172.21.15.104:0/4265360291 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9f8018020 con 0x7fba0014b270 2024-03-28T14:08:39.630 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:39.626+0000 7fb9e7fff640 1 -- 172.21.15.104:0/4265360291 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb9f8014480 con 0x7fba0014b270 2024-03-28T14:08:39.630 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:39.626+0000 7fba0619f640 1 -- 172.21.15.104:0/4265360291 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fba00146eb0 con 0x7fba0014b270 2024-03-28T14:08:39.630 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:39.626+0000 7fb9e7fff640 1 -- 172.21.15.104:0/4265360291 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9f8014870 con 0x7fba0014b270 2024-03-28T14:08:39.630 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:39.626+0000 7fba0619f640 1 -- 172.21.15.104:0/4265360291 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fba000b1ef0 con 0x7fba0014b270 2024-03-28T14:08:39.631 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:39.626+0000 7fba0619f640 1 -- 172.21.15.104:0/4265360291 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fba000687c0 con 0x7fba0014b270 2024-03-28T14:08:39.632 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:39.626+0000 7fb9e7fff640 1 -- 172.21.15.104:0/4265360291 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) ==== 87803+0+0 (secure 0 0 0) 0x7fb9f801b020 con 0x7fba0014b270 2024-03-28T14:08:39.632 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:39.626+0000 7fb9e7fff640 1 --2- 172.21.15.104:0/4265360291 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7fb9ec067fc0 0x7fb9ec06a480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:39.632 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:39.626+0000 7fb9e7fff640 1 -- 172.21.15.104:0/4265360291 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 5227+0+0 (secure 0 0 0) 0x7fb9f8087370 con 0x7fba0014b270 2024-03-28T14:08:39.632 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:39.626+0000 7fb9ff7fe640 1 --2- 172.21.15.104:0/4265360291 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7fb9ec067fc0 0x7fb9ec06a480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:39.632 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:39.626+0000 7fb9ff7fe640 1 --2- 172.21.15.104:0/4265360291 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7fb9ec067fc0 0x7fb9ec06a480 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fb9f40ad000 tx=0x7fb9f40a6000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:08:39.634 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:39.630+0000 7fb9e7fff640 1 -- 172.21.15.104:0/4265360291 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179117 (secure 0 0 0) 0x7fb9f8022070 con 0x7fba0014b270 2024-03-28T14:08:39.797 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:39.794+0000 7fba0619f640 1 -- 172.21.15.104:0/4265360291 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) -- 0x7fba000b20f0 con 0x7fba0014b270 2024-03-28T14:08:40.474 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:40.470+0000 7fb9e7fff640 1 -- 172.21.15.104:0/4265360291 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v28) ==== 170+0+0 (secure 0 0 0) 0x7fb9f8010040 con 0x7fba0014b270 2024-03-28T14:08:40.474 INFO:teuthology.orchestra.run.smithi104.stderr:pool 'cephfs_data' created 2024-03-28T14:08:40.477 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:40.474+0000 7fba0619f640 1 -- 172.21.15.104:0/4265360291 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7fb9ec067fc0 msgr2=0x7fb9ec06a480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:40.477 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:40.474+0000 7fba0619f640 1 --2- 172.21.15.104:0/4265360291 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7fb9ec067fc0 0x7fb9ec06a480 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fb9f40ad000 tx=0x7fb9f40a6000 comp rx=0 tx=0).stop 2024-03-28T14:08:40.478 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:40.474+0000 7fba0619f640 1 -- 172.21.15.104:0/4265360291 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fba0014b270 msgr2=0x7fba001e0710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:40.478 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:40.474+0000 7fba0619f640 1 --2- 172.21.15.104:0/4265360291 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fba0014b270 0x7fba001e0710 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7fb9f8009d30 tx=0x7fb9f800c490 comp rx=0 tx=0).stop 2024-03-28T14:08:40.478 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:40.474+0000 7fba0619f640 1 -- 172.21.15.104:0/4265360291 shutdown_connections 2024-03-28T14:08:40.479 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:40.474+0000 7fba0619f640 1 --2- 172.21.15.104:0/4265360291 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7fb9ec067fc0 0x7fb9ec06a480 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:40.479 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:40.474+0000 7fba0619f640 1 --2- 172.21.15.104:0/4265360291 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fba0014b270 0x7fba001e0710 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:40.479 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:40.474+0000 7fba0619f640 1 --2- 172.21.15.104:0/4265360291 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fba000b59a0 0x7fba001e01d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:40.479 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:40.474+0000 7fba0619f640 1 --2- 172.21.15.104:0/4265360291 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fba0006ebe0 0x7fba00145940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:40.479 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:40.474+0000 7fba0619f640 1 -- 172.21.15.104:0/4265360291 >> 172.21.15.104:0/4265360291 conn(0x7fba000bb170 msgr2=0x7fba001427b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T14:08:40.479 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:40.474+0000 7fba0619f640 1 -- 172.21.15.104:0/4265360291 shutdown_connections 2024-03-28T14:08:40.479 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:40.474+0000 7fba0619f640 1 -- 172.21.15.104:0/4265360291 wait complete. 2024-03-28T14:08:40.512 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-03-28T14:08:40.640 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:40.638+0000 7f27e0ae3640 1 Processor -- start 2024-03-28T14:08:40.640 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:40.638+0000 7f27e0ae3640 1 -- start start 2024-03-28T14:08:40.641 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:40.638+0000 7f27e0ae3640 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f27dc0b1710 0x7f27dc0b1b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:40.641 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:40.638+0000 7f27e0ae3640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f27dc0b20a0 con 0x7f27dc0bb8c0 2024-03-28T14:08:40.641 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:40.638+0000 7f27e0ae3640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f27dc0bbca0 con 0x7f27dc0b6790 2024-03-28T14:08:40.642 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:40.638+0000 7f27e0ae3640 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 -- 0x7f27dc0b7430 con 0x7f27dc0b1710 2024-03-28T14:08:40.642 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:40.638+0000 7f27dad9d640 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f27dc0b1710 0x7f27dc0b1b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:40.642 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:40.638+0000 7f27da59c640 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f27dc0bb8c0 0x7f27dc0b5950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:59820/0 (socket says 172.21.15.104:59820) 2024-03-28T14:08:40.642 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:40.638+0000 7f27da59c640 1 -- 172.21.15.104:0/2788472991 learned_addr learned my addr 172.21.15.104:0/2788472991 (peer_addr_for_me v1:172.21.15.104:0/0) 2024-03-28T14:08:40.642 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:40.638+0000 7f27dad9d640 1 -- 172.21.15.104:0/2788472991 >> v1:172.21.15.171:6789/0 conn(0x7f27dc0b6790 legacy=0x7f27dc13f470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T14:08:40.642 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:40.638+0000 7f27d9d9b640 1 -- 172.21.15.104:0/2788472991 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1700955050 0 0) 0x7f27dc0b20a0 con 0x7f27dc0bb8c0 2024-03-28T14:08:40.642 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:40.638+0000 7f27dad9d640 1 -- 172.21.15.104:0/2788472991 >> v1:172.21.15.104:6789/0 conn(0x7f27dc0bb8c0 legacy=0x7f27dc0b5950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:40.642 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:40.638+0000 7f27dad9d640 1 -- 172.21.15.104:0/2788472991 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f27dc0b6de0 con 0x7f27dc0b1710 2024-03-28T14:08:40.643 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:40.638+0000 7f27dad9d640 1 --2- 172.21.15.104:0/2788472991 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f27dc0b1710 0x7f27dc0b1b60 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f27d4009d90 tx=0x7f27d401c810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9058259d54bcd7a2 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:08:40.644 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:40.642+0000 7f27d9d9b640 1 -- 172.21.15.104:0/2788472991 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27d40a66e0 con 0x7f27dc0b1710 2024-03-28T14:08:40.644 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:40.642+0000 7f27d9d9b640 1 -- 172.21.15.104:0/2788472991 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f27d40a6880 con 0x7f27dc0b1710 2024-03-28T14:08:40.644 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:40.642+0000 7f27d9d9b640 1 -- 172.21.15.104:0/2788472991 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27d40a6c50 con 0x7f27dc0b1710 2024-03-28T14:08:40.645 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:40.642+0000 7f27e0ae3640 1 -- 172.21.15.104:0/2788472991 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f27dc0b1710 msgr2=0x7f27dc0b1b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:40.645 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:40.642+0000 7f27e0ae3640 1 --2- 172.21.15.104:0/2788472991 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f27dc0b1710 0x7f27dc0b1b60 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f27d4009d90 tx=0x7f27d401c810 comp rx=0 tx=0).stop 2024-03-28T14:08:40.645 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:40.642+0000 7f27e0ae3640 1 -- 172.21.15.104:0/2788472991 shutdown_connections 2024-03-28T14:08:40.645 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:40.642+0000 7f27e0ae3640 1 --2- 172.21.15.104:0/2788472991 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f27dc0b1710 0x7f27dc0b1b60 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:40.645 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:40.642+0000 7f27e0ae3640 1 -- 172.21.15.104:0/2788472991 >> 172.21.15.104:0/2788472991 conn(0x7f27dc02c800 msgr2=0x7f27dc0baf50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T14:08:40.645 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:40.642+0000 7f27e0ae3640 1 -- 172.21.15.104:0/2788472991 shutdown_connections 2024-03-28T14:08:40.645 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:40.642+0000 7f27e0ae3640 1 -- 172.21.15.104:0/2788472991 wait complete. 2024-03-28T14:08:40.647 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:40.642+0000 7f27e0ae3640 1 Processor -- start 2024-03-28T14:08:40.647 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:40.642+0000 7f27e0ae3640 1 -- start start 2024-03-28T14:08:40.648 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:40.642+0000 7f27e0ae3640 1 --2- >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f27dc0b1710 0x7f27dc137de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:40.648 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:40.642+0000 7f27e0ae3640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f27dc0b6790 0x7f27dc0ba380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:40.648 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:40.642+0000 7f27e0ae3640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f27dc0bb8c0 0x7f27dc1de6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:40.648 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:40.642+0000 7f27e0ae3640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f27dc1538a0 con 0x7f27dc0b6790 2024-03-28T14:08:40.648 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:40.642+0000 7f27e0ae3640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f27dc13fd90 con 0x7f27dc0bb8c0 2024-03-28T14:08:40.648 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:40.642+0000 7f27e0ae3640 1 -- --> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] -- mon_getmap magic: 0 -- 0x7f27dc153a20 con 0x7f27dc0b1710 2024-03-28T14:08:40.648 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:40.642+0000 7f27d3fff640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f27dc0b6790 0x7f27dc0ba380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:40.648 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:40.642+0000 7f27d3fff640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f27dc0b6790 0x7f27dc0ba380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:50316/0 (socket says 172.21.15.104:50316) 2024-03-28T14:08:40.648 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:40.642+0000 7f27d3fff640 1 -- 172.21.15.104:0/2757686723 learned_addr learned my addr 172.21.15.104:0/2757686723 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-03-28T14:08:40.648 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:40.642+0000 7f27da59c640 1 --2- 172.21.15.104:0/2757686723 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f27dc0b1710 0x7f27dc137de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:40.648 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:40.642+0000 7f27dad9d640 1 --2- 172.21.15.104:0/2757686723 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f27dc0bb8c0 0x7f27dc1de6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:40.648 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:40.642+0000 7f27d3fff640 1 -- 172.21.15.104:0/2757686723 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f27dc0b1710 msgr2=0x7f27dc137de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:40.649 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:40.642+0000 7f27d3fff640 1 --2- 172.21.15.104:0/2757686723 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f27dc0b1710 0x7f27dc137de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:40.649 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:40.642+0000 7f27d3fff640 1 -- 172.21.15.104:0/2757686723 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f27dc0bb8c0 msgr2=0x7f27dc1de6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:40.649 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:40.642+0000 7f27d3fff640 1 --2- 172.21.15.104:0/2757686723 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f27dc0bb8c0 0x7f27dc1de6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:40.649 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:40.642+0000 7f27d3fff640 1 -- 172.21.15.104:0/2757686723 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f27dc135130 con 0x7f27dc0b6790 2024-03-28T14:08:40.649 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:40.642+0000 7f27dad9d640 1 --2- 172.21.15.104:0/2757686723 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f27dc0bb8c0 0x7f27dc1de6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T14:08:40.649 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:40.642+0000 7f27d3fff640 1 --2- 172.21.15.104:0/2757686723 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f27dc0b6790 0x7f27dc0ba380 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f27c400c480 tx=0x7f27c400c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:08:40.649 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:40.642+0000 7f27d2ffd640 1 -- 172.21.15.104:0/2757686723 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27c4012830 con 0x7f27dc0b6790 2024-03-28T14:08:40.649 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:40.642+0000 7f27d2ffd640 1 -- 172.21.15.104:0/2757686723 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f27c4017070 con 0x7f27dc0b6790 2024-03-28T14:08:40.649 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:40.642+0000 7f27d2ffd640 1 -- 172.21.15.104:0/2757686723 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27c4012c30 con 0x7f27dc0b6790 2024-03-28T14:08:40.649 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:40.642+0000 7f27e0ae3640 1 -- 172.21.15.104:0/2757686723 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f27dc135410 con 0x7f27dc0b6790 2024-03-28T14:08:40.649 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:40.642+0000 7f27e0ae3640 1 -- 172.21.15.104:0/2757686723 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f27dc137670 con 0x7f27dc0b6790 2024-03-28T14:08:40.649 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:40.646+0000 7f27e0ae3640 1 -- 172.21.15.104:0/2757686723 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f27dc0687c0 con 0x7f27dc0b6790 2024-03-28T14:08:40.650 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:40.646+0000 7f27d2ffd640 1 -- 172.21.15.104:0/2757686723 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) ==== 87803+0+0 (secure 0 0 0) 0x7f27c401c460 con 0x7f27dc0b6790 2024-03-28T14:08:40.650 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:40.646+0000 7f27d2ffd640 1 --2- 172.21.15.104:0/2757686723 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f27bc067fc0 0x7f27bc06a480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:40.650 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:40.646+0000 7f27d2ffd640 1 -- 172.21.15.104:0/2757686723 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(28..28 src has 1..28) ==== 5609+0+0 (secure 0 0 0) 0x7f27c4086d00 con 0x7f27dc0b6790 2024-03-28T14:08:40.650 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:40.646+0000 7f27da59c640 1 --2- 172.21.15.104:0/2757686723 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f27bc067fc0 0x7f27bc06a480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:40.650 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:40.646+0000 7f27da59c640 1 --2- 172.21.15.104:0/2757686723 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f27bc067fc0 0x7f27bc06a480 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f27cc000f70 tx=0x7f27cc007c10 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:08:40.653 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:40.650+0000 7f27d2ffd640 1 -- 172.21.15.104:0/2757686723 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179117 (secure 0 0 0) 0x7f27c4057680 con 0x7f27dc0b6790 2024-03-28T14:08:40.851 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:40.846+0000 7f27e0ae3640 1 -- 172.21.15.104:0/2757686723 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) -- 0x7f27dc145f90 con 0x7f27dc0b6790 2024-03-28T14:08:41.476 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:41.474+0000 7f27d2ffd640 1 -- 172.21.15.104:0/2757686723 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 4 and data pool 5 v12) ==== 172+0+0 (secure 0 0 0) 0x7f27c4056f80 con 0x7f27dc0b6790 2024-03-28T14:08:41.477 INFO:teuthology.orchestra.run.smithi104.stderr:new fs with metadata pool 4 and data pool 5 2024-03-28T14:08:41.483 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:41.474+0000 7f27e0ae3640 1 -- 172.21.15.104:0/2757686723 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f27bc067fc0 msgr2=0x7f27bc06a480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:41.483 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:41.474+0000 7f27e0ae3640 1 --2- 172.21.15.104:0/2757686723 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f27bc067fc0 0x7f27bc06a480 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f27cc000f70 tx=0x7f27cc007c10 comp rx=0 tx=0).stop 2024-03-28T14:08:41.483 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:41.474+0000 7f27e0ae3640 1 -- 172.21.15.104:0/2757686723 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f27dc0b6790 msgr2=0x7f27dc0ba380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:41.483 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:41.474+0000 7f27e0ae3640 1 --2- 172.21.15.104:0/2757686723 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f27dc0b6790 0x7f27dc0ba380 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f27c400c480 tx=0x7f27c400c940 comp rx=0 tx=0).stop 2024-03-28T14:08:41.483 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:41.478+0000 7f27e0ae3640 1 -- 172.21.15.104:0/2757686723 shutdown_connections 2024-03-28T14:08:41.483 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:41.478+0000 7f27e0ae3640 1 --2- 172.21.15.104:0/2757686723 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f27dc0bb8c0 0x7f27dc1de6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:41.483 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:41.478+0000 7f27e0ae3640 1 --2- 172.21.15.104:0/2757686723 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f27bc067fc0 0x7f27bc06a480 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:41.483 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:41.478+0000 7f27e0ae3640 1 --2- 172.21.15.104:0/2757686723 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f27dc0b6790 0x7f27dc0ba380 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:41.484 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:41.478+0000 7f27e0ae3640 1 --2- 172.21.15.104:0/2757686723 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f27dc0b1710 0x7f27dc137de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:41.484 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:41.478+0000 7f27e0ae3640 1 -- 172.21.15.104:0/2757686723 >> 172.21.15.104:0/2757686723 conn(0x7f27dc02c800 msgr2=0x7f27dc0b5850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T14:08:41.484 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:41.478+0000 7f27e0ae3640 1 -- 172.21.15.104:0/2757686723 shutdown_connections 2024-03-28T14:08:41.484 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:41.478+0000 7f27e0ae3640 1 -- 172.21.15.104:0/2757686723 wait complete. 2024-03-28T14:08:41.514 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-28T14:08:41.633 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:41.630+0000 7f1a3886f640 1 Processor -- start 2024-03-28T14:08:41.633 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:41.630+0000 7f1a3886f640 1 -- start start 2024-03-28T14:08:41.635 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:41.630+0000 7f1a3886f640 1 --2- >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f1a3406d970 0x7f1a3413f3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:41.635 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:41.630+0000 7f1a3886f640 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f1a340732a0 0x7f1a3413f920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:41.635 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:41.630+0000 7f1a3886f640 1 --2- >> v2:172.21.15.171:3300/0 conn(0x7f1a340b5460 0x7f1a340b0c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:41.635 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:41.630+0000 7f1a3886f640 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 -- 0x7f1a340b02a0 con 0x7f1a340732a0 2024-03-28T14:08:41.635 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:41.630+0000 7f1a3886f640 1 -- --> v2:172.21.15.171:3300/0 -- mon_getmap magic: 0 -- 0x7f1a3406dd50 con 0x7f1a340b5460 2024-03-28T14:08:41.635 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:41.630+0000 7f1a3886f640 1 -- --> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] -- mon_getmap magic: 0 -- 0x7f1a340b2d30 con 0x7f1a3406d970 2024-03-28T14:08:41.635 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:41.630+0000 7f1a31d9b640 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f1a340732a0 0x7f1a3413f920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:41.637 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:41.630+0000 7f1a31d9b640 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f1a340732a0 0x7f1a3413f920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:50330/0 (socket says 172.21.15.104:50330) 2024-03-28T14:08:41.637 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:41.630+0000 7f1a31d9b640 1 -- 172.21.15.104:0/2860012815 learned_addr learned my addr 172.21.15.104:0/2860012815 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-03-28T14:08:41.637 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:41.630+0000 7f1a3259c640 1 --2- 172.21.15.104:0/2860012815 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f1a3406d970 0x7f1a3413f3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:41.637 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:41.630+0000 7f1a32d9d640 1 --2- 172.21.15.104:0/2860012815 >> v2:172.21.15.171:3300/0 conn(0x7f1a340b5460 0x7f1a340b0c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:41.637 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:41.630+0000 7f1a31d9b640 1 -- 172.21.15.104:0/2860012815 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f1a3406d970 msgr2=0x7f1a3413f3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T14:08:41.637 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:41.630+0000 7f1a31d9b640 1 --2- 172.21.15.104:0/2860012815 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f1a3406d970 0x7f1a3413f3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:41.637 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:41.630+0000 7f1a31d9b640 1 -- 172.21.15.104:0/2860012815 >> v2:172.21.15.171:3300/0 conn(0x7f1a340b5460 msgr2=0x7f1a340b0c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T14:08:41.637 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:41.630+0000 7f1a31d9b640 1 --2- 172.21.15.104:0/2860012815 >> v2:172.21.15.171:3300/0 conn(0x7f1a340b5460 0x7f1a340b0c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:41.637 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:41.630+0000 7f1a31d9b640 1 -- 172.21.15.104:0/2860012815 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1a340b3010 con 0x7f1a340732a0 2024-03-28T14:08:41.639 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:41.634+0000 7f1a31d9b640 1 --2- 172.21.15.104:0/2860012815 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1a340732a0 0x7f1a3413f920 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f1a340b53f0 tx=0x7f1a2401c460 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cbfd531618c1a817 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:08:41.639 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:41.634+0000 7f1a3159a640 1 -- 172.21.15.104:0/2860012815 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a240ab600 con 0x7f1a340732a0 2024-03-28T14:08:41.639 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:41.634+0000 7f1a3159a640 1 -- 172.21.15.104:0/2860012815 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1a240b0070 con 0x7f1a340732a0 2024-03-28T14:08:41.639 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:41.634+0000 7f1a3159a640 1 -- 172.21.15.104:0/2860012815 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a240ab9c0 con 0x7f1a340732a0 2024-03-28T14:08:41.639 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:41.634+0000 7f1a3886f640 1 -- 172.21.15.104:0/2860012815 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1a340732a0 msgr2=0x7f1a3413f920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:41.639 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:41.634+0000 7f1a3886f640 1 --2- 172.21.15.104:0/2860012815 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1a340732a0 0x7f1a3413f920 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f1a340b53f0 tx=0x7f1a2401c460 comp rx=0 tx=0).stop 2024-03-28T14:08:41.639 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:41.634+0000 7f1a3886f640 1 -- 172.21.15.104:0/2860012815 shutdown_connections 2024-03-28T14:08:41.639 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:41.634+0000 7f1a3886f640 1 --2- 172.21.15.104:0/2860012815 >> v2:172.21.15.171:3300/0 conn(0x7f1a340b5460 0x7f1a340b0c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:41.639 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:41.634+0000 7f1a3886f640 1 --2- 172.21.15.104:0/2860012815 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1a340732a0 0x7f1a3413f920 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:41.639 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:41.634+0000 7f1a3886f640 1 --2- 172.21.15.104:0/2860012815 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f1a3406d970 0x7f1a3413f3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:41.639 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:41.634+0000 7f1a3886f640 1 -- 172.21.15.104:0/2860012815 >> 172.21.15.104:0/2860012815 conn(0x7f1a340b58a0 msgr2=0x7f1a340b5cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T14:08:41.640 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:41.634+0000 7f1a3886f640 1 -- 172.21.15.104:0/2860012815 shutdown_connections 2024-03-28T14:08:41.640 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:41.634+0000 7f1a3886f640 1 -- 172.21.15.104:0/2860012815 wait complete. 2024-03-28T14:08:41.640 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:41.634+0000 7f1a3886f640 1 Processor -- start 2024-03-28T14:08:41.640 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:41.634+0000 7f1a3886f640 1 -- start start 2024-03-28T14:08:41.641 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:41.638+0000 7f1a3886f640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1a3406d970 0x7f1a34137fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:41.641 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:41.638+0000 7f1a3886f640 1 --2- >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f1a340732a0 0x7f1a341384f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:41.642 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:41.638+0000 7f1a3886f640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f1a340b5460 0x7f1a341e3840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:41.642 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:41.638+0000 7f1a3886f640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f1a340b4130 con 0x7f1a3406d970 2024-03-28T14:08:41.642 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:41.638+0000 7f1a3886f640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f1a340b3fb0 con 0x7f1a340b5460 2024-03-28T14:08:41.642 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:41.638+0000 7f1a3886f640 1 -- --> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] -- mon_getmap magic: 0 -- 0x7f1a340b42b0 con 0x7f1a340732a0 2024-03-28T14:08:41.642 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:41.638+0000 7f1a3259c640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1a3406d970 0x7f1a34137fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:41.642 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:41.638+0000 7f1a3259c640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1a3406d970 0x7f1a34137fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:50342/0 (socket says 172.21.15.104:50342) 2024-03-28T14:08:41.642 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:41.638+0000 7f1a3259c640 1 -- 172.21.15.104:0/104772110 learned_addr learned my addr 172.21.15.104:0/104772110 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-03-28T14:08:41.642 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:41.638+0000 7f1a32d9d640 1 --2- 172.21.15.104:0/104772110 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f1a340b5460 0x7f1a341e3840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:41.642 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:41.638+0000 7f1a31d9b640 1 --2- 172.21.15.104:0/104772110 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f1a340732a0 0x7f1a341384f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:41.642 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:41.638+0000 7f1a3259c640 1 -- 172.21.15.104:0/104772110 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f1a340732a0 msgr2=0x7f1a341384f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:41.642 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:41.638+0000 7f1a3259c640 1 --2- 172.21.15.104:0/104772110 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f1a340732a0 0x7f1a341384f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:41.642 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:41.638+0000 7f1a3259c640 1 -- 172.21.15.104:0/104772110 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f1a340b5460 msgr2=0x7f1a341e3840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:41.642 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:41.638+0000 7f1a3259c640 1 --2- 172.21.15.104:0/104772110 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f1a340b5460 0x7f1a341e3840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:41.643 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:41.638+0000 7f1a3259c640 1 -- 172.21.15.104:0/104772110 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1a3409baf0 con 0x7f1a3406d970 2024-03-28T14:08:41.643 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:41.638+0000 7f1a32d9d640 1 --2- 172.21.15.104:0/104772110 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f1a340b5460 0x7f1a341e3840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T14:08:41.643 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:41.638+0000 7f1a3259c640 1 --2- 172.21.15.104:0/104772110 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1a3406d970 0x7f1a34137fb0 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f1a2800d800 tx=0x7f1a2800dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:08:41.643 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:41.638+0000 7f1a22ffd640 1 -- 172.21.15.104:0/104772110 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a2800b6c0 con 0x7f1a3406d970 2024-03-28T14:08:41.643 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:41.638+0000 7f1a22ffd640 1 -- 172.21.15.104:0/104772110 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1a28019070 con 0x7f1a3406d970 2024-03-28T14:08:41.643 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:41.638+0000 7f1a3886f640 1 -- 172.21.15.104:0/104772110 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1a34138a30 con 0x7f1a3406d970 2024-03-28T14:08:41.643 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:41.638+0000 7f1a22ffd640 1 -- 172.21.15.104:0/104772110 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a2800bae0 con 0x7f1a3406d970 2024-03-28T14:08:41.643 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:41.638+0000 7f1a3886f640 1 -- 172.21.15.104:0/104772110 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1a34072100 con 0x7f1a3406d970 2024-03-28T14:08:41.643 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:41.638+0000 7f1a3886f640 1 -- 172.21.15.104:0/104772110 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1a340687c0 con 0x7f1a3406d970 2024-03-28T14:08:41.644 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:41.638+0000 7f1a22ffd640 1 -- 172.21.15.104:0/104772110 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) ==== 87803+0+0 (secure 0 0 0) 0x7f1a2800bc80 con 0x7f1a3406d970 2024-03-28T14:08:41.644 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:41.638+0000 7f1a22ffd640 1 --2- 172.21.15.104:0/104772110 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f1a14067fc0 0x7f1a1406a480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:41.644 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:41.638+0000 7f1a22ffd640 1 -- 172.21.15.104:0/104772110 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(29..29 src has 1..29) ==== 5709+0+0 (secure 0 0 0) 0x7f1a28086990 con 0x7f1a3406d970 2024-03-28T14:08:41.644 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:41.638+0000 7f1a31d9b640 1 --2- 172.21.15.104:0/104772110 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f1a14067fc0 0x7f1a1406a480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:41.644 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:41.642+0000 7f1a31d9b640 1 --2- 172.21.15.104:0/104772110 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f1a14067fc0 0x7f1a1406a480 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f1a240af000 tx=0x7f1a24008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:08:41.647 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:41.642+0000 7f1a22ffd640 1 -- 172.21.15.104:0/104772110 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179117 (secure 0 0 0) 0x7f1a280571d0 con 0x7f1a3406d970 2024-03-28T14:08:41.810 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:41.806+0000 7f1a3886f640 1 -- 172.21.15.104:0/104772110 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f1a3413d1c0 con 0x7f1a3406d970 2024-03-28T14:08:41.812 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:41.810+0000 7f1a22ffd640 1 -- 172.21.15.104:0/104772110 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v29) ==== 74+0+16706 (secure 0 0 0) 0x7f1a28056ad0 con 0x7f1a3406d970 2024-03-28T14:08:41.812 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-03-28T14:08:41.813 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":29,"fsid":"428a90f2-3351-435f-8fcf-f7e66c78a13e","created":"2024-03-28T14:07:34.862458+0000","modified":"2024-03-28T14:08:40.853379+0000","last_up_change":"2024-03-28T14:07:46.028122+0000","last_in_change":"2024-03-28T14:07:37.157486+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-28T14:07:47.622988+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_metadata","create_time":"2024-03-28T14:08:38.687068+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data","create_time":"2024-03-28T14:08:39.805358+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"1cb70540-4e42-402f-a446-aaec4ca83007","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6800","nonce":3745690918},{"type":"v1","addr":"172.21.15.104:6801","nonce":3745690918}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":3745690918},{"type":"v1","addr":"172.21.15.104:6803","nonce":3745690918}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":3745690918},{"type":"v1","addr":"172.21.15.104:6807","nonce":3745690918}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":3745690918},{"type":"v1","addr":"172.21.15.104:6805","nonce":3745690918}]},"public_addr":"172.21.15.104:6801/3745690918","cluster_addr":"172.21.15.104:6803/3745690918","heartbeat_back_addr":"172.21.15.104:6807/3745690918","heartbeat_front_addr":"172.21.15.104:6805/3745690918","state":["exists","up"]},{"osd":1,"uuid":"3ab7afdd-b980-418c-81f2-004e16de3dca","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":19438674},{"type":"v1","addr":"172.21.15.104:6810","nonce":19438674}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":19438674},{"type":"v1","addr":"172.21.15.104:6814","nonce":19438674}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":19438674},{"type":"v1","addr":"172.21.15.104:6823","nonce":19438674}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":19438674},{"type":"v1","addr":"172.21.15.104:6819","nonce":19438674}]},"public_addr":"172.21.15.104:6810/19438674","cluster_addr":"172.21.15.104:6814/19438674","heartbeat_back_addr":"172.21.15.104:6823/19438674","heartbeat_front_addr":"172.21.15.104:6819/19438674","state":["exists","up"]},{"osd":2,"uuid":"3e2651c8-ecae-4a9e-a053-27240a79eac2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":2352012626},{"type":"v1","addr":"172.21.15.104:6825","nonce":2352012626}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6826","nonce":2352012626},{"type":"v1","addr":"172.21.15.104:6827","nonce":2352012626}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":2352012626},{"type":"v1","addr":"172.21.15.104:6831","nonce":2352012626}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6828","nonce":2352012626},{"type":"v1","addr":"172.21.15.104:6829","nonce":2352012626}]},"public_addr":"172.21.15.104:6825/2352012626","cluster_addr":"172.21.15.104:6827/2352012626","heartbeat_back_addr":"172.21.15.104:6831/2352012626","heartbeat_front_addr":"172.21.15.104:6829/2352012626","state":["exists","up"]},{"osd":3,"uuid":"925c7e0a-38ef-47cd-b288-8bec8456fb17","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6809","nonce":3568347381},{"type":"v1","addr":"172.21.15.104:6811","nonce":3568347381}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6813","nonce":3568347381},{"type":"v1","addr":"172.21.15.104:6815","nonce":3568347381}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":3568347381},{"type":"v1","addr":"172.21.15.104:6822","nonce":3568347381}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6817","nonce":3568347381},{"type":"v1","addr":"172.21.15.104:6818","nonce":3568347381}]},"public_addr":"172.21.15.104:6811/3568347381","cluster_addr":"172.21.15.104:6815/3568347381","heartbeat_back_addr":"172.21.15.104:6822/3568347381","heartbeat_front_addr":"172.21.15.104:6818/3568347381","state":["exists","up"]},{"osd":4,"uuid":"4a8fac86-e595-4a0e-96e3-57edb3cc3f57","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6800","nonce":2107001077},{"type":"v1","addr":"172.21.15.171:6801","nonce":2107001077}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6802","nonce":2107001077},{"type":"v1","addr":"172.21.15.171:6803","nonce":2107001077}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6806","nonce":2107001077},{"type":"v1","addr":"172.21.15.171:6807","nonce":2107001077}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6804","nonce":2107001077},{"type":"v1","addr":"172.21.15.171:6805","nonce":2107001077}]},"public_addr":"172.21.15.171:6801/2107001077","cluster_addr":"172.21.15.171:6803/2107001077","heartbeat_back_addr":"172.21.15.171:6807/2107001077","heartbeat_front_addr":"172.21.15.171:6805/2107001077","state":["exists","up"]},{"osd":5,"uuid":"e48d50b0-beae-40ae-b15a-9a84dbfe88ab","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6808","nonce":2934826925},{"type":"v1","addr":"172.21.15.171:6810","nonce":2934826925}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6812","nonce":2934826925},{"type":"v1","addr":"172.21.15.171:6814","nonce":2934826925}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6819","nonce":2934826925},{"type":"v1","addr":"172.21.15.171:6821","nonce":2934826925}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6816","nonce":2934826925},{"type":"v1","addr":"172.21.15.171:6817","nonce":2934826925}]},"public_addr":"172.21.15.171:6810/2934826925","cluster_addr":"172.21.15.171:6814/2934826925","heartbeat_back_addr":"172.21.15.171:6821/2934826925","heartbeat_front_addr":"172.21.15.171:6817/2934826925","state":["exists","up"]},{"osd":6,"uuid":"5e7c3292-9936-4f35-b267-aed9d4e49323","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6824","nonce":436742919},{"type":"v1","addr":"172.21.15.171:6825","nonce":436742919}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6826","nonce":436742919},{"type":"v1","addr":"172.21.15.171:6827","nonce":436742919}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6830","nonce":436742919},{"type":"v1","addr":"172.21.15.171:6831","nonce":436742919}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6828","nonce":436742919},{"type":"v1","addr":"172.21.15.171:6829","nonce":436742919}]},"public_addr":"172.21.15.171:6825/436742919","cluster_addr":"172.21.15.171:6827/436742919","heartbeat_back_addr":"172.21.15.171:6831/436742919","heartbeat_front_addr":"172.21.15.171:6829/436742919","state":["exists","up"]},{"osd":7,"uuid":"339b8414-d7f0-4fb0-a446-efaf635cb7ff","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6809","nonce":2055631175},{"type":"v1","addr":"172.21.15.171:6811","nonce":2055631175}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6813","nonce":2055631175},{"type":"v1","addr":"172.21.15.171:6815","nonce":2055631175}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6822","nonce":2055631175},{"type":"v1","addr":"172.21.15.171:6823","nonce":2055631175}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6818","nonce":2055631175},{"type":"v1","addr":"172.21.15.171:6820","nonce":2055631175}]},"public_addr":"172.21.15.171:6811/2055631175","cluster_addr":"172.21.15.171:6815/2055631175","heartbeat_back_addr":"172.21.15.171:6823/2055631175","heartbeat_front_addr":"172.21.15.171:6820/2055631175","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T14:07:42.668881+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T14:07:42.708397+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T14:07:42.634191+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T14:07:42.733662+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T14:07:42.651541+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T14:07:42.783181+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T14:07:42.766310+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T14:07:42.670797+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-28T14:08:41.815 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:41.810+0000 7f1a3886f640 1 -- 172.21.15.104:0/104772110 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f1a14067fc0 msgr2=0x7f1a1406a480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:41.815 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:41.810+0000 7f1a3886f640 1 --2- 172.21.15.104:0/104772110 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f1a14067fc0 0x7f1a1406a480 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f1a240af000 tx=0x7f1a24008040 comp rx=0 tx=0).stop 2024-03-28T14:08:41.815 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:41.810+0000 7f1a3886f640 1 -- 172.21.15.104:0/104772110 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1a3406d970 msgr2=0x7f1a34137fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:41.815 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:41.810+0000 7f1a3886f640 1 --2- 172.21.15.104:0/104772110 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1a3406d970 0x7f1a34137fb0 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f1a2800d800 tx=0x7f1a2800dcc0 comp rx=0 tx=0).stop 2024-03-28T14:08:41.815 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:41.810+0000 7f1a3886f640 1 -- 172.21.15.104:0/104772110 shutdown_connections 2024-03-28T14:08:41.815 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:41.810+0000 7f1a3886f640 1 --2- 172.21.15.104:0/104772110 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f1a340b5460 0x7f1a341e3840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:41.815 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:41.810+0000 7f1a3886f640 1 --2- 172.21.15.104:0/104772110 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f1a340732a0 0x7f1a341384f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:41.815 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:41.810+0000 7f1a3886f640 1 --2- 172.21.15.104:0/104772110 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f1a14067fc0 0x7f1a1406a480 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:41.815 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:41.810+0000 7f1a3886f640 1 --2- 172.21.15.104:0/104772110 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1a3406d970 0x7f1a34137fb0 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:41.815 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:41.810+0000 7f1a3886f640 1 -- 172.21.15.104:0/104772110 >> 172.21.15.104:0/104772110 conn(0x7f1a340b58a0 msgr2=0x7f1a34144340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T14:08:41.816 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:41.810+0000 7f1a3886f640 1 -- 172.21.15.104:0/104772110 shutdown_connections 2024-03-28T14:08:41.816 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:41.810+0000 7f1a3886f640 1 -- 172.21.15.104:0/104772110 wait complete. 2024-03-28T14:08:41.832 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-28T14:08:41.949 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:41.946+0000 7f1ee1d6a640 1 Processor -- start 2024-03-28T14:08:41.949 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:41.946+0000 7f1ee1d6a640 1 -- start start 2024-03-28T14:08:41.950 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:41.946+0000 7f1ee1d6a640 1 --2- >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f1edc06e570 0x7f1edc06f0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:41.950 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:41.946+0000 7f1ee1d6a640 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f1edc06e950 0x7f1edc06b5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:41.950 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:41.946+0000 7f1ee1d6a640 1 --2- >> v2:172.21.15.171:3300/0 conn(0x7f1edc14af80 0x7f1edc14b410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:41.950 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:41.946+0000 7f1ee1d6a640 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 -- 0x7f1edc073410 con 0x7f1edc06e950 2024-03-28T14:08:41.950 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:41.946+0000 7f1ee1d6a640 1 -- --> v2:172.21.15.171:3300/0 -- mon_getmap magic: 0 -- 0x7f1edc0b9c20 con 0x7f1edc14af80 2024-03-28T14:08:41.951 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:41.946+0000 7f1ee1d6a640 1 -- --> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] -- mon_getmap magic: 0 -- 0x7f1edc06d940 con 0x7f1edc06e570 2024-03-28T14:08:41.951 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:41.946+0000 7f1edaffd640 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f1edc06e950 0x7f1edc06b5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:41.951 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:41.946+0000 7f1edaffd640 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f1edc06e950 0x7f1edc06b5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:50352/0 (socket says 172.21.15.104:50352) 2024-03-28T14:08:41.951 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:41.946+0000 7f1edaffd640 1 -- 172.21.15.104:0/1629283269 learned_addr learned my addr 172.21.15.104:0/1629283269 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-03-28T14:08:41.951 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:41.946+0000 7f1edb7fe640 1 --2- 172.21.15.104:0/1629283269 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f1edc06e570 0x7f1edc06f0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:41.951 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:41.946+0000 7f1edbfff640 1 --2- 172.21.15.104:0/1629283269 >> v2:172.21.15.171:3300/0 conn(0x7f1edc14af80 0x7f1edc14b410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:41.951 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:41.946+0000 7f1edaffd640 1 -- 172.21.15.104:0/1629283269 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f1edc06e570 msgr2=0x7f1edc06f0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T14:08:41.951 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:41.946+0000 7f1edaffd640 1 --2- 172.21.15.104:0/1629283269 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f1edc06e570 0x7f1edc06f0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:41.951 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:41.946+0000 7f1edaffd640 1 -- 172.21.15.104:0/1629283269 >> v2:172.21.15.171:3300/0 conn(0x7f1edc14af80 msgr2=0x7f1edc14b410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T14:08:41.951 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:41.946+0000 7f1edaffd640 1 --2- 172.21.15.104:0/1629283269 >> v2:172.21.15.171:3300/0 conn(0x7f1edc14af80 0x7f1edc14b410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:41.951 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:41.946+0000 7f1edaffd640 1 -- 172.21.15.104:0/1629283269 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1edc0afdc0 con 0x7f1edc06e950 2024-03-28T14:08:41.952 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:41.950+0000 7f1edaffd640 1 --2- 172.21.15.104:0/1629283269 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1edc06e950 0x7f1edc06b5d0 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f1edc0b9b90 tx=0x7f1ed001c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ee5df769fcc88948 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:08:41.952 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:41.950+0000 7f1eda7fc640 1 -- 172.21.15.104:0/1629283269 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ed00b0070 con 0x7f1edc06e950 2024-03-28T14:08:41.952 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:41.950+0000 7f1eda7fc640 1 -- 172.21.15.104:0/1629283269 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1ed00ab600 con 0x7f1edc06e950 2024-03-28T14:08:41.952 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:41.950+0000 7f1eda7fc640 1 -- 172.21.15.104:0/1629283269 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ed00ab9d0 con 0x7f1edc06e950 2024-03-28T14:08:41.953 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:41.950+0000 7f1ee1d6a640 1 -- 172.21.15.104:0/1629283269 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1edc06e950 msgr2=0x7f1edc06b5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:41.953 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:41.950+0000 7f1ee1d6a640 1 --2- 172.21.15.104:0/1629283269 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1edc06e950 0x7f1edc06b5d0 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f1edc0b9b90 tx=0x7f1ed001c490 comp rx=0 tx=0).stop 2024-03-28T14:08:41.953 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:41.950+0000 7f1ee1d6a640 1 -- 172.21.15.104:0/1629283269 shutdown_connections 2024-03-28T14:08:41.953 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:41.950+0000 7f1ee1d6a640 1 --2- 172.21.15.104:0/1629283269 >> v2:172.21.15.171:3300/0 conn(0x7f1edc14af80 0x7f1edc14b410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:41.954 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:41.950+0000 7f1ee1d6a640 1 --2- 172.21.15.104:0/1629283269 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1edc06e950 0x7f1edc06b5d0 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:41.954 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:41.950+0000 7f1ee1d6a640 1 --2- 172.21.15.104:0/1629283269 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f1edc06e570 0x7f1edc06f0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:41.954 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:41.950+0000 7f1ee1d6a640 1 -- 172.21.15.104:0/1629283269 >> 172.21.15.104:0/1629283269 conn(0x7f1edc0b7d90 msgr2=0x7f1edc0b81a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T14:08:41.954 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:41.950+0000 7f1ee1d6a640 1 -- 172.21.15.104:0/1629283269 shutdown_connections 2024-03-28T14:08:41.954 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:41.950+0000 7f1ee1d6a640 1 -- 172.21.15.104:0/1629283269 wait complete. 2024-03-28T14:08:41.956 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:41.950+0000 7f1ee1d6a640 1 Processor -- start 2024-03-28T14:08:41.957 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:41.950+0000 7f1ee1d6a640 1 -- start start 2024-03-28T14:08:41.957 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:41.950+0000 7f1ee1d6a640 1 --2- >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f1edc06e570 0x7f1edc0721f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:41.957 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:41.950+0000 7f1ee1d6a640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f1edc14af80 0x7f1edc0b3040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:41.957 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:41.950+0000 7f1ee1d6a640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1edc14d5a0 0x7f1edc0b3580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:41.957 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:41.950+0000 7f1ee1d6a640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f1edc06bc90 con 0x7f1edc14d5a0 2024-03-28T14:08:41.957 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:41.950+0000 7f1ee1d6a640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f1edc06bb10 con 0x7f1edc14af80 2024-03-28T14:08:41.957 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:41.950+0000 7f1ee1d6a640 1 -- --> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] -- mon_getmap magic: 0 -- 0x7f1edc1484e0 con 0x7f1edc06e570 2024-03-28T14:08:41.957 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:41.950+0000 7f1edbfff640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1edc14d5a0 0x7f1edc0b3580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:41.957 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:41.950+0000 7f1edbfff640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1edc14d5a0 0x7f1edc0b3580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:50360/0 (socket says 172.21.15.104:50360) 2024-03-28T14:08:41.957 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:41.950+0000 7f1edbfff640 1 -- 172.21.15.104:0/1854714570 learned_addr learned my addr 172.21.15.104:0/1854714570 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-03-28T14:08:41.957 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:41.950+0000 7f1edb7fe640 1 --2- 172.21.15.104:0/1854714570 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f1edc06e570 0x7f1edc0721f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:41.957 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:41.950+0000 7f1edaffd640 1 --2- 172.21.15.104:0/1854714570 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f1edc14af80 0x7f1edc0b3040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:41.958 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:41.950+0000 7f1edbfff640 1 -- 172.21.15.104:0/1854714570 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f1edc06e570 msgr2=0x7f1edc0721f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:41.958 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:41.950+0000 7f1edbfff640 1 --2- 172.21.15.104:0/1854714570 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f1edc06e570 0x7f1edc0721f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:41.958 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:41.950+0000 7f1edbfff640 1 -- 172.21.15.104:0/1854714570 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f1edc14af80 msgr2=0x7f1edc0b3040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:41.958 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:41.950+0000 7f1edbfff640 1 --2- 172.21.15.104:0/1854714570 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f1edc14af80 0x7f1edc0b3040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:41.958 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:41.950+0000 7f1edbfff640 1 -- 172.21.15.104:0/1854714570 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1edc14da80 con 0x7f1edc14d5a0 2024-03-28T14:08:41.958 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:41.950+0000 7f1edb7fe640 1 --2- 172.21.15.104:0/1854714570 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f1edc06e570 0x7f1edc0721f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T14:08:41.958 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:41.950+0000 7f1edaffd640 1 --2- 172.21.15.104:0/1854714570 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f1edc14af80 0x7f1edc0b3040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T14:08:41.958 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:41.950+0000 7f1edbfff640 1 --2- 172.21.15.104:0/1854714570 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1edc14d5a0 0x7f1edc0b3580 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f1ed4007e60 tx=0x7f1ed400e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:08:41.958 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:41.950+0000 7f1ec3fff640 1 -- 172.21.15.104:0/1854714570 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ed4014420 con 0x7f1edc14d5a0 2024-03-28T14:08:41.958 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:41.950+0000 7f1ec3fff640 1 -- 172.21.15.104:0/1854714570 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1ed4019070 con 0x7f1edc14d5a0 2024-03-28T14:08:41.958 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:41.950+0000 7f1ee1d6a640 1 -- 172.21.15.104:0/1854714570 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1edc14dd60 con 0x7f1edc14d5a0 2024-03-28T14:08:41.958 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:41.950+0000 7f1ec3fff640 1 -- 172.21.15.104:0/1854714570 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ed4014820 con 0x7f1edc14d5a0 2024-03-28T14:08:41.958 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:41.950+0000 7f1ee1d6a640 1 -- 172.21.15.104:0/1854714570 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1edc14c850 con 0x7f1edc14d5a0 2024-03-28T14:08:41.958 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:41.954+0000 7f1ee1d6a640 1 -- 172.21.15.104:0/1854714570 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1edc14c680 con 0x7f1edc14d5a0 2024-03-28T14:08:41.959 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:41.954+0000 7f1ec3fff640 1 -- 172.21.15.104:0/1854714570 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) ==== 87803+0+0 (secure 0 0 0) 0x7f1ed40149c0 con 0x7f1edc14d5a0 2024-03-28T14:08:41.959 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:41.954+0000 7f1ec3fff640 1 --2- 172.21.15.104:0/1854714570 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f1ec8067fc0 0x7f1ec806a480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:41.959 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:41.954+0000 7f1ec3fff640 1 -- 172.21.15.104:0/1854714570 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(29..29 src has 1..29) ==== 5709+0+0 (secure 0 0 0) 0x7f1ed4086c50 con 0x7f1edc14d5a0 2024-03-28T14:08:41.959 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:41.954+0000 7f1edb7fe640 1 --2- 172.21.15.104:0/1854714570 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f1ec8067fc0 0x7f1ec806a480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:41.959 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:41.954+0000 7f1edb7fe640 1 --2- 172.21.15.104:0/1854714570 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f1ec8067fc0 0x7f1ec806a480 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f1ecc006fd0 tx=0x7f1ecc008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:08:41.961 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:41.958+0000 7f1ec3fff640 1 -- 172.21.15.104:0/1854714570 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179117 (secure 0 0 0) 0x7f1ed408c020 con 0x7f1edc14d5a0 2024-03-28T14:08:42.126 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:42.122+0000 7f1ee1d6a640 1 -- 172.21.15.104:0/1854714570 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f1edc0687c0 con 0x7f1edc14d5a0 2024-03-28T14:08:42.129 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:42.126+0000 7f1ec3fff640 1 -- 172.21.15.104:0/1854714570 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v29) ==== 74+0+16706 (secure 0 0 0) 0x7f1edc14da80 con 0x7f1edc14d5a0 2024-03-28T14:08:42.129 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-03-28T14:08:42.130 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":29,"fsid":"428a90f2-3351-435f-8fcf-f7e66c78a13e","created":"2024-03-28T14:07:34.862458+0000","modified":"2024-03-28T14:08:40.853379+0000","last_up_change":"2024-03-28T14:07:46.028122+0000","last_in_change":"2024-03-28T14:07:37.157486+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-28T14:07:47.622988+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_metadata","create_time":"2024-03-28T14:08:38.687068+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data","create_time":"2024-03-28T14:08:39.805358+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"1cb70540-4e42-402f-a446-aaec4ca83007","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6800","nonce":3745690918},{"type":"v1","addr":"172.21.15.104:6801","nonce":3745690918}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":3745690918},{"type":"v1","addr":"172.21.15.104:6803","nonce":3745690918}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":3745690918},{"type":"v1","addr":"172.21.15.104:6807","nonce":3745690918}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":3745690918},{"type":"v1","addr":"172.21.15.104:6805","nonce":3745690918}]},"public_addr":"172.21.15.104:6801/3745690918","cluster_addr":"172.21.15.104:6803/3745690918","heartbeat_back_addr":"172.21.15.104:6807/3745690918","heartbeat_front_addr":"172.21.15.104:6805/3745690918","state":["exists","up"]},{"osd":1,"uuid":"3ab7afdd-b980-418c-81f2-004e16de3dca","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":19438674},{"type":"v1","addr":"172.21.15.104:6810","nonce":19438674}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":19438674},{"type":"v1","addr":"172.21.15.104:6814","nonce":19438674}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":19438674},{"type":"v1","addr":"172.21.15.104:6823","nonce":19438674}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":19438674},{"type":"v1","addr":"172.21.15.104:6819","nonce":19438674}]},"public_addr":"172.21.15.104:6810/19438674","cluster_addr":"172.21.15.104:6814/19438674","heartbeat_back_addr":"172.21.15.104:6823/19438674","heartbeat_front_addr":"172.21.15.104:6819/19438674","state":["exists","up"]},{"osd":2,"uuid":"3e2651c8-ecae-4a9e-a053-27240a79eac2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":2352012626},{"type":"v1","addr":"172.21.15.104:6825","nonce":2352012626}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6826","nonce":2352012626},{"type":"v1","addr":"172.21.15.104:6827","nonce":2352012626}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":2352012626},{"type":"v1","addr":"172.21.15.104:6831","nonce":2352012626}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6828","nonce":2352012626},{"type":"v1","addr":"172.21.15.104:6829","nonce":2352012626}]},"public_addr":"172.21.15.104:6825/2352012626","cluster_addr":"172.21.15.104:6827/2352012626","heartbeat_back_addr":"172.21.15.104:6831/2352012626","heartbeat_front_addr":"172.21.15.104:6829/2352012626","state":["exists","up"]},{"osd":3,"uuid":"925c7e0a-38ef-47cd-b288-8bec8456fb17","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6809","nonce":3568347381},{"type":"v1","addr":"172.21.15.104:6811","nonce":3568347381}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6813","nonce":3568347381},{"type":"v1","addr":"172.21.15.104:6815","nonce":3568347381}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":3568347381},{"type":"v1","addr":"172.21.15.104:6822","nonce":3568347381}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6817","nonce":3568347381},{"type":"v1","addr":"172.21.15.104:6818","nonce":3568347381}]},"public_addr":"172.21.15.104:6811/3568347381","cluster_addr":"172.21.15.104:6815/3568347381","heartbeat_back_addr":"172.21.15.104:6822/3568347381","heartbeat_front_addr":"172.21.15.104:6818/3568347381","state":["exists","up"]},{"osd":4,"uuid":"4a8fac86-e595-4a0e-96e3-57edb3cc3f57","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6800","nonce":2107001077},{"type":"v1","addr":"172.21.15.171:6801","nonce":2107001077}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6802","nonce":2107001077},{"type":"v1","addr":"172.21.15.171:6803","nonce":2107001077}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6806","nonce":2107001077},{"type":"v1","addr":"172.21.15.171:6807","nonce":2107001077}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6804","nonce":2107001077},{"type":"v1","addr":"172.21.15.171:6805","nonce":2107001077}]},"public_addr":"172.21.15.171:6801/2107001077","cluster_addr":"172.21.15.171:6803/2107001077","heartbeat_back_addr":"172.21.15.171:6807/2107001077","heartbeat_front_addr":"172.21.15.171:6805/2107001077","state":["exists","up"]},{"osd":5,"uuid":"e48d50b0-beae-40ae-b15a-9a84dbfe88ab","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6808","nonce":2934826925},{"type":"v1","addr":"172.21.15.171:6810","nonce":2934826925}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6812","nonce":2934826925},{"type":"v1","addr":"172.21.15.171:6814","nonce":2934826925}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6819","nonce":2934826925},{"type":"v1","addr":"172.21.15.171:6821","nonce":2934826925}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6816","nonce":2934826925},{"type":"v1","addr":"172.21.15.171:6817","nonce":2934826925}]},"public_addr":"172.21.15.171:6810/2934826925","cluster_addr":"172.21.15.171:6814/2934826925","heartbeat_back_addr":"172.21.15.171:6821/2934826925","heartbeat_front_addr":"172.21.15.171:6817/2934826925","state":["exists","up"]},{"osd":6,"uuid":"5e7c3292-9936-4f35-b267-aed9d4e49323","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6824","nonce":436742919},{"type":"v1","addr":"172.21.15.171:6825","nonce":436742919}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6826","nonce":436742919},{"type":"v1","addr":"172.21.15.171:6827","nonce":436742919}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6830","nonce":436742919},{"type":"v1","addr":"172.21.15.171:6831","nonce":436742919}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6828","nonce":436742919},{"type":"v1","addr":"172.21.15.171:6829","nonce":436742919}]},"public_addr":"172.21.15.171:6825/436742919","cluster_addr":"172.21.15.171:6827/436742919","heartbeat_back_addr":"172.21.15.171:6831/436742919","heartbeat_front_addr":"172.21.15.171:6829/436742919","state":["exists","up"]},{"osd":7,"uuid":"339b8414-d7f0-4fb0-a446-efaf635cb7ff","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6809","nonce":2055631175},{"type":"v1","addr":"172.21.15.171:6811","nonce":2055631175}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6813","nonce":2055631175},{"type":"v1","addr":"172.21.15.171:6815","nonce":2055631175}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6822","nonce":2055631175},{"type":"v1","addr":"172.21.15.171:6823","nonce":2055631175}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6818","nonce":2055631175},{"type":"v1","addr":"172.21.15.171:6820","nonce":2055631175}]},"public_addr":"172.21.15.171:6811/2055631175","cluster_addr":"172.21.15.171:6815/2055631175","heartbeat_back_addr":"172.21.15.171:6823/2055631175","heartbeat_front_addr":"172.21.15.171:6820/2055631175","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T14:07:42.668881+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T14:07:42.708397+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T14:07:42.634191+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T14:07:42.733662+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T14:07:42.651541+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T14:07:42.783181+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T14:07:42.766310+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T14:07:42.670797+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-28T14:08:42.132 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:42.130+0000 7f1ee1d6a640 1 -- 172.21.15.104:0/1854714570 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f1ec8067fc0 msgr2=0x7f1ec806a480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:42.132 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:42.130+0000 7f1ee1d6a640 1 --2- 172.21.15.104:0/1854714570 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f1ec8067fc0 0x7f1ec806a480 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f1ecc006fd0 tx=0x7f1ecc008040 comp rx=0 tx=0).stop 2024-03-28T14:08:42.132 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:42.130+0000 7f1ee1d6a640 1 -- 172.21.15.104:0/1854714570 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1edc14d5a0 msgr2=0x7f1edc0b3580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:42.132 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:42.130+0000 7f1ee1d6a640 1 --2- 172.21.15.104:0/1854714570 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1edc14d5a0 0x7f1edc0b3580 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f1ed4007e60 tx=0x7f1ed400e4b0 comp rx=0 tx=0).stop 2024-03-28T14:08:42.132 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:42.130+0000 7f1ee1d6a640 1 -- 172.21.15.104:0/1854714570 shutdown_connections 2024-03-28T14:08:42.132 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:42.130+0000 7f1ee1d6a640 1 --2- 172.21.15.104:0/1854714570 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f1ec8067fc0 0x7f1ec806a480 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:42.133 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:42.130+0000 7f1ee1d6a640 1 --2- 172.21.15.104:0/1854714570 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1edc14d5a0 0x7f1edc0b3580 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:42.133 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:42.130+0000 7f1ee1d6a640 1 --2- 172.21.15.104:0/1854714570 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f1edc14af80 0x7f1edc0b3040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:42.133 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:42.130+0000 7f1ee1d6a640 1 --2- 172.21.15.104:0/1854714570 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f1edc06e570 0x7f1edc0721f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:42.133 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:42.130+0000 7f1ee1d6a640 1 -- 172.21.15.104:0/1854714570 >> 172.21.15.104:0/1854714570 conn(0x7f1edc0b7d90 msgr2=0x7f1edc09f570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T14:08:42.133 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:42.130+0000 7f1ee1d6a640 1 -- 172.21.15.104:0/1854714570 shutdown_connections 2024-03-28T14:08:42.133 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:42.130+0000 7f1ee1d6a640 1 -- 172.21.15.104:0/1854714570 wait complete. 2024-03-28T14:08:42.149 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-03-28T14:08:42.268 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:42.266+0000 7fdbe8df3640 1 Processor -- start 2024-03-28T14:08:42.269 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:42.266+0000 7fdbe8df3640 1 -- start start 2024-03-28T14:08:42.273 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:42.266+0000 7fdbe8df3640 1 --2- >> v2:172.21.15.171:3300/0 conn(0x7fdbe40bd9e0 0x7fdbe414af30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:42.273 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:42.266+0000 7fdbe8df3640 1 --2- >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fdbe414b470 0x7fdbe41413f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:42.273 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:42.266+0000 7fdbe8df3640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdbe40b5050 con 0x7fdbe4141930 2024-03-28T14:08:42.273 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:42.266+0000 7fdbe8df3640 1 -- --> v2:172.21.15.171:3300/0 -- mon_getmap magic: 0 -- 0x7fdbe40b70b0 con 0x7fdbe40bd9e0 2024-03-28T14:08:42.273 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:42.266+0000 7fdbe8df3640 1 -- --> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] -- mon_getmap magic: 0 -- 0x7fdbe414b8d0 con 0x7fdbe414b470 2024-03-28T14:08:42.273 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:42.266+0000 7fdbe2d9d640 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7fdbe4141930 0x7fdbe40b79f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:59834/0 (socket says 172.21.15.104:59834) 2024-03-28T14:08:42.273 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:42.266+0000 7fdbe2d9d640 1 -- 172.21.15.104:0/2324803235 learned_addr learned my addr 172.21.15.104:0/2324803235 (peer_addr_for_me v1:172.21.15.104:0/0) 2024-03-28T14:08:42.273 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:42.266+0000 7fdbe1d9b640 1 --2- 172.21.15.104:0/2324803235 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fdbe414b470 0x7fdbe41413f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:42.273 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:42.266+0000 7fdbe259c640 1 --2- 172.21.15.104:0/2324803235 >> v2:172.21.15.171:3300/0 conn(0x7fdbe40bd9e0 0x7fdbe414af30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:42.273 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:42.266+0000 7fdbe159a640 1 -- 172.21.15.104:0/2324803235 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3826737423 0 0) 0x7fdbe40b5050 con 0x7fdbe4141930 2024-03-28T14:08:42.274 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:42.266+0000 7fdbe159a640 1 -- 172.21.15.104:0/2324803235 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fdbcc003530 con 0x7fdbe4141930 2024-03-28T14:08:42.274 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:42.266+0000 7fdbe159a640 1 -- 172.21.15.104:0/2324803235 <== mon.0 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2818324030 0 0) 0x7fdbcc003530 con 0x7fdbe4141930 2024-03-28T14:08:42.274 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:42.266+0000 7fdbe159a640 1 -- 172.21.15.104:0/2324803235 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fdbe414b470 msgr2=0x7fdbe41413f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T14:08:42.274 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:42.266+0000 7fdbe159a640 1 --2- 172.21.15.104:0/2324803235 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fdbe414b470 0x7fdbe41413f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:42.274 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:42.266+0000 7fdbe159a640 1 -- 172.21.15.104:0/2324803235 >> v2:172.21.15.171:3300/0 conn(0x7fdbe40bd9e0 msgr2=0x7fdbe414af30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T14:08:42.274 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:42.266+0000 7fdbe159a640 1 --2- 172.21.15.104:0/2324803235 >> v2:172.21.15.171:3300/0 conn(0x7fdbe40bd9e0 0x7fdbe414af30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:42.274 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:42.266+0000 7fdbe159a640 1 -- 172.21.15.104:0/2324803235 --> v1:172.21.15.104:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdbe40bddc0 con 0x7fdbe4141930 2024-03-28T14:08:42.274 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:42.266+0000 7fdbe159a640 1 -- 172.21.15.104:0/2324803235 <== mon.0 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3745580849 0 0) 0x7fdbdc001a00 con 0x7fdbe4141930 2024-03-28T14:08:42.274 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:42.266+0000 7fdbe159a640 1 -- 172.21.15.104:0/2324803235 >> v1:172.21.15.104:6789/0 conn(0x7fdbe4141930 legacy=0x7fdbe40b79f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:42.274 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:42.266+0000 7fdbe159a640 1 --2- 172.21.15.104:0/2324803235 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fdbcc0039c0 0x7fdbcc003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:42.274 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:42.266+0000 7fdbe159a640 1 --2- 172.21.15.104:0/2324803235 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fdbcc004810 0x7fdbcc004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:42.274 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:42.266+0000 7fdbe159a640 1 --2- 172.21.15.104:0/2324803235 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fdbe414b470 0x7fdbcc0051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:42.275 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:42.266+0000 7fdbe159a640 1 -- 172.21.15.104:0/2324803235 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7fdbe40b70b0 con 0x7fdbcc004810 2024-03-28T14:08:42.275 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:42.266+0000 7fdbe159a640 1 -- 172.21.15.104:0/2324803235 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7fdbe414b8d0 con 0x7fdbcc0039c0 2024-03-28T14:08:42.275 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:42.266+0000 7fdbe159a640 1 -- 172.21.15.104:0/2324803235 --> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] -- mon_getmap magic: 0 -- 0x7fdbcc00bab0 con 0x7fdbe414b470 2024-03-28T14:08:42.275 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:42.266+0000 7fdbe1d9b640 1 --2- 172.21.15.104:0/2324803235 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fdbcc004810 0x7fdbcc004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:42.275 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:42.266+0000 7fdbe259c640 1 --2- 172.21.15.104:0/2324803235 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fdbcc0039c0 0x7fdbcc003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:42.275 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:42.266+0000 7fdbe1d9b640 1 -- 172.21.15.104:0/2324803235 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fdbe414b470 msgr2=0x7fdbcc0051d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T14:08:42.275 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:42.266+0000 7fdbe2d9d640 1 --2- 172.21.15.104:0/2324803235 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fdbe414b470 0x7fdbcc0051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:42.275 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:42.266+0000 7fdbe1d9b640 1 --2- 172.21.15.104:0/2324803235 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fdbe414b470 0x7fdbcc0051d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:42.275 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:42.266+0000 7fdbe1d9b640 1 -- 172.21.15.104:0/2324803235 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fdbcc0039c0 msgr2=0x7fdbcc003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T14:08:42.275 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:42.266+0000 7fdbe1d9b640 1 --2- 172.21.15.104:0/2324803235 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fdbcc0039c0 0x7fdbcc003de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:42.275 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:42.266+0000 7fdbe1d9b640 1 -- 172.21.15.104:0/2324803235 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdbcc003530 con 0x7fdbcc004810 2024-03-28T14:08:42.276 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:42.266+0000 7fdbe259c640 1 --2- 172.21.15.104:0/2324803235 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fdbcc0039c0 0x7fdbcc003de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T14:08:42.276 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:42.270+0000 7fdbe1d9b640 1 --2- 172.21.15.104:0/2324803235 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fdbcc004810 0x7fdbcc004c90 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7fdbd8009ab0 tx=0x7fdbd801ca10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4978e6579ad2adc1 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:08:42.276 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:42.270+0000 7fdbe159a640 1 -- 172.21.15.104:0/2324803235 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdbd80096b0 con 0x7fdbcc004810 2024-03-28T14:08:42.276 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:42.270+0000 7fdbe159a640 1 -- 172.21.15.104:0/2324803235 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdbd80a6500 con 0x7fdbcc004810 2024-03-28T14:08:42.276 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:42.270+0000 7fdbe159a640 1 -- 172.21.15.104:0/2324803235 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdbd80a68f0 con 0x7fdbcc004810 2024-03-28T14:08:42.280 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:42.270+0000 7fdbe8df3640 1 -- 172.21.15.104:0/2324803235 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fdbcc004810 msgr2=0x7fdbcc004c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:42.280 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:42.270+0000 7fdbe8df3640 1 --2- 172.21.15.104:0/2324803235 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fdbcc004810 0x7fdbcc004c90 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7fdbd8009ab0 tx=0x7fdbd801ca10 comp rx=0 tx=0).stop 2024-03-28T14:08:42.280 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:42.270+0000 7fdbe2d9d640 1 -- 172.21.15.104:0/2324803235 reap_dead start 2024-03-28T14:08:42.281 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:42.270+0000 7fdbe8df3640 1 -- 172.21.15.104:0/2324803235 shutdown_connections 2024-03-28T14:08:42.281 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:42.270+0000 7fdbe8df3640 1 -- 172.21.15.104:0/2324803235 >> 172.21.15.104:0/2324803235 conn(0x7fdbe4073530 msgr2=0x7fdbe40bd070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T14:08:42.281 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:42.270+0000 7fdbe8df3640 1 -- 172.21.15.104:0/2324803235 shutdown_connections 2024-03-28T14:08:42.281 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:42.270+0000 7fdbe8df3640 1 -- 172.21.15.104:0/2324803235 wait complete. 2024-03-28T14:08:42.281 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:42.270+0000 7fdbe8df3640 1 Processor -- start 2024-03-28T14:08:42.281 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:42.270+0000 7fdbe8df3640 1 -- start start 2024-03-28T14:08:42.281 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:42.270+0000 7fdbe8df3640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fdbcc004810 0x7fdbe41e3e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:42.281 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:42.270+0000 7fdbe8df3640 1 --2- >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fdbe4140350 0x7fdbe414d070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:42.281 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:42.270+0000 7fdbe8df3640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fdbe41391b0 0x7fdbe4139630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:42.281 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:42.270+0000 7fdbe8df3640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7fdbe40b6140 con 0x7fdbcc004810 2024-03-28T14:08:42.281 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:42.270+0000 7fdbe8df3640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7fdbe40b5940 con 0x7fdbe41391b0 2024-03-28T14:08:42.281 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:42.270+0000 7fdbe8df3640 1 -- --> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] -- mon_getmap magic: 0 -- 0x7fdbe40b62c0 con 0x7fdbe4140350 2024-03-28T14:08:42.282 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:42.270+0000 7fdbe259c640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fdbcc004810 0x7fdbe41e3e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:42.282 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:42.270+0000 7fdbe259c640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fdbcc004810 0x7fdbe41e3e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:50374/0 (socket says 172.21.15.104:50374) 2024-03-28T14:08:42.282 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:42.270+0000 7fdbe259c640 1 -- 172.21.15.104:0/943867006 learned_addr learned my addr 172.21.15.104:0/943867006 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-03-28T14:08:42.282 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:42.270+0000 7fdbe2d9d640 1 --2- 172.21.15.104:0/943867006 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fdbe41391b0 0x7fdbe4139630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:42.282 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:42.270+0000 7fdbe1d9b640 1 --2- 172.21.15.104:0/943867006 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fdbe4140350 0x7fdbe414d070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:42.282 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:42.274+0000 7fdbe259c640 1 -- 172.21.15.104:0/943867006 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fdbe4140350 msgr2=0x7fdbe414d070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:42.283 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:42.274+0000 7fdbe259c640 1 --2- 172.21.15.104:0/943867006 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fdbe4140350 0x7fdbe414d070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:42.283 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:42.274+0000 7fdbe259c640 1 -- 172.21.15.104:0/943867006 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fdbe41391b0 msgr2=0x7fdbe4139630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:42.283 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:42.274+0000 7fdbe259c640 1 --2- 172.21.15.104:0/943867006 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fdbe41391b0 0x7fdbe4139630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:42.283 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:42.274+0000 7fdbe259c640 1 -- 172.21.15.104:0/943867006 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdbe4142470 con 0x7fdbcc004810 2024-03-28T14:08:42.283 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:42.274+0000 7fdbe1d9b640 1 --2- 172.21.15.104:0/943867006 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fdbe4140350 0x7fdbe414d070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T14:08:42.283 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:42.274+0000 7fdbe259c640 1 --2- 172.21.15.104:0/943867006 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fdbcc004810 0x7fdbe41e3e70 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7fdbd40109b0 tx=0x7fdbd4010e70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:08:42.283 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:42.274+0000 7fdbd2ffd640 1 -- 172.21.15.104:0/943867006 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdbd400f040 con 0x7fdbcc004810 2024-03-28T14:08:42.283 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:42.274+0000 7fdbd2ffd640 1 -- 172.21.15.104:0/943867006 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdbd400ec20 con 0x7fdbcc004810 2024-03-28T14:08:42.283 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:42.274+0000 7fdbd2ffd640 1 -- 172.21.15.104:0/943867006 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdbd4004020 con 0x7fdbcc004810 2024-03-28T14:08:42.283 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:42.274+0000 7fdbe8df3640 1 -- 172.21.15.104:0/943867006 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdbe4142190 con 0x7fdbcc004810 2024-03-28T14:08:42.283 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:42.274+0000 7fdbe8df3640 1 -- 172.21.15.104:0/943867006 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdbe4139e90 con 0x7fdbcc004810 2024-03-28T14:08:42.284 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:42.274+0000 7fdbe8df3640 1 -- 172.21.15.104:0/943867006 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdbe413a0d0 con 0x7fdbcc004810 2024-03-28T14:08:42.284 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:42.274+0000 7fdbd2ffd640 1 -- 172.21.15.104:0/943867006 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) ==== 87803+0+0 (secure 0 0 0) 0x7fdbd40041c0 con 0x7fdbcc004810 2024-03-28T14:08:42.284 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:42.274+0000 7fdbd2ffd640 1 --2- 172.21.15.104:0/943867006 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7fdbc4067ff0 0x7fdbc406a4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:42.284 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:42.274+0000 7fdbd2ffd640 1 -- 172.21.15.104:0/943867006 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(29..29 src has 1..29) ==== 5709+0+0 (secure 0 0 0) 0x7fdbd40868a0 con 0x7fdbcc004810 2024-03-28T14:08:42.284 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:42.274+0000 7fdbe1d9b640 1 --2- 172.21.15.104:0/943867006 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7fdbc4067ff0 0x7fdbc406a4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:42.284 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:42.274+0000 7fdbe1d9b640 1 --2- 172.21.15.104:0/943867006 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7fdbc4067ff0 0x7fdbc406a4b0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fdbd80af000 tx=0x7fdbd80046d0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:08:42.284 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:42.278+0000 7fdbd2ffd640 1 -- 172.21.15.104:0/943867006 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179117 (secure 0 0 0) 0x7fdbe4142470 con 0x7fdbcc004810 2024-03-28T14:08:42.464 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:42.462+0000 7fdbe8df3640 1 -- 172.21.15.104:0/943867006 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) -- 0x7fdbe40687c0 con 0x7fdbcc004810 2024-03-28T14:08:42.511 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:42.506+0000 7fdbd2ffd640 1 -- 172.21.15.104:0/943867006 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v13) ==== 118+0+0 (secure 0 0 0) 0x7fdbd4018030 con 0x7fdbcc004810 2024-03-28T14:08:42.513 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:42.510+0000 7fdbe8df3640 1 -- 172.21.15.104:0/943867006 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7fdbc4067ff0 msgr2=0x7fdbc406a4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:42.513 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:42.510+0000 7fdbe8df3640 1 --2- 172.21.15.104:0/943867006 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7fdbc4067ff0 0x7fdbc406a4b0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fdbd80af000 tx=0x7fdbd80046d0 comp rx=0 tx=0).stop 2024-03-28T14:08:42.513 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:42.510+0000 7fdbe8df3640 1 -- 172.21.15.104:0/943867006 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fdbcc004810 msgr2=0x7fdbe41e3e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:42.513 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:42.510+0000 7fdbe8df3640 1 --2- 172.21.15.104:0/943867006 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fdbcc004810 0x7fdbe41e3e70 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7fdbd40109b0 tx=0x7fdbd4010e70 comp rx=0 tx=0).stop 2024-03-28T14:08:42.513 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:42.510+0000 7fdbe8df3640 1 -- 172.21.15.104:0/943867006 shutdown_connections 2024-03-28T14:08:42.514 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:42.510+0000 7fdbe8df3640 1 --2- 172.21.15.104:0/943867006 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fdbe41391b0 0x7fdbe4139630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:42.514 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:42.510+0000 7fdbe8df3640 1 --2- 172.21.15.104:0/943867006 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fdbe4140350 0x7fdbe414d070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:42.514 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:42.510+0000 7fdbe8df3640 1 --2- 172.21.15.104:0/943867006 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7fdbc4067ff0 0x7fdbc406a4b0 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:42.514 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:42.510+0000 7fdbe8df3640 1 --2- 172.21.15.104:0/943867006 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fdbcc004810 0x7fdbe41e3e70 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:42.514 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:42.510+0000 7fdbe8df3640 1 -- 172.21.15.104:0/943867006 >> 172.21.15.104:0/943867006 conn(0x7fdbe4073530 msgr2=0x7fdbe4146fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T14:08:42.514 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:42.510+0000 7fdbe8df3640 1 -- 172.21.15.104:0/943867006 shutdown_connections 2024-03-28T14:08:42.514 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:42.510+0000 7fdbe8df3640 1 -- 172.21.15.104:0/943867006 wait complete. 2024-03-28T14:08:42.536 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-03-28T14:08:42.536 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-03-28T14:08:42.647 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:42.642+0000 7fd5b57ca640 1 Processor -- start 2024-03-28T14:08:42.648 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:42.642+0000 7fd5b57ca640 1 -- start start 2024-03-28T14:08:42.650 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:42.646+0000 7fd5b57ca640 1 --2- >> v2:172.21.15.171:3300/0 conn(0x7fd5b0072bf0 0x7fd5b00b1e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:42.651 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:42.646+0000 7fd5b57ca640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd5b0072fd0 con 0x7fd5b0145ce0 2024-03-28T14:08:42.651 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:42.646+0000 7fd5b57ca640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd5b00734a0 con 0x7fd5b0071be0 2024-03-28T14:08:42.651 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:42.646+0000 7fd5b57ca640 1 -- --> v2:172.21.15.171:3300/0 -- mon_getmap magic: 0 -- 0x7fd5b0073250 con 0x7fd5b0072bf0 2024-03-28T14:08:42.651 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:42.646+0000 7fd5af7fe640 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7fd5b0145ce0 0x7fd5b014fe90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:59836/0 (socket says 172.21.15.104:59836) 2024-03-28T14:08:42.651 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:42.646+0000 7fd5af7fe640 1 -- 172.21.15.104:0/2016231804 learned_addr learned my addr 172.21.15.104:0/2016231804 (peer_addr_for_me v1:172.21.15.104:0/0) 2024-03-28T14:08:42.651 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:42.646+0000 7fd5ae7fc640 1 -- 172.21.15.104:0/2016231804 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1503715335 0 0) 0x7fd5b0072fd0 con 0x7fd5b0145ce0 2024-03-28T14:08:42.651 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:42.646+0000 7fd5ae7fc640 1 -- 172.21.15.104:0/2016231804 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd59c003530 con 0x7fd5b0145ce0 2024-03-28T14:08:42.651 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:42.646+0000 7fd5aeffd640 1 --2- 172.21.15.104:0/2016231804 >> v2:172.21.15.171:3300/0 conn(0x7fd5b0072bf0 0x7fd5b00b1e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:42.651 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:42.646+0000 7fd5ae7fc640 1 -- 172.21.15.104:0/2016231804 <== mon.0 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2613393006 0 0) 0x7fd59c003530 con 0x7fd5b0145ce0 2024-03-28T14:08:42.651 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:42.646+0000 7fd5ae7fc640 1 -- 172.21.15.104:0/2016231804 >> v2:172.21.15.171:3300/0 conn(0x7fd5b0072bf0 msgr2=0x7fd5b00b1e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T14:08:42.651 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:42.646+0000 7fd5ae7fc640 1 --2- 172.21.15.104:0/2016231804 >> v2:172.21.15.171:3300/0 conn(0x7fd5b0072bf0 0x7fd5b00b1e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:42.652 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:42.646+0000 7fd5ae7fc640 1 -- 172.21.15.104:0/2016231804 >> v1:172.21.15.171:6789/0 conn(0x7fd5b0071be0 legacy=0x7fd5b00b12f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T14:08:42.652 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:42.646+0000 7fd5ae7fc640 1 -- 172.21.15.104:0/2016231804 --> v1:172.21.15.104:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd5b0073780 con 0x7fd5b0145ce0 2024-03-28T14:08:42.652 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:42.646+0000 7fd5ae7fc640 1 -- 172.21.15.104:0/2016231804 <== mon.0 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3745580849 0 0) 0x7fd5a8001a00 con 0x7fd5b0145ce0 2024-03-28T14:08:42.652 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:42.646+0000 7fd5ae7fc640 1 -- 172.21.15.104:0/2016231804 >> v1:172.21.15.104:6789/0 conn(0x7fd5b0145ce0 legacy=0x7fd5b014fe90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:42.652 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:42.646+0000 7fd5ae7fc640 1 --2- 172.21.15.104:0/2016231804 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fd59c0039c0 0x7fd59c003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:42.652 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:42.646+0000 7fd5ae7fc640 1 --2- 172.21.15.104:0/2016231804 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd59c004810 0x7fd59c004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:42.652 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:42.646+0000 7fd5ae7fc640 1 --2- 172.21.15.104:0/2016231804 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fd59c0053d0 0x7fd59c00b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:42.652 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:42.646+0000 7fd5ae7fc640 1 -- 172.21.15.104:0/2016231804 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7fd5b0073250 con 0x7fd59c004810 2024-03-28T14:08:42.652 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:42.646+0000 7fd5ae7fc640 1 -- 172.21.15.104:0/2016231804 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7fd59c00c260 con 0x7fd59c0039c0 2024-03-28T14:08:42.652 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:42.646+0000 7fd5ae7fc640 1 -- 172.21.15.104:0/2016231804 --> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] -- mon_getmap magic: 0 -- 0x7fd59c00c460 con 0x7fd59c0053d0 2024-03-28T14:08:42.652 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:42.646+0000 7fd5a7fff640 1 --2- 172.21.15.104:0/2016231804 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd59c004810 0x7fd59c004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:42.652 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:42.646+0000 7fd5aeffd640 1 --2- 172.21.15.104:0/2016231804 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fd59c0039c0 0x7fd59c003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:42.653 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:42.646+0000 7fd5af7fe640 1 --2- 172.21.15.104:0/2016231804 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fd59c0053d0 0x7fd59c00b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:42.653 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:42.646+0000 7fd5a7fff640 1 -- 172.21.15.104:0/2016231804 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fd59c0053d0 msgr2=0x7fd59c00b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T14:08:42.653 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:42.646+0000 7fd5a7fff640 1 --2- 172.21.15.104:0/2016231804 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fd59c0053d0 0x7fd59c00b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:42.653 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:42.646+0000 7fd5a7fff640 1 -- 172.21.15.104:0/2016231804 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fd59c0039c0 msgr2=0x7fd59c003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T14:08:42.653 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:42.646+0000 7fd5a7fff640 1 --2- 172.21.15.104:0/2016231804 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fd59c0039c0 0x7fd59c003de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:42.653 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:42.646+0000 7fd5a7fff640 1 -- 172.21.15.104:0/2016231804 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd59c003530 con 0x7fd59c004810 2024-03-28T14:08:42.653 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:42.646+0000 7fd5af7fe640 1 -- 172.21.15.104:0/2016231804 reap_dead start 2024-03-28T14:08:42.657 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:42.646+0000 7fd5a7fff640 1 --2- 172.21.15.104:0/2016231804 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd59c004810 0x7fd59c004c90 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7fd598001780 tx=0x7fd59801c400 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c786fefc418405ff server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:08:42.657 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:42.646+0000 7fd5ae7fc640 1 -- 172.21.15.104:0/2016231804 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5980013a0 con 0x7fd59c004810 2024-03-28T14:08:42.657 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:42.650+0000 7fd5ae7fc640 1 -- 172.21.15.104:0/2016231804 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd5980052f0 con 0x7fd59c004810 2024-03-28T14:08:42.657 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:42.650+0000 7fd5ae7fc640 1 -- 172.21.15.104:0/2016231804 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd598003840 con 0x7fd59c004810 2024-03-28T14:08:42.657 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:42.650+0000 7fd5b57ca640 1 -- 172.21.15.104:0/2016231804 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd59c004810 msgr2=0x7fd59c004c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:42.657 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:42.650+0000 7fd5b57ca640 1 --2- 172.21.15.104:0/2016231804 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd59c004810 0x7fd59c004c90 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7fd598001780 tx=0x7fd59801c400 comp rx=0 tx=0).stop 2024-03-28T14:08:42.658 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:42.650+0000 7fd5b57ca640 1 -- 172.21.15.104:0/2016231804 shutdown_connections 2024-03-28T14:08:42.658 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:42.650+0000 7fd5b57ca640 1 --2- 172.21.15.104:0/2016231804 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd59c004810 0x7fd59c004c90 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:42.658 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:42.650+0000 7fd5b57ca640 1 -- 172.21.15.104:0/2016231804 >> 172.21.15.104:0/2016231804 conn(0x7fd5b00b7b10 msgr2=0x7fd5b00b7f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T14:08:42.658 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:42.650+0000 7fd5b57ca640 1 -- 172.21.15.104:0/2016231804 shutdown_connections 2024-03-28T14:08:42.658 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:42.650+0000 7fd5b57ca640 1 -- 172.21.15.104:0/2016231804 wait complete. 2024-03-28T14:08:42.658 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:42.650+0000 7fd5b57ca640 1 Processor -- start 2024-03-28T14:08:42.658 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:42.650+0000 7fd5b57ca640 1 -- start start 2024-03-28T14:08:42.658 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:42.650+0000 7fd5b57ca640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fd59c004810 0x7fd5b00b4b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:42.658 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:42.650+0000 7fd5b57ca640 1 --2- >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fd5b01de510 0x7fd5b01de990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:42.659 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:42.650+0000 7fd5b57ca640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd5b013f290 0x7fd5b013f710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:42.659 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:42.650+0000 7fd5b57ca640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7fd5b0148ce0 con 0x7fd5b013f290 2024-03-28T14:08:42.659 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:42.650+0000 7fd5b57ca640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7fd5b0148b60 con 0x7fd59c004810 2024-03-28T14:08:42.660 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:42.650+0000 7fd5b57ca640 1 -- --> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] -- mon_getmap magic: 0 -- 0x7fd5b0148e60 con 0x7fd5b01de510 2024-03-28T14:08:42.660 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:42.650+0000 7fd5af7fe640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd5b013f290 0x7fd5b013f710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:42.660 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:42.650+0000 7fd5af7fe640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd5b013f290 0x7fd5b013f710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:50392/0 (socket says 172.21.15.104:50392) 2024-03-28T14:08:42.660 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:42.650+0000 7fd5af7fe640 1 -- 172.21.15.104:0/794745992 learned_addr learned my addr 172.21.15.104:0/794745992 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-03-28T14:08:42.660 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:42.650+0000 7fd5a7fff640 1 --2- 172.21.15.104:0/794745992 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fd5b01de510 0x7fd5b01de990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:42.660 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:42.650+0000 7fd5aeffd640 1 --2- 172.21.15.104:0/794745992 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fd59c004810 0x7fd5b00b4b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:42.660 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:42.650+0000 7fd5af7fe640 1 -- 172.21.15.104:0/794745992 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fd5b01de510 msgr2=0x7fd5b01de990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:42.660 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:42.650+0000 7fd5af7fe640 1 --2- 172.21.15.104:0/794745992 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fd5b01de510 0x7fd5b01de990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:42.660 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:42.650+0000 7fd5af7fe640 1 -- 172.21.15.104:0/794745992 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fd59c004810 msgr2=0x7fd5b00b4b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:42.660 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:42.650+0000 7fd5af7fe640 1 --2- 172.21.15.104:0/794745992 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fd59c004810 0x7fd5b00b4b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:42.660 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:42.650+0000 7fd5af7fe640 1 -- 172.21.15.104:0/794745992 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd5b01deed0 con 0x7fd5b013f290 2024-03-28T14:08:42.661 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:42.650+0000 7fd5af7fe640 1 --2- 172.21.15.104:0/794745992 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd5b013f290 0x7fd5b013f710 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7fd5a8001380 tx=0x7fd5a800e530 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:08:42.661 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:42.650+0000 7fd5a77fe640 1 -- 172.21.15.104:0/794745992 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5a8007aa0 con 0x7fd5b013f290 2024-03-28T14:08:42.661 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:42.650+0000 7fd5a77fe640 1 -- 172.21.15.104:0/794745992 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd5a8018070 con 0x7fd5b013f290 2024-03-28T14:08:42.661 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:42.650+0000 7fd5b57ca640 1 -- 172.21.15.104:0/794745992 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd5b013fc50 con 0x7fd5b013f290 2024-03-28T14:08:42.661 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:42.650+0000 7fd5a77fe640 1 -- 172.21.15.104:0/794745992 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5a8013420 con 0x7fd5b013f290 2024-03-28T14:08:42.661 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:42.650+0000 7fd5b57ca640 1 -- 172.21.15.104:0/794745992 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd5b00b53b0 con 0x7fd5b013f290 2024-03-28T14:08:42.661 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:42.654+0000 7fd5b57ca640 1 -- 172.21.15.104:0/794745992 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd5b00687c0 con 0x7fd5b013f290 2024-03-28T14:08:42.661 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:42.654+0000 7fd5a77fe640 1 -- 172.21.15.104:0/794745992 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) ==== 87803+0+0 (secure 0 0 0) 0x7fd5a80135c0 con 0x7fd5b013f290 2024-03-28T14:08:42.662 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:42.654+0000 7fd5a77fe640 1 --2- 172.21.15.104:0/794745992 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7fd590067fc0 0x7fd59006a480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:42.662 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:42.654+0000 7fd5a77fe640 1 -- 172.21.15.104:0/794745992 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 5709+0+0 (secure 0 0 0) 0x7fd5a80867e0 con 0x7fd5b013f290 2024-03-28T14:08:42.662 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:42.654+0000 7fd5aeffd640 1 --2- 172.21.15.104:0/794745992 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7fd590067fc0 0x7fd59006a480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:42.662 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:42.654+0000 7fd5aeffd640 1 --2- 172.21.15.104:0/794745992 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7fd590067fc0 0x7fd59006a480 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fd5b0051260 tx=0x7fd5a0008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:08:42.662 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:42.658+0000 7fd5a77fe640 1 -- 172.21.15.104:0/794745992 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179117 (secure 0 0 0) 0x7fd5b013fc50 con 0x7fd5b013f290 2024-03-28T14:08:42.857 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:42.854+0000 7fd5b57ca640 1 -- 172.21.15.104:0/794745992 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) -- 0x7fd5b01de150 con 0x7fd5b013f290 2024-03-28T14:08:43.525 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:43.522+0000 7fd5a77fe640 1 -- 172.21.15.104:0/794745992 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v14) ==== 122+0+0 (secure 0 0 0) 0x7fd5b01deed0 con 0x7fd5b013f290 2024-03-28T14:08:43.529 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:43.526+0000 7fd5b57ca640 1 -- 172.21.15.104:0/794745992 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7fd590067fc0 msgr2=0x7fd59006a480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:43.529 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:43.526+0000 7fd5b57ca640 1 --2- 172.21.15.104:0/794745992 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7fd590067fc0 0x7fd59006a480 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fd5b0051260 tx=0x7fd5a0008040 comp rx=0 tx=0).stop 2024-03-28T14:08:43.529 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:43.526+0000 7fd5b57ca640 1 -- 172.21.15.104:0/794745992 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd5b013f290 msgr2=0x7fd5b013f710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:43.529 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:43.526+0000 7fd5b57ca640 1 --2- 172.21.15.104:0/794745992 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd5b013f290 0x7fd5b013f710 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7fd5a8001380 tx=0x7fd5a800e530 comp rx=0 tx=0).stop 2024-03-28T14:08:43.529 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:43.526+0000 7fd5b57ca640 1 -- 172.21.15.104:0/794745992 shutdown_connections 2024-03-28T14:08:43.529 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:43.526+0000 7fd5b57ca640 1 --2- 172.21.15.104:0/794745992 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7fd590067fc0 0x7fd59006a480 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:43.529 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:43.526+0000 7fd5b57ca640 1 --2- 172.21.15.104:0/794745992 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd5b013f290 0x7fd5b013f710 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:43.529 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:43.526+0000 7fd5b57ca640 1 --2- 172.21.15.104:0/794745992 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fd5b01de510 0x7fd5b01de990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:43.530 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:43.526+0000 7fd5b57ca640 1 --2- 172.21.15.104:0/794745992 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fd59c004810 0x7fd5b00b4b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:43.530 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:43.526+0000 7fd5b57ca640 1 -- 172.21.15.104:0/794745992 >> 172.21.15.104:0/794745992 conn(0x7fd5b00b7b10 msgr2=0x7fd5b00b9580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T14:08:43.530 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:43.526+0000 7fd5b57ca640 1 -- 172.21.15.104:0/794745992 shutdown_connections 2024-03-28T14:08:43.530 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:43.526+0000 7fd5b57ca640 1 -- 172.21.15.104:0/794745992 wait complete. 2024-03-28T14:08:43.548 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-03-28T14:08:43.548 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T14:08:43.667 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:43.662+0000 7f1042121640 1 Processor -- start 2024-03-28T14:08:43.667 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:43.662+0000 7f1042121640 1 -- start start 2024-03-28T14:08:43.668 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:43.666+0000 7f1042121640 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f103c072bf0 0x7f103c0b1e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:43.668 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:43.666+0000 7f1042121640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f103c072fd0 con 0x7f103c071be0 2024-03-28T14:08:43.668 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:43.666+0000 7f1042121640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f103c0734a0 con 0x7f103c145ce0 2024-03-28T14:08:43.668 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:43.666+0000 7f1042121640 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 -- 0x7f103c073250 con 0x7f103c072bf0 2024-03-28T14:08:43.668 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:43.666+0000 7f103b7fe640 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f103c072bf0 0x7f103c0b1e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:43.669 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:43.666+0000 7f103affd640 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f103c071be0 0x7f103c0b12f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:59850/0 (socket says 172.21.15.104:59850) 2024-03-28T14:08:43.669 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:43.666+0000 7f103affd640 1 -- 172.21.15.104:0/926834038 learned_addr learned my addr 172.21.15.104:0/926834038 (peer_addr_for_me v1:172.21.15.104:0/0) 2024-03-28T14:08:43.669 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:43.666+0000 7f103a7fc640 1 -- 172.21.15.104:0/926834038 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3647337267 0 0) 0x7f103c072fd0 con 0x7f103c071be0 2024-03-28T14:08:43.669 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:43.666+0000 7f103a7fc640 1 -- 172.21.15.104:0/926834038 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f101c003530 con 0x7f103c071be0 2024-03-28T14:08:43.669 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:43.666+0000 7f103b7fe640 1 -- 172.21.15.104:0/926834038 >> v1:172.21.15.171:6789/0 conn(0x7f103c145ce0 legacy=0x7f103c14fe90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T14:08:43.669 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:43.666+0000 7f103b7fe640 1 -- 172.21.15.104:0/926834038 >> v1:172.21.15.104:6789/0 conn(0x7f103c071be0 legacy=0x7f103c0b12f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:43.669 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:43.666+0000 7f103b7fe640 1 -- 172.21.15.104:0/926834038 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f103c073780 con 0x7f103c072bf0 2024-03-28T14:08:43.671 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:43.666+0000 7f103b7fe640 1 --2- 172.21.15.104:0/926834038 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f103c072bf0 0x7f103c0b1e80 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f10340097c0 tx=0x7f103401c340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ea78a32a02616ac9 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:08:43.671 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:43.666+0000 7f103a7fc640 1 -- 172.21.15.104:0/926834038 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10340b0070 con 0x7f103c072bf0 2024-03-28T14:08:43.671 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:43.666+0000 7f103a7fc640 1 -- 172.21.15.104:0/926834038 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f10340ab600 con 0x7f103c072bf0 2024-03-28T14:08:43.671 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:43.666+0000 7f103a7fc640 1 -- 172.21.15.104:0/926834038 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10340ab9d0 con 0x7f103c072bf0 2024-03-28T14:08:43.672 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:43.666+0000 7f1042121640 1 -- 172.21.15.104:0/926834038 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f103c072bf0 msgr2=0x7f103c0b1e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:43.672 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:43.666+0000 7f1042121640 1 --2- 172.21.15.104:0/926834038 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f103c072bf0 0x7f103c0b1e80 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f10340097c0 tx=0x7f103401c340 comp rx=0 tx=0).stop 2024-03-28T14:08:43.672 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:43.666+0000 7f1042121640 1 -- 172.21.15.104:0/926834038 shutdown_connections 2024-03-28T14:08:43.672 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:43.666+0000 7f1042121640 1 --2- 172.21.15.104:0/926834038 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f103c072bf0 0x7f103c0b1e80 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:43.673 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:43.666+0000 7f1042121640 1 -- 172.21.15.104:0/926834038 >> 172.21.15.104:0/926834038 conn(0x7f103c0b7b10 msgr2=0x7f103c0b7f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T14:08:43.673 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:43.666+0000 7f1042121640 1 -- 172.21.15.104:0/926834038 shutdown_connections 2024-03-28T14:08:43.673 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:43.666+0000 7f1042121640 1 -- 172.21.15.104:0/926834038 wait complete. 2024-03-28T14:08:43.673 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:43.670+0000 7f1042121640 1 Processor -- start 2024-03-28T14:08:43.673 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:43.670+0000 7f1042121640 1 -- start start 2024-03-28T14:08:43.675 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:43.670+0000 7f1042121640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f103c071be0 0x7f103c09f1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:43.675 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:43.670+0000 7f1042121640 1 --2- >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f103c072bf0 0x7f103c1e0430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:43.675 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:43.670+0000 7f1042121640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f103c145ce0 0x7f103c1e0970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:43.675 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:43.670+0000 7f1042121640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f103c153dd0 con 0x7f103c145ce0 2024-03-28T14:08:43.675 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:43.670+0000 7f1042121640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f103c1455f0 con 0x7f103c071be0 2024-03-28T14:08:43.675 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:43.670+0000 7f1042121640 1 -- --> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] -- mon_getmap magic: 0 -- 0x7f103c153f50 con 0x7f103c072bf0 2024-03-28T14:08:43.675 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:43.670+0000 7f103bfff640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f103c145ce0 0x7f103c1e0970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:43.675 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:43.670+0000 7f103bfff640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f103c145ce0 0x7f103c1e0970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:50398/0 (socket says 172.21.15.104:50398) 2024-03-28T14:08:43.675 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:43.670+0000 7f103bfff640 1 -- 172.21.15.104:0/3234291076 learned_addr learned my addr 172.21.15.104:0/3234291076 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-03-28T14:08:43.675 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:43.670+0000 7f103affd640 1 --2- 172.21.15.104:0/3234291076 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f103c072bf0 0x7f103c1e0430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:43.675 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:43.670+0000 7f103b7fe640 1 --2- 172.21.15.104:0/3234291076 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f103c071be0 0x7f103c09f1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:43.675 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:43.670+0000 7f103bfff640 1 -- 172.21.15.104:0/3234291076 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f103c072bf0 msgr2=0x7f103c1e0430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:43.675 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:43.670+0000 7f103bfff640 1 --2- 172.21.15.104:0/3234291076 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f103c072bf0 0x7f103c1e0430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:43.676 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:43.670+0000 7f103bfff640 1 -- 172.21.15.104:0/3234291076 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f103c071be0 msgr2=0x7f103c09f1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:43.676 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:43.670+0000 7f103bfff640 1 --2- 172.21.15.104:0/3234291076 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f103c071be0 0x7f103c09f1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:43.676 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:43.670+0000 7f103bfff640 1 -- 172.21.15.104:0/3234291076 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f103c09dec0 con 0x7f103c145ce0 2024-03-28T14:08:43.676 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:43.670+0000 7f103bfff640 1 --2- 172.21.15.104:0/3234291076 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f103c145ce0 0x7f103c1e0970 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f102400b9a0 tx=0x7f102400be60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:08:43.676 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:43.670+0000 7f1033fff640 1 -- 172.21.15.104:0/3234291076 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1024019070 con 0x7f103c145ce0 2024-03-28T14:08:43.676 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:43.670+0000 7f1033fff640 1 -- 172.21.15.104:0/3234291076 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f102400cba0 con 0x7f103c145ce0 2024-03-28T14:08:43.676 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:43.670+0000 7f1033fff640 1 -- 172.21.15.104:0/3234291076 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1024014440 con 0x7f103c145ce0 2024-03-28T14:08:43.676 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:43.670+0000 7f1042121640 1 -- 172.21.15.104:0/3234291076 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f103c0a08c0 con 0x7f103c145ce0 2024-03-28T14:08:43.676 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:43.670+0000 7f1042121640 1 -- 172.21.15.104:0/3234291076 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f103c06e020 con 0x7f103c145ce0 2024-03-28T14:08:43.676 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:43.670+0000 7f1042121640 1 -- 172.21.15.104:0/3234291076 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f103c0687c0 con 0x7f103c145ce0 2024-03-28T14:08:43.677 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:43.670+0000 7f1033fff640 1 -- 172.21.15.104:0/3234291076 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) ==== 87803+0+0 (secure 0 0 0) 0x7f102401c070 con 0x7f103c145ce0 2024-03-28T14:08:43.677 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:43.674+0000 7f1033fff640 1 --2- 172.21.15.104:0/3234291076 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f1020067fc0 0x7f102006a480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:43.677 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:43.674+0000 7f1033fff640 1 -- 172.21.15.104:0/3234291076 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 5709+0+0 (secure 0 0 0) 0x7f1024086130 con 0x7f103c145ce0 2024-03-28T14:08:43.677 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:43.674+0000 7f103b7fe640 1 --2- 172.21.15.104:0/3234291076 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f1020067fc0 0x7f102006a480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:43.677 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:43.674+0000 7f103b7fe640 1 --2- 172.21.15.104:0/3234291076 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f1020067fc0 0x7f102006a480 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f10340ad000 tx=0x7f1034008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:08:43.682 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:43.678+0000 7f1033fff640 1 -- 172.21.15.104:0/3234291076 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179117 (secure 0 0 0) 0x7f1024056970 con 0x7f103c145ce0 2024-03-28T14:08:43.873 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:43.870+0000 7f1042121640 1 -- 172.21.15.104:0/3234291076 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f103c1510b0 con 0x7f103c145ce0 2024-03-28T14:08:43.874 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:43.870+0000 7f1033fff640 1 -- 172.21.15.104:0/3234291076 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 14 v14) ==== 94+0+3537 (secure 0 0 0) 0x7f1024056270 con 0x7f103c145ce0 2024-03-28T14:08:43.874 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-03-28T14:08:43.874 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":14,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4550,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.171:6835/3418793776","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6834","nonce":3418793776},{"type":"v1","addr":"172.21.15.171:6835","nonce":3418793776}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11}],"filesystems":[{"mdsmap":{"epoch":14,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T14:08:40.854108+0000","modified":"2024-03-28T14:08:42.859657+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4959},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4959":{"gid":4959,"name":"a","rank":0,"incarnation":12,"state":"up:active","state_seq":3,"addr":"172.21.15.104:6833/1094990707","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":1094990707},{"type":"v1","addr":"172.21.15.104:6833","nonce":1094990707}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4959,"qdb_cluster":[4959]},"id":2}]} 2024-03-28T14:08:43.874 INFO:teuthology.orchestra.run.smithi104.stderr:dumped fsmap epoch 14 2024-03-28T14:08:43.876 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:43.870+0000 7f1042121640 1 -- 172.21.15.104:0/3234291076 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f1020067fc0 msgr2=0x7f102006a480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:43.876 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:43.870+0000 7f1042121640 1 --2- 172.21.15.104:0/3234291076 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f1020067fc0 0x7f102006a480 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f10340ad000 tx=0x7f1034008040 comp rx=0 tx=0).stop 2024-03-28T14:08:43.876 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:43.870+0000 7f1042121640 1 -- 172.21.15.104:0/3234291076 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f103c145ce0 msgr2=0x7f103c1e0970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:43.876 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:43.870+0000 7f1042121640 1 --2- 172.21.15.104:0/3234291076 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f103c145ce0 0x7f103c1e0970 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f102400b9a0 tx=0x7f102400be60 comp rx=0 tx=0).stop 2024-03-28T14:08:43.876 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:43.874+0000 7f1042121640 1 -- 172.21.15.104:0/3234291076 shutdown_connections 2024-03-28T14:08:43.876 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:43.874+0000 7f1042121640 1 --2- 172.21.15.104:0/3234291076 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f1020067fc0 0x7f102006a480 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:43.876 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:43.874+0000 7f1042121640 1 --2- 172.21.15.104:0/3234291076 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f103c145ce0 0x7f103c1e0970 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:43.876 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:43.874+0000 7f1042121640 1 --2- 172.21.15.104:0/3234291076 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f103c072bf0 0x7f103c1e0430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:43.876 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:43.874+0000 7f1042121640 1 --2- 172.21.15.104:0/3234291076 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f103c071be0 0x7f103c09f1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:43.876 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:43.874+0000 7f1042121640 1 -- 172.21.15.104:0/3234291076 >> 172.21.15.104:0/3234291076 conn(0x7f103c0b7b10 msgr2=0x7f103c0b6160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T14:08:43.876 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:43.874+0000 7f1042121640 1 -- 172.21.15.104:0/3234291076 shutdown_connections 2024-03-28T14:08:43.877 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:43.874+0000 7f1042121640 1 -- 172.21.15.104:0/3234291076 wait complete. 2024-03-28T14:08:43.891 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-28T14:08:44.003 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:43.998+0000 7f42f57cf640 1 Processor -- start 2024-03-28T14:08:44.003 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:43.998+0000 7f42f57cf640 1 -- start start 2024-03-28T14:08:44.004 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:44.002+0000 7f42f57cf640 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f42f006b830 0x7f42f009aa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:44.004 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:44.002+0000 7f42f57cf640 1 --2- >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f42f00b4fa0 0x7f42f009c930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:44.004 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:44.002+0000 7f42f57cf640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f42f009c450 con 0x7f42f013f4c0 2024-03-28T14:08:44.004 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:44.002+0000 7f42f57cf640 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 -- 0x7f42f0140000 con 0x7f42f006b830 2024-03-28T14:08:44.004 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:44.002+0000 7f42f57cf640 1 -- --> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] -- mon_getmap magic: 0 -- 0x7f42f006bc10 con 0x7f42f00b4fa0 2024-03-28T14:08:44.005 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:44.002+0000 7f42eeffd640 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f42f006b830 0x7f42f009aa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:44.005 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:44.002+0000 7f42eeffd640 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f42f006b830 0x7f42f009aa10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:50412/0 (socket says 172.21.15.104:50412) 2024-03-28T14:08:44.005 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:44.002+0000 7f42eeffd640 1 -- 172.21.15.104:0/1277028290 learned_addr learned my addr 172.21.15.104:0/1277028290 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-03-28T14:08:44.005 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:44.002+0000 7f42e67fc640 1 --2- 172.21.15.104:0/1277028290 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f42f00b4fa0 0x7f42f009c930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:44.005 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:44.002+0000 7f42eeffd640 1 -- 172.21.15.104:0/1277028290 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f42f00b4fa0 msgr2=0x7f42f009c930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T14:08:44.005 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:44.002+0000 7f42eeffd640 1 --2- 172.21.15.104:0/1277028290 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f42f00b4fa0 0x7f42f009c930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:44.005 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:44.002+0000 7f42eeffd640 1 -- 172.21.15.104:0/1277028290 >> v1:172.21.15.171:6789/0 conn(0x7f42f013f4c0 legacy=0x7f42f013f8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T14:08:44.005 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:44.002+0000 7f42eeffd640 1 -- 172.21.15.104:0/1277028290 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f42f0072ea0 con 0x7f42f006b830 2024-03-28T14:08:44.006 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:44.002+0000 7f42eeffd640 1 --2- 172.21.15.104:0/1277028290 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f42f006b830 0x7f42f009aa10 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f42f0140180 tx=0x7f42e001c470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2f1aaa8cc2fbe5ba server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:08:44.007 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:44.002+0000 7f42ee7fc640 1 -- 172.21.15.104:0/1277028290 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42e00b0070 con 0x7f42f006b830 2024-03-28T14:08:44.007 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:44.002+0000 7f42ee7fc640 1 -- 172.21.15.104:0/1277028290 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f42e00ab600 con 0x7f42f006b830 2024-03-28T14:08:44.007 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:44.002+0000 7f42ee7fc640 1 -- 172.21.15.104:0/1277028290 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42e00ab9d0 con 0x7f42f006b830 2024-03-28T14:08:44.008 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:44.006+0000 7f42f57cf640 1 -- 172.21.15.104:0/1277028290 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f42f006b830 msgr2=0x7f42f009aa10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:44.008 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:44.006+0000 7f42f57cf640 1 --2- 172.21.15.104:0/1277028290 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f42f006b830 0x7f42f009aa10 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f42f0140180 tx=0x7f42e001c470 comp rx=0 tx=0).stop 2024-03-28T14:08:44.008 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:44.006+0000 7f42f57cf640 1 -- 172.21.15.104:0/1277028290 shutdown_connections 2024-03-28T14:08:44.008 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:44.006+0000 7f42f57cf640 1 --2- 172.21.15.104:0/1277028290 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f42f00b4fa0 0x7f42f009c930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:44.008 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:44.006+0000 7f42f57cf640 1 --2- 172.21.15.104:0/1277028290 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f42f006b830 0x7f42f009aa10 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:44.008 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:44.006+0000 7f42f57cf640 1 -- 172.21.15.104:0/1277028290 >> 172.21.15.104:0/1277028290 conn(0x7f42f00ba390 msgr2=0x7f42f00ba7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T14:08:44.009 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:44.006+0000 7f42f57cf640 1 -- 172.21.15.104:0/1277028290 shutdown_connections 2024-03-28T14:08:44.009 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:44.006+0000 7f42f57cf640 1 -- 172.21.15.104:0/1277028290 wait complete. 2024-03-28T14:08:44.011 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:44.006+0000 7f42f57cf640 1 Processor -- start 2024-03-28T14:08:44.011 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:44.006+0000 7f42f57cf640 1 -- start start 2024-03-28T14:08:44.011 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:44.006+0000 7f42f57cf640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f42f006b830 0x7f42f01415c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:44.011 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:44.006+0000 7f42f57cf640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f42f00b4fa0 0x7f42f00aefb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:44.011 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:44.006+0000 7f42f57cf640 1 --2- >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f42f013f4c0 0x7f42f00af4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:44.012 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:44.006+0000 7f42f57cf640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f42f0152900 con 0x7f42f00b4fa0 2024-03-28T14:08:44.012 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:44.006+0000 7f42f57cf640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f42f00b6780 con 0x7f42f006b830 2024-03-28T14:08:44.012 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:44.006+0000 7f42f57cf640 1 -- --> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] -- mon_getmap magic: 0 -- 0x7f42f0152a80 con 0x7f42f013f4c0 2024-03-28T14:08:44.012 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:44.006+0000 7f42e67fc640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f42f00b4fa0 0x7f42f00aefb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:44.012 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:44.006+0000 7f42e67fc640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f42f00b4fa0 0x7f42f00aefb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:50418/0 (socket says 172.21.15.104:50418) 2024-03-28T14:08:44.012 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:44.006+0000 7f42e67fc640 1 -- 172.21.15.104:0/3654469908 learned_addr learned my addr 172.21.15.104:0/3654469908 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-03-28T14:08:44.012 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:44.006+0000 7f42eeffd640 1 --2- 172.21.15.104:0/3654469908 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f42f006b830 0x7f42f01415c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:44.012 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:44.006+0000 7f42ef7fe640 1 --2- 172.21.15.104:0/3654469908 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f42f013f4c0 0x7f42f00af4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:44.012 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:44.006+0000 7f42e67fc640 1 -- 172.21.15.104:0/3654469908 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f42f013f4c0 msgr2=0x7f42f00af4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:44.012 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:44.006+0000 7f42e67fc640 1 --2- 172.21.15.104:0/3654469908 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f42f013f4c0 0x7f42f00af4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:44.012 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:44.006+0000 7f42e67fc640 1 -- 172.21.15.104:0/3654469908 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f42f006b830 msgr2=0x7f42f01415c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:44.012 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:44.006+0000 7f42e67fc640 1 --2- 172.21.15.104:0/3654469908 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f42f006b830 0x7f42f01415c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:44.013 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:44.006+0000 7f42e67fc640 1 -- 172.21.15.104:0/3654469908 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f42f0072b10 con 0x7f42f00b4fa0 2024-03-28T14:08:44.013 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:44.006+0000 7f42eeffd640 1 --2- 172.21.15.104:0/3654469908 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f42f006b830 0x7f42f01415c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T14:08:44.013 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:44.006+0000 7f42ef7fe640 1 --2- 172.21.15.104:0/3654469908 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f42f013f4c0 0x7f42f00af4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T14:08:44.013 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:44.006+0000 7f42e67fc640 1 --2- 172.21.15.104:0/3654469908 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f42f00b4fa0 0x7f42f00aefb0 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f42d800d830 tx=0x7f42d800dcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:08:44.013 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:44.006+0000 7f42e7fff640 1 -- 172.21.15.104:0/3654469908 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42d800b6c0 con 0x7f42f00b4fa0 2024-03-28T14:08:44.013 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:44.006+0000 7f42e7fff640 1 -- 172.21.15.104:0/3654469908 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f42d8019070 con 0x7f42f00b4fa0 2024-03-28T14:08:44.013 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:44.006+0000 7f42f57cf640 1 -- 172.21.15.104:0/3654469908 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f42f0140960 con 0x7f42f00b4fa0 2024-03-28T14:08:44.013 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:44.006+0000 7f42e7fff640 1 -- 172.21.15.104:0/3654469908 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42d800ba60 con 0x7f42f00b4fa0 2024-03-28T14:08:44.013 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:44.006+0000 7f42f57cf640 1 -- 172.21.15.104:0/3654469908 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f42f0140670 con 0x7f42f00b4fa0 2024-03-28T14:08:44.013 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:44.006+0000 7f42f57cf640 1 -- 172.21.15.104:0/3654469908 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f42f00687c0 con 0x7f42f00b4fa0 2024-03-28T14:08:44.014 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:44.010+0000 7f42e7fff640 1 -- 172.21.15.104:0/3654469908 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) ==== 87803+0+0 (secure 0 0 0) 0x7f42d801e460 con 0x7f42f00b4fa0 2024-03-28T14:08:44.014 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:44.010+0000 7f42e7fff640 1 --2- 172.21.15.104:0/3654469908 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f42d0067ff0 0x7f42d006a4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:44.014 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:44.010+0000 7f42e7fff640 1 -- 172.21.15.104:0/3654469908 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 5709+0+0 (secure 0 0 0) 0x7f42d8086ae0 con 0x7f42f00b4fa0 2024-03-28T14:08:44.014 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:44.010+0000 7f42eeffd640 1 --2- 172.21.15.104:0/3654469908 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f42d0067ff0 0x7f42d006a4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:44.014 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:44.010+0000 7f42eeffd640 1 --2- 172.21.15.104:0/3654469908 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f42d0067ff0 0x7f42d006a4b0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f42e00ad000 tx=0x7f42e0008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:08:44.017 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:44.014+0000 7f42e7fff640 1 -- 172.21.15.104:0/3654469908 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179117 (secure 0 0 0) 0x7f42d8057320 con 0x7f42f00b4fa0 2024-03-28T14:08:44.220 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:44.214+0000 7f42f57cf640 1 -- 172.21.15.104:0/3654469908 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f42f0144d40 con 0x7f42f00b4fa0 2024-03-28T14:08:44.222 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:44.218+0000 7f42e7fff640 1 -- 172.21.15.104:0/3654469908 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v30) ==== 74+0+16693 (secure 0 0 0) 0x7f42d8056c20 con 0x7f42f00b4fa0 2024-03-28T14:08:44.231 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-03-28T14:08:44.231 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":30,"fsid":"428a90f2-3351-435f-8fcf-f7e66c78a13e","created":"2024-03-28T14:07:34.862458+0000","modified":"2024-03-28T14:08:42.477796+0000","last_up_change":"2024-03-28T14:07:46.028122+0000","last_in_change":"2024-03-28T14:07:37.157486+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-28T14:07:47.622988+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_metadata","create_time":"2024-03-28T14:08:38.687068+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data","create_time":"2024-03-28T14:08:39.805358+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"30","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"1cb70540-4e42-402f-a446-aaec4ca83007","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6800","nonce":3745690918},{"type":"v1","addr":"172.21.15.104:6801","nonce":3745690918}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":3745690918},{"type":"v1","addr":"172.21.15.104:6803","nonce":3745690918}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":3745690918},{"type":"v1","addr":"172.21.15.104:6807","nonce":3745690918}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":3745690918},{"type":"v1","addr":"172.21.15.104:6805","nonce":3745690918}]},"public_addr":"172.21.15.104:6801/3745690918","cluster_addr":"172.21.15.104:6803/3745690918","heartbeat_back_addr":"172.21.15.104:6807/3745690918","heartbeat_front_addr":"172.21.15.104:6805/3745690918","state":["exists","up"]},{"osd":1,"uuid":"3ab7afdd-b980-418c-81f2-004e16de3dca","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":19438674},{"type":"v1","addr":"172.21.15.104:6810","nonce":19438674}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":19438674},{"type":"v1","addr":"172.21.15.104:6814","nonce":19438674}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":19438674},{"type":"v1","addr":"172.21.15.104:6823","nonce":19438674}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":19438674},{"type":"v1","addr":"172.21.15.104:6819","nonce":19438674}]},"public_addr":"172.21.15.104:6810/19438674","cluster_addr":"172.21.15.104:6814/19438674","heartbeat_back_addr":"172.21.15.104:6823/19438674","heartbeat_front_addr":"172.21.15.104:6819/19438674","state":["exists","up"]},{"osd":2,"uuid":"3e2651c8-ecae-4a9e-a053-27240a79eac2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":2352012626},{"type":"v1","addr":"172.21.15.104:6825","nonce":2352012626}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6826","nonce":2352012626},{"type":"v1","addr":"172.21.15.104:6827","nonce":2352012626}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":2352012626},{"type":"v1","addr":"172.21.15.104:6831","nonce":2352012626}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6828","nonce":2352012626},{"type":"v1","addr":"172.21.15.104:6829","nonce":2352012626}]},"public_addr":"172.21.15.104:6825/2352012626","cluster_addr":"172.21.15.104:6827/2352012626","heartbeat_back_addr":"172.21.15.104:6831/2352012626","heartbeat_front_addr":"172.21.15.104:6829/2352012626","state":["exists","up"]},{"osd":3,"uuid":"925c7e0a-38ef-47cd-b288-8bec8456fb17","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6809","nonce":3568347381},{"type":"v1","addr":"172.21.15.104:6811","nonce":3568347381}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6813","nonce":3568347381},{"type":"v1","addr":"172.21.15.104:6815","nonce":3568347381}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":3568347381},{"type":"v1","addr":"172.21.15.104:6822","nonce":3568347381}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6817","nonce":3568347381},{"type":"v1","addr":"172.21.15.104:6818","nonce":3568347381}]},"public_addr":"172.21.15.104:6811/3568347381","cluster_addr":"172.21.15.104:6815/3568347381","heartbeat_back_addr":"172.21.15.104:6822/3568347381","heartbeat_front_addr":"172.21.15.104:6818/3568347381","state":["exists","up"]},{"osd":4,"uuid":"4a8fac86-e595-4a0e-96e3-57edb3cc3f57","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6800","nonce":2107001077},{"type":"v1","addr":"172.21.15.171:6801","nonce":2107001077}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6802","nonce":2107001077},{"type":"v1","addr":"172.21.15.171:6803","nonce":2107001077}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6806","nonce":2107001077},{"type":"v1","addr":"172.21.15.171:6807","nonce":2107001077}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6804","nonce":2107001077},{"type":"v1","addr":"172.21.15.171:6805","nonce":2107001077}]},"public_addr":"172.21.15.171:6801/2107001077","cluster_addr":"172.21.15.171:6803/2107001077","heartbeat_back_addr":"172.21.15.171:6807/2107001077","heartbeat_front_addr":"172.21.15.171:6805/2107001077","state":["exists","up"]},{"osd":5,"uuid":"e48d50b0-beae-40ae-b15a-9a84dbfe88ab","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6808","nonce":2934826925},{"type":"v1","addr":"172.21.15.171:6810","nonce":2934826925}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6812","nonce":2934826925},{"type":"v1","addr":"172.21.15.171:6814","nonce":2934826925}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6819","nonce":2934826925},{"type":"v1","addr":"172.21.15.171:6821","nonce":2934826925}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6816","nonce":2934826925},{"type":"v1","addr":"172.21.15.171:6817","nonce":2934826925}]},"public_addr":"172.21.15.171:6810/2934826925","cluster_addr":"172.21.15.171:6814/2934826925","heartbeat_back_addr":"172.21.15.171:6821/2934826925","heartbeat_front_addr":"172.21.15.171:6817/2934826925","state":["exists","up"]},{"osd":6,"uuid":"5e7c3292-9936-4f35-b267-aed9d4e49323","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6824","nonce":436742919},{"type":"v1","addr":"172.21.15.171:6825","nonce":436742919}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6826","nonce":436742919},{"type":"v1","addr":"172.21.15.171:6827","nonce":436742919}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6830","nonce":436742919},{"type":"v1","addr":"172.21.15.171:6831","nonce":436742919}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6828","nonce":436742919},{"type":"v1","addr":"172.21.15.171:6829","nonce":436742919}]},"public_addr":"172.21.15.171:6825/436742919","cluster_addr":"172.21.15.171:6827/436742919","heartbeat_back_addr":"172.21.15.171:6831/436742919","heartbeat_front_addr":"172.21.15.171:6829/436742919","state":["exists","up"]},{"osd":7,"uuid":"339b8414-d7f0-4fb0-a446-efaf635cb7ff","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6809","nonce":2055631175},{"type":"v1","addr":"172.21.15.171:6811","nonce":2055631175}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6813","nonce":2055631175},{"type":"v1","addr":"172.21.15.171:6815","nonce":2055631175}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6822","nonce":2055631175},{"type":"v1","addr":"172.21.15.171:6823","nonce":2055631175}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6818","nonce":2055631175},{"type":"v1","addr":"172.21.15.171:6820","nonce":2055631175}]},"public_addr":"172.21.15.171:6811/2055631175","cluster_addr":"172.21.15.171:6815/2055631175","heartbeat_back_addr":"172.21.15.171:6823/2055631175","heartbeat_front_addr":"172.21.15.171:6820/2055631175","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T14:07:42.668881+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T14:07:42.708397+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T14:07:42.634191+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T14:07:42.733662+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T14:07:42.651541+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T14:07:42.783181+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T14:07:42.766310+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T14:07:42.670797+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-28T14:08:44.232 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:44.222+0000 7f42f57cf640 1 -- 172.21.15.104:0/3654469908 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f42d0067ff0 msgr2=0x7f42d006a4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:44.232 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:44.222+0000 7f42f57cf640 1 --2- 172.21.15.104:0/3654469908 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f42d0067ff0 0x7f42d006a4b0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f42e00ad000 tx=0x7f42e0008040 comp rx=0 tx=0).stop 2024-03-28T14:08:44.232 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:44.222+0000 7f42f57cf640 1 -- 172.21.15.104:0/3654469908 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f42f00b4fa0 msgr2=0x7f42f00aefb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:44.232 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:44.222+0000 7f42f57cf640 1 --2- 172.21.15.104:0/3654469908 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f42f00b4fa0 0x7f42f00aefb0 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f42d800d830 tx=0x7f42d800dcf0 comp rx=0 tx=0).stop 2024-03-28T14:08:44.232 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:44.222+0000 7f42f57cf640 1 -- 172.21.15.104:0/3654469908 shutdown_connections 2024-03-28T14:08:44.232 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:44.222+0000 7f42f57cf640 1 --2- 172.21.15.104:0/3654469908 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f42f013f4c0 0x7f42f00af4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:44.233 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:44.222+0000 7f42f57cf640 1 --2- 172.21.15.104:0/3654469908 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f42d0067ff0 0x7f42d006a4b0 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:44.233 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:44.222+0000 7f42f57cf640 1 --2- 172.21.15.104:0/3654469908 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f42f00b4fa0 0x7f42f00aefb0 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:44.233 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:44.222+0000 7f42f57cf640 1 --2- 172.21.15.104:0/3654469908 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f42f006b830 0x7f42f01415c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:44.233 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:44.222+0000 7f42f57cf640 1 -- 172.21.15.104:0/3654469908 >> 172.21.15.104:0/3654469908 conn(0x7f42f00ba390 msgr2=0x7f42f006e1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T14:08:44.233 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:44.222+0000 7f42f57cf640 1 -- 172.21.15.104:0/3654469908 shutdown_connections 2024-03-28T14:08:44.233 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:44.222+0000 7f42f57cf640 1 -- 172.21.15.104:0/3654469908 wait complete. 2024-03-28T14:08:44.241 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-03-28T14:08:44.241 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-28T14:08:44.351 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:44.346+0000 7fd57d002640 1 Processor -- start 2024-03-28T14:08:44.351 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:44.346+0000 7fd57d002640 1 -- start start 2024-03-28T14:08:44.352 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:44.346+0000 7fd57d002640 1 --2- >> v2:172.21.15.171:3300/0 conn(0x7fd5780b5600 0x7fd5781417b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:44.352 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:44.346+0000 7fd57d002640 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fd5780b5120 0x7fd578141cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:44.352 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:44.346+0000 7fd57d002640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd57806e0f0 con 0x7fd57806dd10 2024-03-28T14:08:44.352 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:44.346+0000 7fd57d002640 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 -- 0x7fd5780b0ee0 con 0x7fd5780b5120 2024-03-28T14:08:44.352 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:44.346+0000 7fd57d002640 1 -- --> v2:172.21.15.171:3300/0 -- mon_getmap magic: 0 -- 0x7fd5780b0b70 con 0x7fd5780b5600 2024-03-28T14:08:44.353 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:44.350+0000 7fd57759e640 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fd5780b5120 0x7fd578141cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:44.353 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:44.350+0000 7fd57759e640 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fd5780b5120 0x7fd578141cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:50434/0 (socket says 172.21.15.104:50434) 2024-03-28T14:08:44.353 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:44.350+0000 7fd57759e640 1 -- 172.21.15.104:0/1105796923 learned_addr learned my addr 172.21.15.104:0/1105796923 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-03-28T14:08:44.353 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:44.350+0000 7fd56ffff640 1 --2- 172.21.15.104:0/1105796923 >> v2:172.21.15.171:3300/0 conn(0x7fd5780b5600 0x7fd5781417b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:44.353 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:44.350+0000 7fd57759e640 1 -- 172.21.15.104:0/1105796923 >> v2:172.21.15.171:3300/0 conn(0x7fd5780b5600 msgr2=0x7fd5781417b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T14:08:44.353 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:44.350+0000 7fd57759e640 1 --2- 172.21.15.104:0/1105796923 >> v2:172.21.15.171:3300/0 conn(0x7fd5780b5600 0x7fd5781417b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:44.353 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:44.350+0000 7fd57759e640 1 -- 172.21.15.104:0/1105796923 >> v1:172.21.15.171:6789/0 conn(0x7fd57806dd10 legacy=0x7fd578072cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T14:08:44.353 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:44.350+0000 7fd57759e640 1 -- 172.21.15.104:0/1105796923 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd5780b6130 con 0x7fd5780b5120 2024-03-28T14:08:44.353 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:44.350+0000 7fd56ffff640 1 --2- 172.21.15.104:0/1105796923 >> v2:172.21.15.171:3300/0 conn(0x7fd5780b5600 0x7fd5781417b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T14:08:44.354 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:44.350+0000 7fd57759e640 1 --2- 172.21.15.104:0/1105796923 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd5780b5120 0x7fd578141cf0 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7fd578145610 tx=0x7fd57001c9b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e28859b5415547d2 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:08:44.354 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:44.350+0000 7fd57659c640 1 -- 172.21.15.104:0/1105796923 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5700a5700 con 0x7fd5780b5120 2024-03-28T14:08:44.354 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:44.350+0000 7fd57659c640 1 -- 172.21.15.104:0/1105796923 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd5700a58a0 con 0x7fd5780b5120 2024-03-28T14:08:44.354 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:44.350+0000 7fd57659c640 1 -- 172.21.15.104:0/1105796923 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5700a5c70 con 0x7fd5780b5120 2024-03-28T14:08:44.355 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:44.350+0000 7fd57d002640 1 -- 172.21.15.104:0/1105796923 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd5780b5120 msgr2=0x7fd578141cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:44.355 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:44.350+0000 7fd57d002640 1 --2- 172.21.15.104:0/1105796923 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd5780b5120 0x7fd578141cf0 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7fd578145610 tx=0x7fd57001c9b0 comp rx=0 tx=0).stop 2024-03-28T14:08:44.355 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:44.350+0000 7fd57d002640 1 -- 172.21.15.104:0/1105796923 shutdown_connections 2024-03-28T14:08:44.355 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:44.350+0000 7fd57d002640 1 --2- 172.21.15.104:0/1105796923 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd5780b5120 0x7fd578141cf0 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:44.356 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:44.350+0000 7fd57d002640 1 --2- 172.21.15.104:0/1105796923 >> v2:172.21.15.171:3300/0 conn(0x7fd5780b5600 0x7fd5781417b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:44.356 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:44.350+0000 7fd57d002640 1 -- 172.21.15.104:0/1105796923 >> 172.21.15.104:0/1105796923 conn(0x7fd5780ba390 msgr2=0x7fd5780ba7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T14:08:44.356 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:44.350+0000 7fd57d002640 1 -- 172.21.15.104:0/1105796923 shutdown_connections 2024-03-28T14:08:44.356 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:44.350+0000 7fd57d002640 1 -- 172.21.15.104:0/1105796923 wait complete. 2024-03-28T14:08:44.357 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:44.354+0000 7fd57d002640 1 Processor -- start 2024-03-28T14:08:44.357 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:44.354+0000 7fd57d002640 1 -- start start 2024-03-28T14:08:44.359 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:44.354+0000 7fd57d002640 1 --2- >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fd57806dd10 0x7fd578144760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:44.359 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:44.354+0000 7fd57d002640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fd5780b5120 0x7fd578144ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:44.360 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:44.354+0000 7fd57d002640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd5780b5600 0x7fd5781e3ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:44.360 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:44.354+0000 7fd57d002640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7fd578154e30 con 0x7fd5780b5600 2024-03-28T14:08:44.360 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:44.354+0000 7fd57d002640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7fd578154cb0 con 0x7fd5780b5120 2024-03-28T14:08:44.360 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:44.354+0000 7fd57d002640 1 -- --> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] -- mon_getmap magic: 0 -- 0x7fd578154fb0 con 0x7fd57806dd10 2024-03-28T14:08:44.360 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:44.354+0000 7fd57759e640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd5780b5600 0x7fd5781e3ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:44.360 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:44.354+0000 7fd57759e640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd5780b5600 0x7fd5781e3ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:50444/0 (socket says 172.21.15.104:50444) 2024-03-28T14:08:44.360 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:44.354+0000 7fd57759e640 1 -- 172.21.15.104:0/1615035299 learned_addr learned my addr 172.21.15.104:0/1615035299 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-03-28T14:08:44.360 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:44.354+0000 7fd576d9d640 1 --2- 172.21.15.104:0/1615035299 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fd57806dd10 0x7fd578144760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:44.360 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:44.354+0000 7fd57759e640 1 -- 172.21.15.104:0/1615035299 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fd57806dd10 msgr2=0x7fd578144760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:44.360 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:44.354+0000 7fd57759e640 1 --2- 172.21.15.104:0/1615035299 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fd57806dd10 0x7fd578144760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:44.360 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:44.354+0000 7fd57759e640 1 -- 172.21.15.104:0/1615035299 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fd5780b5120 msgr2=0x7fd578144ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:44.360 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:44.354+0000 7fd57759e640 1 --2- 172.21.15.104:0/1615035299 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fd5780b5120 0x7fd578144ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:44.360 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:44.354+0000 7fd57759e640 1 -- 172.21.15.104:0/1615035299 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd578148df0 con 0x7fd5780b5600 2024-03-28T14:08:44.360 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:44.354+0000 7fd57759e640 1 --2- 172.21.15.104:0/1615035299 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd5780b5600 0x7fd5781e3ab0 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7fd57001a5e0 tx=0x7fd570019750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:08:44.361 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:44.354+0000 7fd56f7fe640 1 -- 172.21.15.104:0/1615035299 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5700bc020 con 0x7fd5780b5600 2024-03-28T14:08:44.361 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:44.354+0000 7fd56f7fe640 1 -- 172.21.15.104:0/1615035299 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd570007b30 con 0x7fd5780b5600 2024-03-28T14:08:44.361 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:44.354+0000 7fd57d002640 1 -- 172.21.15.104:0/1615035299 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd5781490d0 con 0x7fd5780b5600 2024-03-28T14:08:44.361 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:44.354+0000 7fd56f7fe640 1 -- 172.21.15.104:0/1615035299 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd570009510 con 0x7fd5780b5600 2024-03-28T14:08:44.361 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:44.354+0000 7fd57d002640 1 -- 172.21.15.104:0/1615035299 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd5781442c0 con 0x7fd5780b5600 2024-03-28T14:08:44.361 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:44.358+0000 7fd57d002640 1 -- 172.21.15.104:0/1615035299 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd5780687c0 con 0x7fd5780b5600 2024-03-28T14:08:44.362 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:44.358+0000 7fd56f7fe640 1 -- 172.21.15.104:0/1615035299 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) ==== 87803+0+0 (secure 0 0 0) 0x7fd5700bf020 con 0x7fd5780b5600 2024-03-28T14:08:44.362 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:44.358+0000 7fd56f7fe640 1 --2- 172.21.15.104:0/1615035299 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7fd558067fc0 0x7fd55806a480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:44.362 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:44.358+0000 7fd56f7fe640 1 -- 172.21.15.104:0/1615035299 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 5709+0+0 (secure 0 0 0) 0x7fd570121cd0 con 0x7fd5780b5600 2024-03-28T14:08:44.362 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:44.358+0000 7fd576d9d640 1 --2- 172.21.15.104:0/1615035299 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7fd558067fc0 0x7fd55806a480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:44.362 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:44.358+0000 7fd576d9d640 1 --2- 172.21.15.104:0/1615035299 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7fd558067fc0 0x7fd55806a480 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fd568007fd0 tx=0x7fd568009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:08:44.364 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:44.362+0000 7fd56f7fe640 1 -- 172.21.15.104:0/1615035299 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179117 (secure 0 0 0) 0x7fd5700c6030 con 0x7fd5780b5600 2024-03-28T14:08:44.521 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:44.518+0000 7fd57d002640 1 -- 172.21.15.104:0/1615035299 --> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fd578142eb0 con 0x7fd558067fc0 2024-03-28T14:08:44.541 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:44.538+0000 7fd56f7fe640 1 -- 172.21.15.104:0/1615035299 <== mgr.4100 v2:172.21.15.171:6832/1522752412 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+251919 (secure 0 0 0) 0x7fd578142eb0 con 0x7fd558067fc0 2024-03-28T14:08:44.543 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-03-28T14:08:44.545 INFO:teuthology.orchestra.run.smithi104.stderr:dumped all 2024-03-28T14:08:44.546 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:44.542+0000 7fd57d002640 1 -- 172.21.15.104:0/1615035299 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7fd558067fc0 msgr2=0x7fd55806a480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:44.547 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:44.542+0000 7fd57d002640 1 --2- 172.21.15.104:0/1615035299 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7fd558067fc0 0x7fd55806a480 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fd568007fd0 tx=0x7fd568009040 comp rx=0 tx=0).stop 2024-03-28T14:08:44.547 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:44.542+0000 7fd57d002640 1 -- 172.21.15.104:0/1615035299 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd5780b5600 msgr2=0x7fd5781e3ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:44.547 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:44.542+0000 7fd57d002640 1 --2- 172.21.15.104:0/1615035299 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd5780b5600 0x7fd5781e3ab0 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7fd57001a5e0 tx=0x7fd570019750 comp rx=0 tx=0).stop 2024-03-28T14:08:44.547 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:44.542+0000 7fd57d002640 1 -- 172.21.15.104:0/1615035299 shutdown_connections 2024-03-28T14:08:44.547 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:44.542+0000 7fd57d002640 1 --2- 172.21.15.104:0/1615035299 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7fd558067fc0 0x7fd55806a480 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:44.547 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:44.542+0000 7fd57d002640 1 --2- 172.21.15.104:0/1615035299 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd5780b5600 0x7fd5781e3ab0 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:44.547 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:44.542+0000 7fd57d002640 1 --2- 172.21.15.104:0/1615035299 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fd5780b5120 0x7fd578144ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:44.547 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:44.542+0000 7fd57d002640 1 --2- 172.21.15.104:0/1615035299 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fd57806dd10 0x7fd578144760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:44.547 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:44.542+0000 7fd57d002640 1 -- 172.21.15.104:0/1615035299 >> 172.21.15.104:0/1615035299 conn(0x7fd5780ba390 msgr2=0x7fd578148220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T14:08:44.547 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:44.542+0000 7fd57d002640 1 -- 172.21.15.104:0/1615035299 shutdown_connections 2024-03-28T14:08:44.547 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:44.542+0000 7fd57d002640 1 -- 172.21.15.104:0/1615035299 wait complete. 2024-03-28T14:08:44.560 INFO:teuthology.orchestra.run.smithi104.stdout:{"pg_ready":true,"pg_map":{"version":52,"stamp":"2024-03-28T14:08:43.592591+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":203,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":83,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219920,"kb_used_data":3744,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754754800,"statfs":{"total":773094113280,"available":772868915200,"internally_reserved":0,"allocated":3833856,"data_stored":2164516,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":221236785},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":51,"apply_latency_ms":51,"commit_latency_ns":51000000,"apply_latency_ns":51000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.125403"},"pg_stats":[{"pgid":"5.3f","version":"0'0","reported_seq":9,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-28T14:08:41.535396+0000","last_change":"2024-03-28T14:08:41.533604+0000","last_active":"2024-03-28T14:08:41.535396+0000","last_peered":"2024-03-28T14:08:41.535396+0000","last_clean":"2024-03-28T14:08:41.535396+0000","last_became_active":"2024-03-28T14:08:41.532934+0000","last_became_peered":"2024-03-28T14:08:41.532934+0000","last_unstale":"2024-03-28T14:08:41.535396+0000","last_undegraded":"2024-03-28T14:08:41.535396+0000","last_fullsized":"2024-03-28T14:08:41.535396+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:08:40.465288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:08:40.465288+0000","last_clean_scrub_stamp":"2024-03-28T14:08:40.465288+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:59:08.216187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.3e","version":"0'0","reported_seq":9,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-28T14:08:41.509903+0000","last_change":"2024-03-28T14:08:41.504770+0000","last_active":"2024-03-28T14:08:41.509903+0000","last_peered":"2024-03-28T14:08:41.509903+0000","last_clean":"2024-03-28T14:08:41.509903+0000","last_became_active":"2024-03-28T14:08:41.504477+0000","last_became_peered":"2024-03-28T14:08:41.504477+0000","last_unstale":"2024-03-28T14:08:41.509903+0000","last_undegraded":"2024-03-28T14:08:41.509903+0000","last_fullsized":"2024-03-28T14:08:41.509903+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:08:40.465288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:08:40.465288+0000","last_clean_scrub_stamp":"2024-03-28T14:08:40.465288+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:56:54.080557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.3d","version":"0'0","reported_seq":9,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-28T14:08:41.542038+0000","last_change":"2024-03-28T14:08:41.528015+0000","last_active":"2024-03-28T14:08:41.542038+0000","last_peered":"2024-03-28T14:08:41.542038+0000","last_clean":"2024-03-28T14:08:41.542038+0000","last_became_active":"2024-03-28T14:08:41.526802+0000","last_became_peered":"2024-03-28T14:08:41.526802+0000","last_unstale":"2024-03-28T14:08:41.542038+0000","last_undegraded":"2024-03-28T14:08:41.542038+0000","last_fullsized":"2024-03-28T14:08:41.542038+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:08:40.465288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:08:40.465288+0000","last_clean_scrub_stamp":"2024-03-28T14:08:40.465288+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T16:19:05.943365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3c","version":"0'0","reported_seq":9,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-28T14:08:41.514524+0000","last_change":"2024-03-28T14:08:41.505145+0000","last_active":"2024-03-28T14:08:41.514524+0000","last_peered":"2024-03-28T14:08:41.514524+0000","last_clean":"2024-03-28T14:08:41.514524+0000","last_became_active":"2024-03-28T14:08:41.504901+0000","last_became_peered":"2024-03-28T14:08:41.504901+0000","last_unstale":"2024-03-28T14:08:41.514524+0000","last_undegraded":"2024-03-28T14:08:41.514524+0000","last_fullsized":"2024-03-28T14:08:41.514524+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:08:40.465288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:08:40.465288+0000","last_clean_scrub_stamp":"2024-03-28T14:08:40.465288+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T15:29:19.342685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3b","version":"0'0","reported_seq":9,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-28T14:08:41.505644+0000","last_change":"2024-03-28T14:08:41.503669+0000","last_active":"2024-03-28T14:08:41.505644+0000","last_peered":"2024-03-28T14:08:41.505644+0000","last_clean":"2024-03-28T14:08:41.505644+0000","last_became_active":"2024-03-28T14:08:41.503273+0000","last_became_peered":"2024-03-28T14:08:41.503273+0000","last_unstale":"2024-03-28T14:08:41.505644+0000","last_undegraded":"2024-03-28T14:08:41.505644+0000","last_fullsized":"2024-03-28T14:08:41.505644+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:08:40.465288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:08:40.465288+0000","last_clean_scrub_stamp":"2024-03-28T14:08:40.465288+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:35:29.396001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.3a","version":"0'0","reported_seq":9,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-28T14:08:41.532474+0000","last_change":"2024-03-28T14:08:41.529164+0000","last_active":"2024-03-28T14:08:41.532474+0000","last_peered":"2024-03-28T14:08:41.532474+0000","last_clean":"2024-03-28T14:08:41.532474+0000","last_became_active":"2024-03-28T14:08:41.528845+0000","last_became_peered":"2024-03-28T14:08:41.528845+0000","last_unstale":"2024-03-28T14:08:41.532474+0000","last_undegraded":"2024-03-28T14:08:41.532474+0000","last_fullsized":"2024-03-28T14:08:41.532474+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:08:40.465288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:08:40.465288+0000","last_clean_scrub_stamp":"2024-03-28T14:08:40.465288+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:49:54.840279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.39","version":"0'0","reported_seq":9,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-28T14:08:41.539164+0000","last_change":"2024-03-28T14:08:41.524620+0000","last_active":"2024-03-28T14:08:41.539164+0000","last_peered":"2024-03-28T14:08:41.539164+0000","last_clean":"2024-03-28T14:08:41.539164+0000","last_became_active":"2024-03-28T14:08:41.522895+0000","last_became_peered":"2024-03-28T14:08:41.522895+0000","last_unstale":"2024-03-28T14:08:41.539164+0000","last_undegraded":"2024-03-28T14:08:41.539164+0000","last_fullsized":"2024-03-28T14:08:41.539164+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:08:40.465288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:08:40.465288+0000","last_clean_scrub_stamp":"2024-03-28T14:08:40.465288+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T16:19:05.943365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.38","version":"0'0","reported_seq":9,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-28T14:08:41.540347+0000","last_change":"2024-03-28T14:08:41.517882+0000","last_active":"2024-03-28T14:08:41.540347+0000","last_peered":"2024-03-28T14:08:41.540347+0000","last_clean":"2024-03-28T14:08:41.540347+0000","last_became_active":"2024-03-28T14:08:41.517145+0000","last_became_peered":"2024-03-28T14:08:41.517145+0000","last_unstale":"2024-03-28T14:08:41.540347+0000","last_undegraded":"2024-03-28T14:08:41.540347+0000","last_fullsized":"2024-03-28T14:08:41.540347+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:08:40.465288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:08:40.465288+0000","last_clean_scrub_stamp":"2024-03-28T14:08:40.465288+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T16:51:22.768085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.37","version":"0'0","reported_seq":9,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-28T14:08:41.513044+0000","last_change":"2024-03-28T14:08:41.491910+0000","last_active":"2024-03-28T14:08:41.513044+0000","last_peered":"2024-03-28T14:08:41.513044+0000","last_clean":"2024-03-28T14:08:41.513044+0000","last_became_active":"2024-03-28T14:08:41.491064+0000","last_became_peered":"2024-03-28T14:08:41.491064+0000","last_unstale":"2024-03-28T14:08:41.513044+0000","last_undegraded":"2024-03-28T14:08:41.513044+0000","last_fullsized":"2024-03-28T14:08:41.513044+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:08:40.465288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:08:40.465288+0000","last_clean_scrub_stamp":"2024-03-28T14:08:40.465288+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:59:08.216187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.36","version":"0'0","reported_seq":9,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-28T14:08:41.514114+0000","last_change":"2024-03-28T14:08:41.507695+0000","last_active":"2024-03-28T14:08:41.514114+0000","last_peered":"2024-03-28T14:08:41.514114+0000","last_clean":"2024-03-28T14:08:41.514114+0000","last_became_active":"2024-03-28T14:08:41.507228+0000","last_became_peered":"2024-03-28T14:08:41.507228+0000","last_unstale":"2024-03-28T14:08:41.514114+0000","last_undegraded":"2024-03-28T14:08:41.514114+0000","last_fullsized":"2024-03-28T14:08:41.514114+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:08:40.465288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:08:40.465288+0000","last_clean_scrub_stamp":"2024-03-28T14:08:40.465288+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:36:46.949506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.35","version":"0'0","reported_seq":9,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-28T14:08:41.538674+0000","last_change":"2024-03-28T14:08:41.535048+0000","last_active":"2024-03-28T14:08:41.538674+0000","last_peered":"2024-03-28T14:08:41.538674+0000","last_clean":"2024-03-28T14:08:41.538674+0000","last_became_active":"2024-03-28T14:08:41.534360+0000","last_became_peered":"2024-03-28T14:08:41.534360+0000","last_unstale":"2024-03-28T14:08:41.538674+0000","last_undegraded":"2024-03-28T14:08:41.538674+0000","last_fullsized":"2024-03-28T14:08:41.538674+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:08:40.465288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:08:40.465288+0000","last_clean_scrub_stamp":"2024-03-28T14:08:40.465288+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:36:46.949506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.34","version":"0'0","reported_seq":9,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-28T14:08:41.543162+0000","last_change":"2024-03-28T14:08:41.531326+0000","last_active":"2024-03-28T14:08:41.543162+0000","last_peered":"2024-03-28T14:08:41.543162+0000","last_clean":"2024-03-28T14:08:41.543162+0000","last_became_active":"2024-03-28T14:08:41.530414+0000","last_became_peered":"2024-03-28T14:08:41.530414+0000","last_unstale":"2024-03-28T14:08:41.543162+0000","last_undegraded":"2024-03-28T14:08:41.543162+0000","last_fullsized":"2024-03-28T14:08:41.543162+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:08:40.465288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:08:40.465288+0000","last_clean_scrub_stamp":"2024-03-28T14:08:40.465288+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T16:14:54.560822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.33","version":"0'0","reported_seq":9,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-28T14:08:41.508551+0000","last_change":"2024-03-28T14:08:41.507821+0000","last_active":"2024-03-28T14:08:41.508551+0000","last_peered":"2024-03-28T14:08:41.508551+0000","last_clean":"2024-03-28T14:08:41.508551+0000","last_became_active":"2024-03-28T14:08:41.507485+0000","last_became_peered":"2024-03-28T14:08:41.507485+0000","last_unstale":"2024-03-28T14:08:41.508551+0000","last_undegraded":"2024-03-28T14:08:41.508551+0000","last_fullsized":"2024-03-28T14:08:41.508551+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:08:40.465288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:08:40.465288+0000","last_clean_scrub_stamp":"2024-03-28T14:08:40.465288+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:03:29.841049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.32","version":"0'0","reported_seq":9,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-28T14:08:41.509568+0000","last_change":"2024-03-28T14:08:41.503262+0000","last_active":"2024-03-28T14:08:41.509568+0000","last_peered":"2024-03-28T14:08:41.509568+0000","last_clean":"2024-03-28T14:08:41.509568+0000","last_became_active":"2024-03-28T14:08:41.502877+0000","last_became_peered":"2024-03-28T14:08:41.502877+0000","last_unstale":"2024-03-28T14:08:41.509568+0000","last_undegraded":"2024-03-28T14:08:41.509568+0000","last_fullsized":"2024-03-28T14:08:41.509568+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:08:40.465288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:08:40.465288+0000","last_clean_scrub_stamp":"2024-03-28T14:08:40.465288+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T14:33:01.059177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.31","version":"0'0","reported_seq":9,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-28T14:08:41.543811+0000","last_change":"2024-03-28T14:08:41.529966+0000","last_active":"2024-03-28T14:08:41.543811+0000","last_peered":"2024-03-28T14:08:41.543811+0000","last_clean":"2024-03-28T14:08:41.543811+0000","last_became_active":"2024-03-28T14:08:41.529034+0000","last_became_peered":"2024-03-28T14:08:41.529034+0000","last_unstale":"2024-03-28T14:08:41.543811+0000","last_undegraded":"2024-03-28T14:08:41.543811+0000","last_fullsized":"2024-03-28T14:08:41.543811+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:08:40.465288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:08:40.465288+0000","last_clean_scrub_stamp":"2024-03-28T14:08:40.465288+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:02:48.110646+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.30","version":"0'0","reported_seq":9,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-28T14:08:41.540304+0000","last_change":"2024-03-28T14:08:41.524413+0000","last_active":"2024-03-28T14:08:41.540304+0000","last_peered":"2024-03-28T14:08:41.540304+0000","last_clean":"2024-03-28T14:08:41.540304+0000","last_became_active":"2024-03-28T14:08:41.517290+0000","last_became_peered":"2024-03-28T14:08:41.517290+0000","last_unstale":"2024-03-28T14:08:41.540304+0000","last_undegraded":"2024-03-28T14:08:41.540304+0000","last_fullsized":"2024-03-28T14:08:41.540304+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:08:40.465288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:08:40.465288+0000","last_clean_scrub_stamp":"2024-03-28T14:08:40.465288+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T15:29:19.342685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2f","version":"0'0","reported_seq":9,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-28T14:08:41.538613+0000","last_change":"2024-03-28T14:08:41.523393+0000","last_active":"2024-03-28T14:08:41.538613+0000","last_peered":"2024-03-28T14:08:41.538613+0000","last_clean":"2024-03-28T14:08:41.538613+0000","last_became_active":"2024-03-28T14:08:41.522278+0000","last_became_peered":"2024-03-28T14:08:41.522278+0000","last_unstale":"2024-03-28T14:08:41.538613+0000","last_undegraded":"2024-03-28T14:08:41.538613+0000","last_fullsized":"2024-03-28T14:08:41.538613+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:08:40.465288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:08:40.465288+0000","last_clean_scrub_stamp":"2024-03-28T14:08:40.465288+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T17:12:47.786885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2e","version":"0'0","reported_seq":9,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-28T14:08:41.516902+0000","last_change":"2024-03-28T14:08:41.512212+0000","last_active":"2024-03-28T14:08:41.516902+0000","last_peered":"2024-03-28T14:08:41.516902+0000","last_clean":"2024-03-28T14:08:41.516902+0000","last_became_active":"2024-03-28T14:08:41.511884+0000","last_became_peered":"2024-03-28T14:08:41.511884+0000","last_unstale":"2024-03-28T14:08:41.516902+0000","last_undegraded":"2024-03-28T14:08:41.516902+0000","last_fullsized":"2024-03-28T14:08:41.516902+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:08:40.465288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:08:40.465288+0000","last_clean_scrub_stamp":"2024-03-28T14:08:40.465288+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:27:50.434958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2d","version":"0'0","reported_seq":9,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-28T14:08:41.516970+0000","last_change":"2024-03-28T14:08:41.512567+0000","last_active":"2024-03-28T14:08:41.516970+0000","last_peered":"2024-03-28T14:08:41.516970+0000","last_clean":"2024-03-28T14:08:41.516970+0000","last_became_active":"2024-03-28T14:08:41.512173+0000","last_became_peered":"2024-03-28T14:08:41.512173+0000","last_unstale":"2024-03-28T14:08:41.516970+0000","last_undegraded":"2024-03-28T14:08:41.516970+0000","last_fullsized":"2024-03-28T14:08:41.516970+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:08:40.465288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:08:40.465288+0000","last_clean_scrub_stamp":"2024-03-28T14:08:40.465288+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:04:24.834273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2c","version":"0'0","reported_seq":9,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-28T14:08:41.535125+0000","last_change":"2024-03-28T14:08:41.530715+0000","last_active":"2024-03-28T14:08:41.535125+0000","last_peered":"2024-03-28T14:08:41.535125+0000","last_clean":"2024-03-28T14:08:41.535125+0000","last_became_active":"2024-03-28T14:08:41.530416+0000","last_became_peered":"2024-03-28T14:08:41.530416+0000","last_unstale":"2024-03-28T14:08:41.535125+0000","last_undegraded":"2024-03-28T14:08:41.535125+0000","last_fullsized":"2024-03-28T14:08:41.535125+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:08:40.465288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:08:40.465288+0000","last_clean_scrub_stamp":"2024-03-28T14:08:40.465288+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:27:13.776391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.2b","version":"0'0","reported_seq":9,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-28T14:08:41.535767+0000","last_change":"2024-03-28T14:08:41.522747+0000","last_active":"2024-03-28T14:08:41.535767+0000","last_peered":"2024-03-28T14:08:41.535767+0000","last_clean":"2024-03-28T14:08:41.535767+0000","last_became_active":"2024-03-28T14:08:41.520721+0000","last_became_peered":"2024-03-28T14:08:41.520721+0000","last_unstale":"2024-03-28T14:08:41.535767+0000","last_undegraded":"2024-03-28T14:08:41.535767+0000","last_fullsized":"2024-03-28T14:08:41.535767+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:08:40.465288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:08:40.465288+0000","last_clean_scrub_stamp":"2024-03-28T14:08:40.465288+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T17:12:47.786885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.2a","version":"0'0","reported_seq":9,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-28T14:08:41.530823+0000","last_change":"2024-03-28T14:08:41.520125+0000","last_active":"2024-03-28T14:08:41.530823+0000","last_peered":"2024-03-28T14:08:41.530823+0000","last_clean":"2024-03-28T14:08:41.530823+0000","last_became_active":"2024-03-28T14:08:41.519384+0000","last_became_peered":"2024-03-28T14:08:41.519384+0000","last_unstale":"2024-03-28T14:08:41.530823+0000","last_undegraded":"2024-03-28T14:08:41.530823+0000","last_fullsized":"2024-03-28T14:08:41.530823+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:08:40.465288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:08:40.465288+0000","last_clean_scrub_stamp":"2024-03-28T14:08:40.465288+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T18:29:52.676969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.29","version":"0'0","reported_seq":9,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-28T14:08:41.519976+0000","last_change":"2024-03-28T14:08:41.517020+0000","last_active":"2024-03-28T14:08:41.519976+0000","last_peered":"2024-03-28T14:08:41.519976+0000","last_clean":"2024-03-28T14:08:41.519976+0000","last_became_active":"2024-03-28T14:08:41.516686+0000","last_became_peered":"2024-03-28T14:08:41.516686+0000","last_unstale":"2024-03-28T14:08:41.519976+0000","last_undegraded":"2024-03-28T14:08:41.519976+0000","last_fullsized":"2024-03-28T14:08:41.519976+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:08:40.465288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:08:40.465288+0000","last_clean_scrub_stamp":"2024-03-28T14:08:40.465288+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T16:19:05.943365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.28","version":"0'0","reported_seq":9,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-28T14:08:41.541762+0000","last_change":"2024-03-28T14:08:41.527726+0000","last_active":"2024-03-28T14:08:41.541762+0000","last_peered":"2024-03-28T14:08:41.541762+0000","last_clean":"2024-03-28T14:08:41.541762+0000","last_became_active":"2024-03-28T14:08:41.526721+0000","last_became_peered":"2024-03-28T14:08:41.526721+0000","last_unstale":"2024-03-28T14:08:41.541762+0000","last_undegraded":"2024-03-28T14:08:41.541762+0000","last_fullsized":"2024-03-28T14:08:41.541762+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:08:40.465288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:08:40.465288+0000","last_clean_scrub_stamp":"2024-03-28T14:08:40.465288+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:21:52.007093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.27","version":"0'0","reported_seq":9,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-28T14:08:41.514117+0000","last_change":"2024-03-28T14:08:41.504761+0000","last_active":"2024-03-28T14:08:41.514117+0000","last_peered":"2024-03-28T14:08:41.514117+0000","last_clean":"2024-03-28T14:08:41.514117+0000","last_became_active":"2024-03-28T14:08:41.504535+0000","last_became_peered":"2024-03-28T14:08:41.504535+0000","last_unstale":"2024-03-28T14:08:41.514117+0000","last_undegraded":"2024-03-28T14:08:41.514117+0000","last_fullsized":"2024-03-28T14:08:41.514117+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:08:40.465288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:08:40.465288+0000","last_clean_scrub_stamp":"2024-03-28T14:08:40.465288+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T16:51:22.768085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.26","version":"0'0","reported_seq":9,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-28T14:08:41.538261+0000","last_change":"2024-03-28T14:08:41.534067+0000","last_active":"2024-03-28T14:08:41.538261+0000","last_peered":"2024-03-28T14:08:41.538261+0000","last_clean":"2024-03-28T14:08:41.538261+0000","last_became_active":"2024-03-28T14:08:41.533740+0000","last_became_peered":"2024-03-28T14:08:41.533740+0000","last_unstale":"2024-03-28T14:08:41.538261+0000","last_undegraded":"2024-03-28T14:08:41.538261+0000","last_fullsized":"2024-03-28T14:08:41.538261+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:08:40.465288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:08:40.465288+0000","last_clean_scrub_stamp":"2024-03-28T14:08:40.465288+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T15:52:40.770832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.25","version":"0'0","reported_seq":9,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-28T14:08:41.510860+0000","last_change":"2024-03-28T14:08:41.498869+0000","last_active":"2024-03-28T14:08:41.510860+0000","last_peered":"2024-03-28T14:08:41.510860+0000","last_clean":"2024-03-28T14:08:41.510860+0000","last_became_active":"2024-03-28T14:08:41.498422+0000","last_became_peered":"2024-03-28T14:08:41.498422+0000","last_unstale":"2024-03-28T14:08:41.510860+0000","last_undegraded":"2024-03-28T14:08:41.510860+0000","last_fullsized":"2024-03-28T14:08:41.510860+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:08:40.465288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:08:40.465288+0000","last_clean_scrub_stamp":"2024-03-28T14:08:40.465288+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T16:19:05.943365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.24","version":"0'0","reported_seq":9,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-28T14:08:41.543119+0000","last_change":"2024-03-28T14:08:41.530599+0000","last_active":"2024-03-28T14:08:41.543119+0000","last_peered":"2024-03-28T14:08:41.543119+0000","last_clean":"2024-03-28T14:08:41.543119+0000","last_became_active":"2024-03-28T14:08:41.529682+0000","last_became_peered":"2024-03-28T14:08:41.529682+0000","last_unstale":"2024-03-28T14:08:41.543119+0000","last_undegraded":"2024-03-28T14:08:41.543119+0000","last_fullsized":"2024-03-28T14:08:41.543119+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:08:40.465288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:08:40.465288+0000","last_clean_scrub_stamp":"2024-03-28T14:08:40.465288+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:30:28.443913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.23","version":"0'0","reported_seq":9,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-28T14:08:41.536030+0000","last_change":"2024-03-28T14:08:41.524450+0000","last_active":"2024-03-28T14:08:41.536030+0000","last_peered":"2024-03-28T14:08:41.536030+0000","last_clean":"2024-03-28T14:08:41.536030+0000","last_became_active":"2024-03-28T14:08:41.522044+0000","last_became_peered":"2024-03-28T14:08:41.522044+0000","last_unstale":"2024-03-28T14:08:41.536030+0000","last_undegraded":"2024-03-28T14:08:41.536030+0000","last_fullsized":"2024-03-28T14:08:41.536030+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:08:40.465288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:08:40.465288+0000","last_clean_scrub_stamp":"2024-03-28T14:08:40.465288+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:21:52.007093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.22","version":"0'0","reported_seq":9,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-28T14:08:41.538668+0000","last_change":"2024-03-28T14:08:41.524549+0000","last_active":"2024-03-28T14:08:41.538668+0000","last_peered":"2024-03-28T14:08:41.538668+0000","last_clean":"2024-03-28T14:08:41.538668+0000","last_became_active":"2024-03-28T14:08:41.523781+0000","last_became_peered":"2024-03-28T14:08:41.523781+0000","last_unstale":"2024-03-28T14:08:41.538668+0000","last_undegraded":"2024-03-28T14:08:41.538668+0000","last_fullsized":"2024-03-28T14:08:41.538668+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:08:40.465288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:08:40.465288+0000","last_clean_scrub_stamp":"2024-03-28T14:08:40.465288+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:21:52.007093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.21","version":"0'0","reported_seq":9,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-28T14:08:41.518176+0000","last_change":"2024-03-28T14:08:41.513458+0000","last_active":"2024-03-28T14:08:41.518176+0000","last_peered":"2024-03-28T14:08:41.518176+0000","last_clean":"2024-03-28T14:08:41.518176+0000","last_became_active":"2024-03-28T14:08:41.512291+0000","last_became_peered":"2024-03-28T14:08:41.512291+0000","last_unstale":"2024-03-28T14:08:41.518176+0000","last_undegraded":"2024-03-28T14:08:41.518176+0000","last_fullsized":"2024-03-28T14:08:41.518176+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:08:40.465288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:08:40.465288+0000","last_clean_scrub_stamp":"2024-03-28T14:08:40.465288+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:32:42.783009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.20","version":"0'0","reported_seq":9,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-28T14:08:41.546365+0000","last_change":"2024-03-28T14:08:41.539577+0000","last_active":"2024-03-28T14:08:41.546365+0000","last_peered":"2024-03-28T14:08:41.546365+0000","last_clean":"2024-03-28T14:08:41.546365+0000","last_became_active":"2024-03-28T14:08:41.538345+0000","last_became_peered":"2024-03-28T14:08:41.538345+0000","last_unstale":"2024-03-28T14:08:41.546365+0000","last_undegraded":"2024-03-28T14:08:41.546365+0000","last_fullsized":"2024-03-28T14:08:41.546365+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:08:40.465288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:08:40.465288+0000","last_clean_scrub_stamp":"2024-03-28T14:08:40.465288+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:02:48.110646+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":9,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-28T14:08:41.509819+0000","last_change":"2024-03-28T14:08:41.507115+0000","last_active":"2024-03-28T14:08:41.509819+0000","last_peered":"2024-03-28T14:08:41.509819+0000","last_clean":"2024-03-28T14:08:41.509819+0000","last_became_active":"2024-03-28T14:08:41.506763+0000","last_became_peered":"2024-03-28T14:08:41.506763+0000","last_unstale":"2024-03-28T14:08:41.509819+0000","last_undegraded":"2024-03-28T14:08:41.509819+0000","last_fullsized":"2024-03-28T14:08:41.509819+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:08:40.465288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:08:40.465288+0000","last_clean_scrub_stamp":"2024-03-28T14:08:40.465288+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T17:12:47.786885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1f","version":"29'4","reported_seq":16,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-28T14:08:41.542766+0000","last_change":"2024-03-28T14:08:40.509298+0000","last_active":"2024-03-28T14:08:41.542766+0000","last_peered":"2024-03-28T14:08:41.542766+0000","last_clean":"2024-03-28T14:08:41.542766+0000","last_became_active":"2024-03-28T14:08:40.508347+0000","last_became_peered":"2024-03-28T14:08:40.508347+0000","last_unstale":"2024-03-28T14:08:41.542766+0000","last_undegraded":"2024-03-28T14:08:41.542766+0000","last_fullsized":"2024-03-28T14:08:41.542766+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:08:39.461488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:08:39.461488+0000","last_clean_scrub_stamp":"2024-03-28T14:08:39.461488+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:27:15.765153+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1f","version":"0'0","reported_seq":9,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-28T14:08:41.545858+0000","last_change":"2024-03-28T14:08:41.543753+0000","last_active":"2024-03-28T14:08:41.545858+0000","last_peered":"2024-03-28T14:08:41.545858+0000","last_clean":"2024-03-28T14:08:41.545858+0000","last_became_active":"2024-03-28T14:08:41.543201+0000","last_became_peered":"2024-03-28T14:08:41.543201+0000","last_unstale":"2024-03-28T14:08:41.545858+0000","last_undegraded":"2024-03-28T14:08:41.545858+0000","last_fullsized":"2024-03-28T14:08:41.545858+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:08:40.465288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:08:40.465288+0000","last_clean_scrub_stamp":"2024-03-28T14:08:40.465288+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T18:29:52.676969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1e","version":"29'1","reported_seq":13,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-28T14:08:41.526613+0000","last_change":"2024-03-28T14:08:40.522075+0000","last_active":"2024-03-28T14:08:41.526613+0000","last_peered":"2024-03-28T14:08:41.526613+0000","last_clean":"2024-03-28T14:08:41.526613+0000","last_became_active":"2024-03-28T14:08:40.521636+0000","last_became_peered":"2024-03-28T14:08:40.521636+0000","last_unstale":"2024-03-28T14:08:41.526613+0000","last_undegraded":"2024-03-28T14:08:41.526613+0000","last_fullsized":"2024-03-28T14:08:41.526613+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:08:39.461488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:08:39.461488+0000","last_clean_scrub_stamp":"2024-03-28T14:08:39.461488+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:21:17.037306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":9,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-28T14:08:41.521201+0000","last_change":"2024-03-28T14:08:41.518008+0000","last_active":"2024-03-28T14:08:41.521201+0000","last_peered":"2024-03-28T14:08:41.521201+0000","last_clean":"2024-03-28T14:08:41.521201+0000","last_became_active":"2024-03-28T14:08:41.516598+0000","last_became_peered":"2024-03-28T14:08:41.516598+0000","last_unstale":"2024-03-28T14:08:41.521201+0000","last_undegraded":"2024-03-28T14:08:41.521201+0000","last_fullsized":"2024-03-28T14:08:41.521201+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:08:40.465288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:08:40.465288+0000","last_clean_scrub_stamp":"2024-03-28T14:08:40.465288+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:19:23.400242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"29'2","reported_seq":14,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-28T14:08:41.514610+0000","last_change":"2024-03-28T14:08:40.495924+0000","last_active":"2024-03-28T14:08:41.514610+0000","last_peered":"2024-03-28T14:08:41.514610+0000","last_clean":"2024-03-28T14:08:41.514610+0000","last_became_active":"2024-03-28T14:08:40.495575+0000","last_became_peered":"2024-03-28T14:08:40.495575+0000","last_unstale":"2024-03-28T14:08:41.514610+0000","last_undegraded":"2024-03-28T14:08:41.514610+0000","last_fullsized":"2024-03-28T14:08:41.514610+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:08:39.461488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:08:39.461488+0000","last_clean_scrub_stamp":"2024-03-28T14:08:39.461488+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:30:27.440113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":9,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-28T14:08:41.510836+0000","last_change":"2024-03-28T14:08:41.498685+0000","last_active":"2024-03-28T14:08:41.510836+0000","last_peered":"2024-03-28T14:08:41.510836+0000","last_clean":"2024-03-28T14:08:41.510836+0000","last_became_active":"2024-03-28T14:08:41.498095+0000","last_became_peered":"2024-03-28T14:08:41.498095+0000","last_unstale":"2024-03-28T14:08:41.510836+0000","last_undegraded":"2024-03-28T14:08:41.510836+0000","last_fullsized":"2024-03-28T14:08:41.510836+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:08:40.465288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:08:40.465288+0000","last_clean_scrub_stamp":"2024-03-28T14:08:40.465288+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:21:52.007093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1c","version":"29'2","reported_seq":14,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-28T14:08:41.515262+0000","last_change":"2024-03-28T14:08:40.517365+0000","last_active":"2024-03-28T14:08:41.515262+0000","last_peered":"2024-03-28T14:08:41.515262+0000","last_clean":"2024-03-28T14:08:41.515262+0000","last_became_active":"2024-03-28T14:08:40.516963+0000","last_became_peered":"2024-03-28T14:08:40.516963+0000","last_unstale":"2024-03-28T14:08:41.515262+0000","last_undegraded":"2024-03-28T14:08:41.515262+0000","last_fullsized":"2024-03-28T14:08:41.515262+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:08:39.461488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:08:39.461488+0000","last_clean_scrub_stamp":"2024-03-28T14:08:39.461488+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:11:12.076772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":9,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-28T14:08:41.546389+0000","last_change":"2024-03-28T14:08:41.544034+0000","last_active":"2024-03-28T14:08:41.546389+0000","last_peered":"2024-03-28T14:08:41.546389+0000","last_clean":"2024-03-28T14:08:41.546389+0000","last_became_active":"2024-03-28T14:08:41.543317+0000","last_became_peered":"2024-03-28T14:08:41.543317+0000","last_unstale":"2024-03-28T14:08:41.546389+0000","last_undegraded":"2024-03-28T14:08:41.546389+0000","last_fullsized":"2024-03-28T14:08:41.546389+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:08:40.465288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:08:40.465288+0000","last_clean_scrub_stamp":"2024-03-28T14:08:40.465288+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T16:14:54.560822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1b","version":"29'1","reported_seq":13,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-28T14:08:41.535843+0000","last_change":"2024-03-28T14:08:40.508833+0000","last_active":"2024-03-28T14:08:41.535843+0000","last_peered":"2024-03-28T14:08:41.535843+0000","last_clean":"2024-03-28T14:08:41.535843+0000","last_became_active":"2024-03-28T14:08:40.508097+0000","last_became_peered":"2024-03-28T14:08:40.508097+0000","last_unstale":"2024-03-28T14:08:41.535843+0000","last_undegraded":"2024-03-28T14:08:41.535843+0000","last_fullsized":"2024-03-28T14:08:41.535843+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:08:39.461488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:08:39.461488+0000","last_clean_scrub_stamp":"2024-03-28T14:08:39.461488+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:47:33.379209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":9,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-28T14:08:41.542291+0000","last_change":"2024-03-28T14:08:41.528811+0000","last_active":"2024-03-28T14:08:41.542291+0000","last_peered":"2024-03-28T14:08:41.542291+0000","last_clean":"2024-03-28T14:08:41.542291+0000","last_became_active":"2024-03-28T14:08:41.527982+0000","last_became_peered":"2024-03-28T14:08:41.527982+0000","last_unstale":"2024-03-28T14:08:41.542291+0000","last_undegraded":"2024-03-28T14:08:41.542291+0000","last_fullsized":"2024-03-28T14:08:41.542291+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:08:40.465288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:08:40.465288+0000","last_clean_scrub_stamp":"2024-03-28T14:08:40.465288+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:36:46.949506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"29'1","reported_seq":13,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-28T14:08:41.530631+0000","last_change":"2024-03-28T14:08:40.496942+0000","last_active":"2024-03-28T14:08:41.530631+0000","last_peered":"2024-03-28T14:08:41.530631+0000","last_clean":"2024-03-28T14:08:41.530631+0000","last_became_active":"2024-03-28T14:08:40.496220+0000","last_became_peered":"2024-03-28T14:08:40.496220+0000","last_unstale":"2024-03-28T14:08:41.530631+0000","last_undegraded":"2024-03-28T14:08:41.530631+0000","last_fullsized":"2024-03-28T14:08:41.530631+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:08:39.461488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:08:39.461488+0000","last_clean_scrub_stamp":"2024-03-28T14:08:39.461488+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:40:36.997030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":9,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-28T14:08:41.542467+0000","last_change":"2024-03-28T14:08:41.538113+0000","last_active":"2024-03-28T14:08:41.542467+0000","last_peered":"2024-03-28T14:08:41.542467+0000","last_clean":"2024-03-28T14:08:41.542467+0000","last_became_active":"2024-03-28T14:08:41.537473+0000","last_became_peered":"2024-03-28T14:08:41.537473+0000","last_unstale":"2024-03-28T14:08:41.542467+0000","last_undegraded":"2024-03-28T14:08:41.542467+0000","last_fullsized":"2024-03-28T14:08:41.542467+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:08:40.465288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:08:40.465288+0000","last_clean_scrub_stamp":"2024-03-28T14:08:40.465288+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T15:29:19.342685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":12,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-28T14:08:41.497897+0000","last_change":"2024-03-28T14:08:40.506135+0000","last_active":"2024-03-28T14:08:41.497897+0000","last_peered":"2024-03-28T14:08:41.497897+0000","last_clean":"2024-03-28T14:08:41.497897+0000","last_became_active":"2024-03-28T14:08:40.505705+0000","last_became_peered":"2024-03-28T14:08:40.505705+0000","last_unstale":"2024-03-28T14:08:41.497897+0000","last_undegraded":"2024-03-28T14:08:41.497897+0000","last_fullsized":"2024-03-28T14:08:41.497897+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:08:39.461488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:08:39.461488+0000","last_clean_scrub_stamp":"2024-03-28T14:08:39.461488+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:36:45.945706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":9,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-28T14:08:41.538114+0000","last_change":"2024-03-28T14:08:41.531930+0000","last_active":"2024-03-28T14:08:41.538114+0000","last_peered":"2024-03-28T14:08:41.538114+0000","last_clean":"2024-03-28T14:08:41.538114+0000","last_became_active":"2024-03-28T14:08:41.531651+0000","last_became_peered":"2024-03-28T14:08:41.531651+0000","last_unstale":"2024-03-28T14:08:41.538114+0000","last_undegraded":"2024-03-28T14:08:41.538114+0000","last_fullsized":"2024-03-28T14:08:41.538114+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:08:40.465288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:08:40.465288+0000","last_clean_scrub_stamp":"2024-03-28T14:08:40.465288+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T16:14:54.560822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":12,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-28T14:08:41.485979+0000","last_change":"2024-03-28T14:08:40.508061+0000","last_active":"2024-03-28T14:08:41.485979+0000","last_peered":"2024-03-28T14:08:41.485979+0000","last_clean":"2024-03-28T14:08:41.485979+0000","last_became_active":"2024-03-28T14:08:40.507264+0000","last_became_peered":"2024-03-28T14:08:40.507264+0000","last_unstale":"2024-03-28T14:08:41.485979+0000","last_undegraded":"2024-03-28T14:08:41.485979+0000","last_fullsized":"2024-03-28T14:08:41.485979+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:08:39.461488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:08:39.461488+0000","last_clean_scrub_stamp":"2024-03-28T14:08:39.461488+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:27:12.772591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":9,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-28T14:08:41.541821+0000","last_change":"2024-03-28T14:08:41.528643+0000","last_active":"2024-03-28T14:08:41.541821+0000","last_peered":"2024-03-28T14:08:41.541821+0000","last_clean":"2024-03-28T14:08:41.541821+0000","last_became_active":"2024-03-28T14:08:41.527599+0000","last_became_peered":"2024-03-28T14:08:41.527599+0000","last_unstale":"2024-03-28T14:08:41.541821+0000","last_undegraded":"2024-03-28T14:08:41.541821+0000","last_fullsized":"2024-03-28T14:08:41.541821+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:08:40.465288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:08:40.465288+0000","last_clean_scrub_stamp":"2024-03-28T14:08:40.465288+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T16:51:22.768085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":12,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-28T14:08:41.499979+0000","last_change":"2024-03-28T14:08:40.524054+0000","last_active":"2024-03-28T14:08:41.499979+0000","last_peered":"2024-03-28T14:08:41.499979+0000","last_clean":"2024-03-28T14:08:41.499979+0000","last_became_active":"2024-03-28T14:08:40.523618+0000","last_became_peered":"2024-03-28T14:08:40.523618+0000","last_unstale":"2024-03-28T14:08:41.499979+0000","last_undegraded":"2024-03-28T14:08:41.499979+0000","last_fullsized":"2024-03-28T14:08:41.499979+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:08:39.461488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:08:39.461488+0000","last_clean_scrub_stamp":"2024-03-28T14:08:39.461488+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T15:46:07.679056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":9,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-28T14:08:41.516953+0000","last_change":"2024-03-28T14:08:41.512531+0000","last_active":"2024-03-28T14:08:41.516953+0000","last_peered":"2024-03-28T14:08:41.516953+0000","last_clean":"2024-03-28T14:08:41.516953+0000","last_became_active":"2024-03-28T14:08:41.512196+0000","last_became_peered":"2024-03-28T14:08:41.512196+0000","last_unstale":"2024-03-28T14:08:41.516953+0000","last_undegraded":"2024-03-28T14:08:41.516953+0000","last_fullsized":"2024-03-28T14:08:41.516953+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:08:40.465288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:08:40.465288+0000","last_clean_scrub_stamp":"2024-03-28T14:08:40.465288+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:40:18.761368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":12,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-28T14:08:41.490390+0000","last_change":"2024-03-28T14:08:40.509986+0000","last_active":"2024-03-28T14:08:41.490390+0000","last_peered":"2024-03-28T14:08:41.490390+0000","last_clean":"2024-03-28T14:08:41.490390+0000","last_became_active":"2024-03-28T14:08:40.509535+0000","last_became_peered":"2024-03-28T14:08:40.509535+0000","last_unstale":"2024-03-28T14:08:41.490390+0000","last_undegraded":"2024-03-28T14:08:41.490390+0000","last_fullsized":"2024-03-28T14:08:41.490390+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:08:39.461488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:08:39.461488+0000","last_clean_scrub_stamp":"2024-03-28T14:08:39.461488+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:32:41.779209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"0'0","reported_seq":9,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-28T14:08:41.538220+0000","last_change":"2024-03-28T14:08:41.533404+0000","last_active":"2024-03-28T14:08:41.538220+0000","last_peered":"2024-03-28T14:08:41.538220+0000","last_clean":"2024-03-28T14:08:41.538220+0000","last_became_active":"2024-03-28T14:08:41.533141+0000","last_became_peered":"2024-03-28T14:08:41.533141+0000","last_unstale":"2024-03-28T14:08:41.538220+0000","last_undegraded":"2024-03-28T14:08:41.538220+0000","last_fullsized":"2024-03-28T14:08:41.538220+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:08:40.465288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:08:40.465288+0000","last_clean_scrub_stamp":"2024-03-28T14:08:40.465288+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T15:22:58.145356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.15","version":"29'3","reported_seq":15,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-28T14:08:41.542152+0000","last_change":"2024-03-28T14:08:40.523797+0000","last_active":"2024-03-28T14:08:41.542152+0000","last_peered":"2024-03-28T14:08:41.542152+0000","last_clean":"2024-03-28T14:08:41.542152+0000","last_became_active":"2024-03-28T14:08:40.523127+0000","last_became_peered":"2024-03-28T14:08:40.523127+0000","last_unstale":"2024-03-28T14:08:41.542152+0000","last_undegraded":"2024-03-28T14:08:41.542152+0000","last_fullsized":"2024-03-28T14:08:41.542152+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:08:39.461488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:08:39.461488+0000","last_clean_scrub_stamp":"2024-03-28T14:08:39.461488+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T17:12:46.783085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"0'0","reported_seq":9,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-28T14:08:41.542080+0000","last_change":"2024-03-28T14:08:41.528460+0000","last_active":"2024-03-28T14:08:41.542080+0000","last_peered":"2024-03-28T14:08:41.542080+0000","last_clean":"2024-03-28T14:08:41.542080+0000","last_became_active":"2024-03-28T14:08:41.527740+0000","last_became_peered":"2024-03-28T14:08:41.527740+0000","last_unstale":"2024-03-28T14:08:41.542080+0000","last_undegraded":"2024-03-28T14:08:41.542080+0000","last_fullsized":"2024-03-28T14:08:41.542080+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:08:40.465288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:08:40.465288+0000","last_clean_scrub_stamp":"2024-03-28T14:08:40.465288+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T15:29:19.342685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"29'6","reported_seq":18,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-28T14:08:41.538564+0000","last_change":"2024-03-28T14:08:40.495856+0000","last_active":"2024-03-28T14:08:41.538564+0000","last_peered":"2024-03-28T14:08:41.538564+0000","last_clean":"2024-03-28T14:08:41.538564+0000","last_became_active":"2024-03-28T14:08:40.495479+0000","last_became_peered":"2024-03-28T14:08:40.495479+0000","last_unstale":"2024-03-28T14:08:41.538564+0000","last_undegraded":"2024-03-28T14:08:41.538564+0000","last_fullsized":"2024-03-28T14:08:41.538564+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:08:39.461488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:08:39.461488+0000","last_clean_scrub_stamp":"2024-03-28T14:08:39.461488+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T16:14:53.557022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":9,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-28T14:08:41.534955+0000","last_change":"2024-03-28T14:08:41.529102+0000","last_active":"2024-03-28T14:08:41.534955+0000","last_peered":"2024-03-28T14:08:41.534955+0000","last_clean":"2024-03-28T14:08:41.534955+0000","last_became_active":"2024-03-28T14:08:41.528836+0000","last_became_peered":"2024-03-28T14:08:41.528836+0000","last_unstale":"2024-03-28T14:08:41.534955+0000","last_undegraded":"2024-03-28T14:08:41.534955+0000","last_fullsized":"2024-03-28T14:08:41.534955+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:08:40.465288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:08:40.465288+0000","last_clean_scrub_stamp":"2024-03-28T14:08:40.465288+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:30:28.443913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"29'2","reported_seq":14,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-28T14:08:41.546144+0000","last_change":"2024-03-28T14:08:40.515762+0000","last_active":"2024-03-28T14:08:41.546144+0000","last_peered":"2024-03-28T14:08:41.546144+0000","last_clean":"2024-03-28T14:08:41.546144+0000","last_became_active":"2024-03-28T14:08:40.514769+0000","last_became_peered":"2024-03-28T14:08:40.514769+0000","last_unstale":"2024-03-28T14:08:41.546144+0000","last_undegraded":"2024-03-28T14:08:41.546144+0000","last_fullsized":"2024-03-28T14:08:41.546144+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:08:39.461488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:08:39.461488+0000","last_clean_scrub_stamp":"2024-03-28T14:08:39.461488+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:49:14.011336+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":9,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-28T14:08:41.516873+0000","last_change":"2024-03-28T14:08:41.511329+0000","last_active":"2024-03-28T14:08:41.516873+0000","last_peered":"2024-03-28T14:08:41.516873+0000","last_clean":"2024-03-28T14:08:41.516873+0000","last_became_active":"2024-03-28T14:08:41.510972+0000","last_became_peered":"2024-03-28T14:08:41.510972+0000","last_unstale":"2024-03-28T14:08:41.516873+0000","last_undegraded":"2024-03-28T14:08:41.516873+0000","last_fullsized":"2024-03-28T14:08:41.516873+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:08:40.465288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:08:40.465288+0000","last_clean_scrub_stamp":"2024-03-28T14:08:40.465288+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:12:57.613588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":12,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-28T14:08:41.483578+0000","last_change":"2024-03-28T14:08:40.501882+0000","last_active":"2024-03-28T14:08:41.483578+0000","last_peered":"2024-03-28T14:08:41.483578+0000","last_clean":"2024-03-28T14:08:41.483578+0000","last_became_active":"2024-03-28T14:08:40.501346+0000","last_became_peered":"2024-03-28T14:08:40.501346+0000","last_unstale":"2024-03-28T14:08:41.483578+0000","last_undegraded":"2024-03-28T14:08:41.483578+0000","last_fullsized":"2024-03-28T14:08:41.483578+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:08:39.461488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:08:39.461488+0000","last_clean_scrub_stamp":"2024-03-28T14:08:39.461488+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:59:07.212387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":9,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-28T14:08:41.546410+0000","last_change":"2024-03-28T14:08:41.540015+0000","last_active":"2024-03-28T14:08:41.546410+0000","last_peered":"2024-03-28T14:08:41.546410+0000","last_clean":"2024-03-28T14:08:41.546410+0000","last_became_active":"2024-03-28T14:08:41.539044+0000","last_became_peered":"2024-03-28T14:08:41.539044+0000","last_unstale":"2024-03-28T14:08:41.546410+0000","last_undegraded":"2024-03-28T14:08:41.546410+0000","last_fullsized":"2024-03-28T14:08:41.546410+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:08:40.465288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:08:40.465288+0000","last_clean_scrub_stamp":"2024-03-28T14:08:40.465288+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:30:28.443913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":12,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-28T14:08:41.498040+0000","last_change":"2024-03-28T14:08:40.522631+0000","last_active":"2024-03-28T14:08:41.498040+0000","last_peered":"2024-03-28T14:08:41.498040+0000","last_clean":"2024-03-28T14:08:41.498040+0000","last_became_active":"2024-03-28T14:08:40.522368+0000","last_became_peered":"2024-03-28T14:08:40.522368+0000","last_unstale":"2024-03-28T14:08:41.498040+0000","last_undegraded":"2024-03-28T14:08:41.498040+0000","last_fullsized":"2024-03-28T14:08:41.498040+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:08:39.461488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:08:39.461488+0000","last_clean_scrub_stamp":"2024-03-28T14:08:39.461488+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:02:47.106846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":9,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-28T14:08:41.544936+0000","last_change":"2024-03-28T14:08:41.540916+0000","last_active":"2024-03-28T14:08:41.544936+0000","last_peered":"2024-03-28T14:08:41.544936+0000","last_clean":"2024-03-28T14:08:41.544936+0000","last_became_active":"2024-03-28T14:08:41.540414+0000","last_became_peered":"2024-03-28T14:08:41.540414+0000","last_unstale":"2024-03-28T14:08:41.544936+0000","last_undegraded":"2024-03-28T14:08:41.544936+0000","last_fullsized":"2024-03-28T14:08:41.544936+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:08:40.465288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:08:40.465288+0000","last_clean_scrub_stamp":"2024-03-28T14:08:40.465288+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:19:23.400242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.10","version":"29'8","reported_seq":20,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-28T14:08:41.514972+0000","last_change":"2024-03-28T14:08:40.495790+0000","last_active":"2024-03-28T14:08:41.514972+0000","last_peered":"2024-03-28T14:08:41.514972+0000","last_clean":"2024-03-28T14:08:41.514972+0000","last_became_active":"2024-03-28T14:08:40.495441+0000","last_became_peered":"2024-03-28T14:08:40.495441+0000","last_unstale":"2024-03-28T14:08:41.514972+0000","last_undegraded":"2024-03-28T14:08:41.514972+0000","last_fullsized":"2024-03-28T14:08:41.514972+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:08:39.461488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:08:39.461488+0000","last_clean_scrub_stamp":"2024-03-28T14:08:39.461488+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T15:29:18.338885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"0'0","reported_seq":9,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-28T14:08:41.535815+0000","last_change":"2024-03-28T14:08:41.530295+0000","last_active":"2024-03-28T14:08:41.535815+0000","last_peered":"2024-03-28T14:08:41.535815+0000","last_clean":"2024-03-28T14:08:41.535815+0000","last_became_active":"2024-03-28T14:08:41.529583+0000","last_became_peered":"2024-03-28T14:08:41.529583+0000","last_unstale":"2024-03-28T14:08:41.535815+0000","last_undegraded":"2024-03-28T14:08:41.535815+0000","last_fullsized":"2024-03-28T14:08:41.535815+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:08:40.465288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:08:40.465288+0000","last_clean_scrub_stamp":"2024-03-28T14:08:40.465288+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T16:19:05.943365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.f","version":"29'1","reported_seq":13,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-28T14:08:41.503214+0000","last_change":"2024-03-28T14:08:40.503423+0000","last_active":"2024-03-28T14:08:41.503214+0000","last_peered":"2024-03-28T14:08:41.503214+0000","last_clean":"2024-03-28T14:08:41.503214+0000","last_became_active":"2024-03-28T14:08:40.503039+0000","last_became_peered":"2024-03-28T14:08:40.503039+0000","last_unstale":"2024-03-28T14:08:41.503214+0000","last_undegraded":"2024-03-28T14:08:41.503214+0000","last_fullsized":"2024-03-28T14:08:41.503214+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:08:39.461488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:08:39.461488+0000","last_clean_scrub_stamp":"2024-03-28T14:08:39.461488+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:19:22.396442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":9,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-28T14:08:41.543673+0000","last_change":"2024-03-28T14:08:41.532505+0000","last_active":"2024-03-28T14:08:41.543673+0000","last_peered":"2024-03-28T14:08:41.543673+0000","last_clean":"2024-03-28T14:08:41.543673+0000","last_became_active":"2024-03-28T14:08:41.530486+0000","last_became_peered":"2024-03-28T14:08:41.530486+0000","last_unstale":"2024-03-28T14:08:41.543673+0000","last_undegraded":"2024-03-28T14:08:41.543673+0000","last_fullsized":"2024-03-28T14:08:41.543673+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:08:40.465288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:08:40.465288+0000","last_clean_scrub_stamp":"2024-03-28T14:08:40.465288+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T14:33:01.059177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.e","version":"29'2","reported_seq":14,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-28T14:08:41.545482+0000","last_change":"2024-03-28T14:08:40.513533+0000","last_active":"2024-03-28T14:08:41.545482+0000","last_peered":"2024-03-28T14:08:41.545482+0000","last_clean":"2024-03-28T14:08:41.545482+0000","last_became_active":"2024-03-28T14:08:40.512865+0000","last_became_peered":"2024-03-28T14:08:40.512865+0000","last_unstale":"2024-03-28T14:08:41.545482+0000","last_undegraded":"2024-03-28T14:08:41.545482+0000","last_fullsized":"2024-03-28T14:08:41.545482+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:08:39.461488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:08:39.461488+0000","last_clean_scrub_stamp":"2024-03-28T14:08:39.461488+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:32:41.779209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":9,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-28T14:08:41.520058+0000","last_change":"2024-03-28T14:08:41.518443+0000","last_active":"2024-03-28T14:08:41.520058+0000","last_peered":"2024-03-28T14:08:41.520058+0000","last_clean":"2024-03-28T14:08:41.520058+0000","last_became_active":"2024-03-28T14:08:41.518226+0000","last_became_peered":"2024-03-28T14:08:41.518226+0000","last_unstale":"2024-03-28T14:08:41.520058+0000","last_undegraded":"2024-03-28T14:08:41.520058+0000","last_fullsized":"2024-03-28T14:08:41.520058+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:08:40.465288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:08:40.465288+0000","last_clean_scrub_stamp":"2024-03-28T14:08:40.465288+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:02:48.110646+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.d","version":"29'1","reported_seq":13,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-28T14:08:41.521487+0000","last_change":"2024-03-28T14:08:40.508324+0000","last_active":"2024-03-28T14:08:41.521487+0000","last_peered":"2024-03-28T14:08:41.521487+0000","last_clean":"2024-03-28T14:08:41.521487+0000","last_became_active":"2024-03-28T14:08:40.507410+0000","last_became_peered":"2024-03-28T14:08:40.507410+0000","last_unstale":"2024-03-28T14:08:41.521487+0000","last_undegraded":"2024-03-28T14:08:41.521487+0000","last_fullsized":"2024-03-28T14:08:41.521487+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:08:39.461488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:08:39.461488+0000","last_clean_scrub_stamp":"2024-03-28T14:08:39.461488+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:27:15.765153+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"0'0","reported_seq":9,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-28T14:08:41.526009+0000","last_change":"2024-03-28T14:08:41.524631+0000","last_active":"2024-03-28T14:08:41.526009+0000","last_peered":"2024-03-28T14:08:41.526009+0000","last_clean":"2024-03-28T14:08:41.526009+0000","last_became_active":"2024-03-28T14:08:41.524356+0000","last_became_peered":"2024-03-28T14:08:41.524356+0000","last_unstale":"2024-03-28T14:08:41.526009+0000","last_undegraded":"2024-03-28T14:08:41.526009+0000","last_fullsized":"2024-03-28T14:08:41.526009+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:08:40.465288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:08:40.465288+0000","last_clean_scrub_stamp":"2024-03-28T14:08:40.465288+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:54:36.883461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":12,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-28T14:08:41.486790+0000","last_change":"2024-03-28T14:08:40.509241+0000","last_active":"2024-03-28T14:08:41.486790+0000","last_peered":"2024-03-28T14:08:41.486790+0000","last_clean":"2024-03-28T14:08:41.486790+0000","last_became_active":"2024-03-28T14:08:40.507984+0000","last_became_peered":"2024-03-28T14:08:40.507984+0000","last_unstale":"2024-03-28T14:08:41.486790+0000","last_undegraded":"2024-03-28T14:08:41.486790+0000","last_fullsized":"2024-03-28T14:08:41.486790+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:08:39.461488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:08:39.461488+0000","last_clean_scrub_stamp":"2024-03-28T14:08:39.461488+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:37:19.550407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":9,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-28T14:08:41.522657+0000","last_change":"2024-03-28T14:08:41.520093+0000","last_active":"2024-03-28T14:08:41.522657+0000","last_peered":"2024-03-28T14:08:41.522657+0000","last_clean":"2024-03-28T14:08:41.522657+0000","last_became_active":"2024-03-28T14:08:41.519746+0000","last_became_peered":"2024-03-28T14:08:41.519746+0000","last_unstale":"2024-03-28T14:08:41.522657+0000","last_undegraded":"2024-03-28T14:08:41.522657+0000","last_fullsized":"2024-03-28T14:08:41.522657+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:08:40.465288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:08:40.465288+0000","last_clean_scrub_stamp":"2024-03-28T14:08:40.465288+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:00:07.685993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.b","version":"29'2","reported_seq":14,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-28T14:08:41.514745+0000","last_change":"2024-03-28T14:08:40.502639+0000","last_active":"2024-03-28T14:08:41.514745+0000","last_peered":"2024-03-28T14:08:41.514745+0000","last_clean":"2024-03-28T14:08:41.514745+0000","last_became_active":"2024-03-28T14:08:40.502250+0000","last_became_peered":"2024-03-28T14:08:40.502250+0000","last_unstale":"2024-03-28T14:08:41.514745+0000","last_undegraded":"2024-03-28T14:08:41.514745+0000","last_fullsized":"2024-03-28T14:08:41.514745+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:08:39.461488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:08:39.461488+0000","last_clean_scrub_stamp":"2024-03-28T14:08:39.461488+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:19:22.396442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":9,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-28T14:08:41.514052+0000","last_change":"2024-03-28T14:08:41.511022+0000","last_active":"2024-03-28T14:08:41.514052+0000","last_peered":"2024-03-28T14:08:41.514052+0000","last_clean":"2024-03-28T14:08:41.514052+0000","last_became_active":"2024-03-28T14:08:41.510604+0000","last_became_peered":"2024-03-28T14:08:41.510604+0000","last_unstale":"2024-03-28T14:08:41.514052+0000","last_undegraded":"2024-03-28T14:08:41.514052+0000","last_fullsized":"2024-03-28T14:08:41.514052+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:08:40.465288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:08:40.465288+0000","last_clean_scrub_stamp":"2024-03-28T14:08:40.465288+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:52:05.146099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":12,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-28T14:08:41.500718+0000","last_change":"2024-03-28T14:08:40.503771+0000","last_active":"2024-03-28T14:08:41.500718+0000","last_peered":"2024-03-28T14:08:41.500718+0000","last_clean":"2024-03-28T14:08:41.500718+0000","last_became_active":"2024-03-28T14:08:40.502947+0000","last_became_peered":"2024-03-28T14:08:40.502947+0000","last_unstale":"2024-03-28T14:08:41.500718+0000","last_undegraded":"2024-03-28T14:08:41.500718+0000","last_fullsized":"2024-03-28T14:08:41.500718+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:08:39.461488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:08:39.461488+0000","last_clean_scrub_stamp":"2024-03-28T14:08:39.461488+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:49:14.011336+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":9,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-28T14:08:41.501380+0000","last_change":"2024-03-28T14:08:41.498351+0000","last_active":"2024-03-28T14:08:41.501380+0000","last_peered":"2024-03-28T14:08:41.501380+0000","last_clean":"2024-03-28T14:08:41.501380+0000","last_became_active":"2024-03-28T14:08:41.497530+0000","last_became_peered":"2024-03-28T14:08:41.497530+0000","last_unstale":"2024-03-28T14:08:41.501380+0000","last_undegraded":"2024-03-28T14:08:41.501380+0000","last_fullsized":"2024-03-28T14:08:41.501380+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:08:40.465288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:08:40.465288+0000","last_clean_scrub_stamp":"2024-03-28T14:08:40.465288+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:50:39.985024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":12,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-28T14:08:41.486476+0000","last_change":"2024-03-28T14:08:40.510590+0000","last_active":"2024-03-28T14:08:41.486476+0000","last_peered":"2024-03-28T14:08:41.486476+0000","last_clean":"2024-03-28T14:08:41.486476+0000","last_became_active":"2024-03-28T14:08:40.509879+0000","last_became_peered":"2024-03-28T14:08:40.509879+0000","last_unstale":"2024-03-28T14:08:41.486476+0000","last_undegraded":"2024-03-28T14:08:41.486476+0000","last_fullsized":"2024-03-28T14:08:41.486476+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:08:39.461488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:08:39.461488+0000","last_clean_scrub_stamp":"2024-03-28T14:08:39.461488+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:54:15.294534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":9,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-28T14:08:41.539118+0000","last_change":"2024-03-28T14:08:41.518804+0000","last_active":"2024-03-28T14:08:41.539118+0000","last_peered":"2024-03-28T14:08:41.539118+0000","last_clean":"2024-03-28T14:08:41.539118+0000","last_became_active":"2024-03-28T14:08:41.517961+0000","last_became_peered":"2024-03-28T14:08:41.517961+0000","last_unstale":"2024-03-28T14:08:41.539118+0000","last_undegraded":"2024-03-28T14:08:41.539118+0000","last_fullsized":"2024-03-28T14:08:41.539118+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:08:40.465288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:08:40.465288+0000","last_clean_scrub_stamp":"2024-03-28T14:08:40.465288+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:19:23.400242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":12,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-28T14:08:41.497880+0000","last_change":"2024-03-28T14:08:40.526487+0000","last_active":"2024-03-28T14:08:41.497880+0000","last_peered":"2024-03-28T14:08:41.497880+0000","last_clean":"2024-03-28T14:08:41.497880+0000","last_became_active":"2024-03-28T14:08:40.525646+0000","last_became_peered":"2024-03-28T14:08:40.525646+0000","last_unstale":"2024-03-28T14:08:41.497880+0000","last_undegraded":"2024-03-28T14:08:41.497880+0000","last_fullsized":"2024-03-28T14:08:41.497880+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:08:39.461488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:08:39.461488+0000","last_clean_scrub_stamp":"2024-03-28T14:08:39.461488+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T18:29:51.673169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":9,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-28T14:08:41.532168+0000","last_change":"2024-03-28T14:08:41.528486+0000","last_active":"2024-03-28T14:08:41.532168+0000","last_peered":"2024-03-28T14:08:41.532168+0000","last_clean":"2024-03-28T14:08:41.532168+0000","last_became_active":"2024-03-28T14:08:41.528180+0000","last_became_peered":"2024-03-28T14:08:41.528180+0000","last_unstale":"2024-03-28T14:08:41.532168+0000","last_undegraded":"2024-03-28T14:08:41.532168+0000","last_fullsized":"2024-03-28T14:08:41.532168+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:08:40.465288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:08:40.465288+0000","last_clean_scrub_stamp":"2024-03-28T14:08:40.465288+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T14:18:16.544200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"29'2","reported_seq":14,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-28T14:08:41.520878+0000","last_change":"2024-03-28T14:08:40.517457+0000","last_active":"2024-03-28T14:08:41.520878+0000","last_peered":"2024-03-28T14:08:41.520878+0000","last_clean":"2024-03-28T14:08:41.520878+0000","last_became_active":"2024-03-28T14:08:40.516812+0000","last_became_peered":"2024-03-28T14:08:40.516812+0000","last_unstale":"2024-03-28T14:08:41.520878+0000","last_undegraded":"2024-03-28T14:08:41.520878+0000","last_fullsized":"2024-03-28T14:08:41.520878+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:08:39.461488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:08:39.461488+0000","last_clean_scrub_stamp":"2024-03-28T14:08:39.461488+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T18:29:51.673169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":9,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-28T14:08:41.543624+0000","last_change":"2024-03-28T14:08:41.531881+0000","last_active":"2024-03-28T14:08:41.543624+0000","last_peered":"2024-03-28T14:08:41.543624+0000","last_clean":"2024-03-28T14:08:41.543624+0000","last_became_active":"2024-03-28T14:08:41.529671+0000","last_became_peered":"2024-03-28T14:08:41.529671+0000","last_unstale":"2024-03-28T14:08:41.543624+0000","last_undegraded":"2024-03-28T14:08:41.543624+0000","last_fullsized":"2024-03-28T14:08:41.543624+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:08:40.465288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:08:40.465288+0000","last_clean_scrub_stamp":"2024-03-28T14:08:40.465288+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T15:46:08.682856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":12,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-28T14:08:41.490689+0000","last_change":"2024-03-28T14:08:40.502674+0000","last_active":"2024-03-28T14:08:41.490689+0000","last_peered":"2024-03-28T14:08:41.490689+0000","last_clean":"2024-03-28T14:08:41.490689+0000","last_became_active":"2024-03-28T14:08:40.502316+0000","last_became_peered":"2024-03-28T14:08:40.502316+0000","last_unstale":"2024-03-28T14:08:41.490689+0000","last_undegraded":"2024-03-28T14:08:41.490689+0000","last_fullsized":"2024-03-28T14:08:41.490689+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:08:39.461488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:08:39.461488+0000","last_clean_scrub_stamp":"2024-03-28T14:08:39.461488+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:59:07.212387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":9,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-28T14:08:41.501593+0000","last_change":"2024-03-28T14:08:41.496026+0000","last_active":"2024-03-28T14:08:41.501593+0000","last_peered":"2024-03-28T14:08:41.501593+0000","last_clean":"2024-03-28T14:08:41.501593+0000","last_became_active":"2024-03-28T14:08:41.495649+0000","last_became_peered":"2024-03-28T14:08:41.495649+0000","last_unstale":"2024-03-28T14:08:41.501593+0000","last_undegraded":"2024-03-28T14:08:41.501593+0000","last_fullsized":"2024-03-28T14:08:41.501593+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:08:40.465288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:08:40.465288+0000","last_clean_scrub_stamp":"2024-03-28T14:08:40.465288+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T17:00:12.873136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":12,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-28T14:08:41.484542+0000","last_change":"2024-03-28T14:08:40.507920+0000","last_active":"2024-03-28T14:08:41.484542+0000","last_peered":"2024-03-28T14:08:41.484542+0000","last_clean":"2024-03-28T14:08:41.484542+0000","last_became_active":"2024-03-28T14:08:40.507479+0000","last_became_peered":"2024-03-28T14:08:40.507479+0000","last_unstale":"2024-03-28T14:08:41.484542+0000","last_undegraded":"2024-03-28T14:08:41.484542+0000","last_fullsized":"2024-03-28T14:08:41.484542+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:08:39.461488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:08:39.461488+0000","last_clean_scrub_stamp":"2024-03-28T14:08:39.461488+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T17:12:46.783085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":9,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-28T14:08:41.524113+0000","last_change":"2024-03-28T14:08:41.511743+0000","last_active":"2024-03-28T14:08:41.524113+0000","last_peered":"2024-03-28T14:08:41.524113+0000","last_clean":"2024-03-28T14:08:41.524113+0000","last_became_active":"2024-03-28T14:08:41.510984+0000","last_became_peered":"2024-03-28T14:08:41.510984+0000","last_unstale":"2024-03-28T14:08:41.524113+0000","last_undegraded":"2024-03-28T14:08:41.524113+0000","last_fullsized":"2024-03-28T14:08:41.524113+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:08:40.465288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:08:40.465288+0000","last_clean_scrub_stamp":"2024-03-28T14:08:40.465288+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:21:18.041106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":12,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-28T14:08:41.504372+0000","last_change":"2024-03-28T14:08:40.504570+0000","last_active":"2024-03-28T14:08:41.504372+0000","last_peered":"2024-03-28T14:08:41.504372+0000","last_clean":"2024-03-28T14:08:41.504372+0000","last_became_active":"2024-03-28T14:08:40.504198+0000","last_became_peered":"2024-03-28T14:08:40.504198+0000","last_unstale":"2024-03-28T14:08:41.504372+0000","last_undegraded":"2024-03-28T14:08:41.504372+0000","last_fullsized":"2024-03-28T14:08:41.504372+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:08:39.461488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:08:39.461488+0000","last_clean_scrub_stamp":"2024-03-28T14:08:39.461488+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T18:29:51.673169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":9,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-28T14:08:41.516693+0000","last_change":"2024-03-28T14:08:41.511129+0000","last_active":"2024-03-28T14:08:41.516693+0000","last_peered":"2024-03-28T14:08:41.516693+0000","last_clean":"2024-03-28T14:08:41.516693+0000","last_became_active":"2024-03-28T14:08:41.510286+0000","last_became_peered":"2024-03-28T14:08:41.510286+0000","last_unstale":"2024-03-28T14:08:41.516693+0000","last_undegraded":"2024-03-28T14:08:41.516693+0000","last_fullsized":"2024-03-28T14:08:41.516693+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:08:40.465288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:08:40.465288+0000","last_clean_scrub_stamp":"2024-03-28T14:08:40.465288+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:21:52.007093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":12,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-28T14:08:41.483610+0000","last_change":"2024-03-28T14:08:40.508674+0000","last_active":"2024-03-28T14:08:41.483610+0000","last_peered":"2024-03-28T14:08:41.483610+0000","last_clean":"2024-03-28T14:08:41.483610+0000","last_became_active":"2024-03-28T14:08:40.508240+0000","last_became_peered":"2024-03-28T14:08:40.508240+0000","last_unstale":"2024-03-28T14:08:41.483610+0000","last_undegraded":"2024-03-28T14:08:41.483610+0000","last_fullsized":"2024-03-28T14:08:41.483610+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:08:39.461488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:08:39.461488+0000","last_clean_scrub_stamp":"2024-03-28T14:08:39.461488+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T17:00:11.869336+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":9,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-28T14:08:41.516984+0000","last_change":"2024-03-28T14:08:41.512590+0000","last_active":"2024-03-28T14:08:41.516984+0000","last_peered":"2024-03-28T14:08:41.516984+0000","last_clean":"2024-03-28T14:08:41.516984+0000","last_became_active":"2024-03-28T14:08:41.512351+0000","last_became_peered":"2024-03-28T14:08:41.512351+0000","last_unstale":"2024-03-28T14:08:41.516984+0000","last_undegraded":"2024-03-28T14:08:41.516984+0000","last_fullsized":"2024-03-28T14:08:41.516984+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:08:40.465288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:08:40.465288+0000","last_clean_scrub_stamp":"2024-03-28T14:08:40.465288+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T17:13:46.853110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":12,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-28T14:08:41.486921+0000","last_change":"2024-03-28T14:08:40.516523+0000","last_active":"2024-03-28T14:08:41.486921+0000","last_peered":"2024-03-28T14:08:41.486921+0000","last_clean":"2024-03-28T14:08:41.486921+0000","last_became_active":"2024-03-28T14:08:40.515289+0000","last_became_peered":"2024-03-28T14:08:40.515289+0000","last_unstale":"2024-03-28T14:08:41.486921+0000","last_undegraded":"2024-03-28T14:08:41.486921+0000","last_fullsized":"2024-03-28T14:08:41.486921+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:08:39.461488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:08:39.461488+0000","last_clean_scrub_stamp":"2024-03-28T14:08:39.461488+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:21:17.037306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":9,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-28T14:08:41.521289+0000","last_change":"2024-03-28T14:08:41.506696+0000","last_active":"2024-03-28T14:08:41.521289+0000","last_peered":"2024-03-28T14:08:41.521289+0000","last_clean":"2024-03-28T14:08:41.521289+0000","last_became_active":"2024-03-28T14:08:41.505946+0000","last_became_peered":"2024-03-28T14:08:41.505946+0000","last_unstale":"2024-03-28T14:08:41.521289+0000","last_undegraded":"2024-03-28T14:08:41.521289+0000","last_fullsized":"2024-03-28T14:08:41.521289+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:08:40.465288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:08:40.465288+0000","last_clean_scrub_stamp":"2024-03-28T14:08:40.465288+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T17:00:12.873136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":12,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-28T14:08:41.497600+0000","last_change":"2024-03-28T14:08:40.525713+0000","last_active":"2024-03-28T14:08:41.497600+0000","last_peered":"2024-03-28T14:08:41.497600+0000","last_clean":"2024-03-28T14:08:41.497600+0000","last_became_active":"2024-03-28T14:08:40.525183+0000","last_became_peered":"2024-03-28T14:08:40.525183+0000","last_unstale":"2024-03-28T14:08:41.497600+0000","last_undegraded":"2024-03-28T14:08:41.497600+0000","last_fullsized":"2024-03-28T14:08:41.497600+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:08:39.461488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:08:39.461488+0000","last_clean_scrub_stamp":"2024-03-28T14:08:39.461488+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T16:51:21.764285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":9,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-28T14:08:41.538482+0000","last_change":"2024-03-28T14:08:41.522760+0000","last_active":"2024-03-28T14:08:41.538482+0000","last_peered":"2024-03-28T14:08:41.538482+0000","last_clean":"2024-03-28T14:08:41.538482+0000","last_became_active":"2024-03-28T14:08:41.522046+0000","last_became_peered":"2024-03-28T14:08:41.522046+0000","last_unstale":"2024-03-28T14:08:41.538482+0000","last_undegraded":"2024-03-28T14:08:41.538482+0000","last_fullsized":"2024-03-28T14:08:41.538482+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:08:40.465288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:08:40.465288+0000","last_clean_scrub_stamp":"2024-03-28T14:08:40.465288+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T18:29:52.676969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":12,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-28T14:08:41.500774+0000","last_change":"2024-03-28T14:08:40.511404+0000","last_active":"2024-03-28T14:08:41.500774+0000","last_peered":"2024-03-28T14:08:41.500774+0000","last_clean":"2024-03-28T14:08:41.500774+0000","last_became_active":"2024-03-28T14:08:40.510675+0000","last_became_peered":"2024-03-28T14:08:40.510675+0000","last_unstale":"2024-03-28T14:08:41.500774+0000","last_undegraded":"2024-03-28T14:08:41.500774+0000","last_fullsized":"2024-03-28T14:08:41.500774+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:08:39.461488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:08:39.461488+0000","last_clean_scrub_stamp":"2024-03-28T14:08:39.461488+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:37:19.550407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":184,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-28T14:08:41.496713+0000","last_change":"2024-03-28T14:07:49.234448+0000","last_active":"2024-03-28T14:08:41.496713+0000","last_peered":"2024-03-28T14:08:41.496713+0000","last_clean":"2024-03-28T14:08:41.496713+0000","last_became_active":"2024-03-28T14:07:49.233895+0000","last_became_peered":"2024-03-28T14:07:49.233895+0000","last_unstale":"2024-03-28T14:08:41.496713+0000","last_undegraded":"2024-03-28T14:08:41.496713+0000","last_fullsized":"2024-03-28T14:08:41.496713+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:07:48.069157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:07:48.069157+0000","last_clean_scrub_stamp":"2024-03-28T14:07:48.069157+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:11:39.977557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574863,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27928,"kb_used_data":900,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343912,"statfs":{"total":96636764160,"available":96608165888,"internally_reserved":0,"allocated":921600,"data_stored":712841,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":11,"apply_latency_ms":11,"commit_latency_ns":11000000,"apply_latency_ns":11000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574864,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27344,"kb_used_data":324,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344496,"statfs":{"total":96636764160,"available":96608763904,"internally_reserved":0,"allocated":331776,"data_stored":122999,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574862,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27344,"kb_used_data":324,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344496,"statfs":{"total":96636764160,"available":96608763904,"internally_reserved":0,"allocated":331776,"data_stored":123055,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":11,"apply_latency_ms":11,"commit_latency_ns":11000000,"apply_latency_ns":11000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574863,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27336,"kb_used_data":316,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344504,"statfs":{"total":96636764160,"available":96608772096,"internally_reserved":0,"allocated":323584,"data_stored":122383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574865,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27352,"kb_used_data":332,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344488,"statfs":{"total":96636764160,"available":96608755712,"internally_reserved":0,"allocated":339968,"data_stored":124017,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574863,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27340,"kb_used_data":320,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344500,"statfs":{"total":96636764160,"available":96608768000,"internally_reserved":0,"allocated":327680,"data_stored":122453,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574861,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27352,"kb_used_data":332,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344488,"statfs":{"total":96636764160,"available":96608755712,"internally_reserved":0,"allocated":339968,"data_stored":124017,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574864,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27924,"kb_used_data":896,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343916,"statfs":{"total":96636764160,"available":96608169984,"internally_reserved":0,"allocated":917504,"data_stored":712751,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-28T14:08:44.565 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-28T14:08:44.715 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:44.710+0000 7f8541403640 1 Processor -- start 2024-03-28T14:08:44.715 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:44.710+0000 7f8541403640 1 -- start start 2024-03-28T14:08:44.717 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:44.714+0000 7f8541403640 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f853c0b5360 0x7f853c0bb7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:44.717 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:44.714+0000 7f8541403640 1 --2- >> v2:172.21.15.171:3300/0 conn(0x7f853c0b71d0 0x7f853c148b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:44.717 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:44.714+0000 7f8541403640 1 --2- >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f853c0b7650 0x7f853c071ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:44.717 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:44.714+0000 7f8541403640 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 -- 0x7f853c0b64c0 con 0x7f853c0b5360 2024-03-28T14:08:44.717 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:44.714+0000 7f8541403640 1 -- --> v2:172.21.15.171:3300/0 -- mon_getmap magic: 0 -- 0x7f853c0b5e70 con 0x7f853c0b71d0 2024-03-28T14:08:44.717 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:44.714+0000 7f8541403640 1 -- --> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] -- mon_getmap magic: 0 -- 0x7f853c0b6150 con 0x7f853c0b7650 2024-03-28T14:08:44.717 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:44.714+0000 7f853affd640 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f853c0b5360 0x7f853c0bb7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:44.717 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:44.714+0000 7f853affd640 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f853c0b5360 0x7f853c0bb7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:50458/0 (socket says 172.21.15.104:50458) 2024-03-28T14:08:44.717 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:44.714+0000 7f853affd640 1 -- 172.21.15.104:0/3508179269 learned_addr learned my addr 172.21.15.104:0/3508179269 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-03-28T14:08:44.718 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:44.714+0000 7f853b7fe640 1 --2- 172.21.15.104:0/3508179269 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f853c0b7650 0x7f853c071ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:44.718 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:44.714+0000 7f853affd640 1 -- 172.21.15.104:0/3508179269 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f853c0b7650 msgr2=0x7f853c071ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T14:08:44.718 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:44.714+0000 7f853affd640 1 --2- 172.21.15.104:0/3508179269 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f853c0b7650 0x7f853c071ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:44.718 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:44.714+0000 7f853affd640 1 -- 172.21.15.104:0/3508179269 >> v2:172.21.15.171:3300/0 conn(0x7f853c0b71d0 msgr2=0x7f853c148b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T14:08:44.718 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:44.714+0000 7f853a7fc640 1 --2- 172.21.15.104:0/3508179269 >> v2:172.21.15.171:3300/0 conn(0x7f853c0b71d0 0x7f853c148b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:44.718 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:44.714+0000 7f853affd640 1 --2- 172.21.15.104:0/3508179269 >> v2:172.21.15.171:3300/0 conn(0x7f853c0b71d0 0x7f853c148b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:44.718 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:44.714+0000 7f853affd640 1 -- 172.21.15.104:0/3508179269 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f853c0b6950 con 0x7f853c0b5360 2024-03-28T14:08:44.718 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:44.714+0000 7f853b7fe640 1 --2- 172.21.15.104:0/3508179269 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f853c0b7650 0x7f853c071ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T14:08:44.720 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:44.714+0000 7f853affd640 1 --2- 172.21.15.104:0/3508179269 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f853c0b5360 0x7f853c0bb7d0 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f853c02d1b0 tx=0x7f852c01c1d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7d556c2a710ccc97 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:08:44.720 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:44.714+0000 7f8539ffb640 1 -- 172.21.15.104:0/3508179269 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f852c0b0070 con 0x7f853c0b5360 2024-03-28T14:08:44.720 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:44.714+0000 7f8539ffb640 1 -- 172.21.15.104:0/3508179269 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f852c0ab600 con 0x7f853c0b5360 2024-03-28T14:08:44.720 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:44.714+0000 7f8539ffb640 1 -- 172.21.15.104:0/3508179269 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f852c0ab9d0 con 0x7f853c0b5360 2024-03-28T14:08:44.720 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:44.714+0000 7f8541403640 1 -- 172.21.15.104:0/3508179269 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f853c0b5360 msgr2=0x7f853c0bb7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:44.720 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:44.714+0000 7f8541403640 1 --2- 172.21.15.104:0/3508179269 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f853c0b5360 0x7f853c0bb7d0 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f853c02d1b0 tx=0x7f852c01c1d0 comp rx=0 tx=0).stop 2024-03-28T14:08:44.720 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:44.714+0000 7f8541403640 1 -- 172.21.15.104:0/3508179269 shutdown_connections 2024-03-28T14:08:44.720 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:44.714+0000 7f8541403640 1 --2- 172.21.15.104:0/3508179269 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f853c0b7650 0x7f853c071ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:44.720 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:44.714+0000 7f8541403640 1 --2- 172.21.15.104:0/3508179269 >> v2:172.21.15.171:3300/0 conn(0x7f853c0b71d0 0x7f853c148b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:44.720 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:44.714+0000 7f8541403640 1 --2- 172.21.15.104:0/3508179269 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f853c0b5360 0x7f853c0bb7d0 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:44.720 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:44.714+0000 7f8541403640 1 -- 172.21.15.104:0/3508179269 >> 172.21.15.104:0/3508179269 conn(0x7f853c02c800 msgr2=0x7f853c0bb0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T14:08:44.720 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:44.714+0000 7f8541403640 1 -- 172.21.15.104:0/3508179269 shutdown_connections 2024-03-28T14:08:44.721 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:44.714+0000 7f8541403640 1 -- 172.21.15.104:0/3508179269 wait complete. 2024-03-28T14:08:44.723 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:44.718+0000 7f8541403640 1 Processor -- start 2024-03-28T14:08:44.723 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:44.718+0000 7f8541403640 1 -- start start 2024-03-28T14:08:44.723 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:44.718+0000 7f8541403640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f853c0b5360 0x7f853c143e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:44.723 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:44.718+0000 7f8541403640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f853c0b71d0 0x7f853c06d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:44.723 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:44.718+0000 7f8541403640 1 --2- >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f853c0b7650 0x7f853c145200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:44.723 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:44.718+0000 7f8541403640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f853c151ec0 con 0x7f853c0b5360 2024-03-28T14:08:44.723 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:44.718+0000 7f8541403640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f853c149270 con 0x7f853c0b71d0 2024-03-28T14:08:44.723 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:44.718+0000 7f8541403640 1 -- --> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] -- mon_getmap magic: 0 -- 0x7f853c152040 con 0x7f853c0b7650 2024-03-28T14:08:44.723 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:44.718+0000 7f853affd640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f853c0b5360 0x7f853c143e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:44.723 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:44.718+0000 7f853affd640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f853c0b5360 0x7f853c143e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:50470/0 (socket says 172.21.15.104:50470) 2024-03-28T14:08:44.723 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:44.718+0000 7f853affd640 1 -- 172.21.15.104:0/1637247139 learned_addr learned my addr 172.21.15.104:0/1637247139 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-03-28T14:08:44.724 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:44.718+0000 7f853b7fe640 1 --2- 172.21.15.104:0/1637247139 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f853c0b7650 0x7f853c145200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:44.724 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:44.718+0000 7f853a7fc640 1 --2- 172.21.15.104:0/1637247139 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f853c0b71d0 0x7f853c06d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:44.724 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:44.718+0000 7f853affd640 1 -- 172.21.15.104:0/1637247139 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f853c0b7650 msgr2=0x7f853c145200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:44.724 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:44.718+0000 7f853affd640 1 --2- 172.21.15.104:0/1637247139 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f853c0b7650 0x7f853c145200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:44.724 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:44.718+0000 7f853affd640 1 -- 172.21.15.104:0/1637247139 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f853c0b71d0 msgr2=0x7f853c06d190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:44.724 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:44.718+0000 7f853affd640 1 --2- 172.21.15.104:0/1637247139 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f853c0b71d0 0x7f853c06d190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:44.724 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:44.718+0000 7f853affd640 1 -- 172.21.15.104:0/1637247139 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f853c06e470 con 0x7f853c0b5360 2024-03-28T14:08:44.724 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:44.718+0000 7f853affd640 1 --2- 172.21.15.104:0/1637247139 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f853c0b5360 0x7f853c143e60 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f852c0ae040 tx=0x7f852c018f70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:08:44.724 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:44.718+0000 7f85237fe640 1 -- 172.21.15.104:0/1637247139 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f852c0b0040 con 0x7f853c0b5360 2024-03-28T14:08:44.724 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:44.718+0000 7f85237fe640 1 -- 172.21.15.104:0/1637247139 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f852c0b4640 con 0x7f853c0b5360 2024-03-28T14:08:44.724 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:44.718+0000 7f8541403640 1 -- 172.21.15.104:0/1637247139 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f853c1443a0 con 0x7f853c0b5360 2024-03-28T14:08:44.724 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:44.718+0000 7f85237fe640 1 -- 172.21.15.104:0/1637247139 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f852c0b4a30 con 0x7f853c0b5360 2024-03-28T14:08:44.724 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:44.718+0000 7f8541403640 1 -- 172.21.15.104:0/1637247139 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f853c144620 con 0x7f853c0b5360 2024-03-28T14:08:44.724 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:44.718+0000 7f8541403640 1 -- 172.21.15.104:0/1637247139 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f853c14faa0 con 0x7f853c0b5360 2024-03-28T14:08:44.725 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:44.718+0000 7f85237fe640 1 -- 172.21.15.104:0/1637247139 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) ==== 87803+0+0 (secure 0 0 0) 0x7f852c0ab4c0 con 0x7f853c0b5360 2024-03-28T14:08:44.725 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:44.722+0000 7f85237fe640 1 --2- 172.21.15.104:0/1637247139 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f8528067fc0 0x7f852806a480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:44.725 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:44.722+0000 7f85237fe640 1 -- 172.21.15.104:0/1637247139 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 5709+0+0 (secure 0 0 0) 0x7f852c120250 con 0x7f853c0b5360 2024-03-28T14:08:44.726 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:44.722+0000 7f853a7fc640 1 --2- 172.21.15.104:0/1637247139 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f8528067fc0 0x7f852806a480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:44.726 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:44.722+0000 7f853a7fc640 1 --2- 172.21.15.104:0/1637247139 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f8528067fc0 0x7f852806a480 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f8530006fd0 tx=0x7f8530008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:08:44.728 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:44.726+0000 7f85237fe640 1 -- 172.21.15.104:0/1637247139 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179117 (secure 0 0 0) 0x7f853c14faa0 con 0x7f853c0b5360 2024-03-28T14:08:44.875 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:44.870+0000 7f8541403640 1 -- 172.21.15.104:0/1637247139 --> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f853c0687c0 con 0x7f8528067fc0 2024-03-28T14:08:44.885 INFO:tasks.ceph.mds.a.smithi104.stderr:2024-03-28T14:08:44.882+0000 7fd6f06a8640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-28T14:08:44.893 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:44.890+0000 7f85237fe640 1 -- 172.21.15.104:0/1637247139 <== mgr.4100 v2:172.21.15.171:6832/1522752412 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+251919 (secure 0 0 0) 0x7f853c0687c0 con 0x7f8528067fc0 2024-03-28T14:08:44.895 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-03-28T14:08:44.897 INFO:teuthology.orchestra.run.smithi104.stderr:dumped all 2024-03-28T14:08:44.898 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:44.894+0000 7f8541403640 1 -- 172.21.15.104:0/1637247139 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f8528067fc0 msgr2=0x7f852806a480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:44.898 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:44.894+0000 7f8541403640 1 --2- 172.21.15.104:0/1637247139 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f8528067fc0 0x7f852806a480 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f8530006fd0 tx=0x7f8530008040 comp rx=0 tx=0).stop 2024-03-28T14:08:44.898 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:44.894+0000 7f8541403640 1 -- 172.21.15.104:0/1637247139 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f853c0b5360 msgr2=0x7f853c143e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:44.898 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:44.894+0000 7f8541403640 1 --2- 172.21.15.104:0/1637247139 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f853c0b5360 0x7f853c143e60 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f852c0ae040 tx=0x7f852c018f70 comp rx=0 tx=0).stop 2024-03-28T14:08:44.898 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:44.894+0000 7f8541403640 1 -- 172.21.15.104:0/1637247139 shutdown_connections 2024-03-28T14:08:44.898 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:44.894+0000 7f8541403640 1 --2- 172.21.15.104:0/1637247139 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f853c0b7650 0x7f853c145200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:44.898 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:44.894+0000 7f8541403640 1 --2- 172.21.15.104:0/1637247139 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f853c0b71d0 0x7f853c06d190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:44.898 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:44.894+0000 7f8541403640 1 --2- 172.21.15.104:0/1637247139 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f8528067fc0 0x7f852806a480 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:44.898 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:44.894+0000 7f8541403640 1 --2- 172.21.15.104:0/1637247139 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f853c0b5360 0x7f853c143e60 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:44.898 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:44.894+0000 7f8541403640 1 -- 172.21.15.104:0/1637247139 >> 172.21.15.104:0/1637247139 conn(0x7f853c02c800 msgr2=0x7f853c0b9b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T14:08:44.899 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:44.894+0000 7f8541403640 1 -- 172.21.15.104:0/1637247139 shutdown_connections 2024-03-28T14:08:44.899 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:44.894+0000 7f8541403640 1 -- 172.21.15.104:0/1637247139 wait complete. 2024-03-28T14:08:44.912 INFO:teuthology.orchestra.run.smithi104.stdout:{"pg_ready":true,"pg_map":{"version":52,"stamp":"2024-03-28T14:08:43.592591+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":203,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":83,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219920,"kb_used_data":3744,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754754800,"statfs":{"total":773094113280,"available":772868915200,"internally_reserved":0,"allocated":3833856,"data_stored":2164516,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":221236785},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":51,"apply_latency_ms":51,"commit_latency_ns":51000000,"apply_latency_ns":51000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.125403"},"pg_stats":[{"pgid":"5.3f","version":"0'0","reported_seq":9,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-28T14:08:41.535396+0000","last_change":"2024-03-28T14:08:41.533604+0000","last_active":"2024-03-28T14:08:41.535396+0000","last_peered":"2024-03-28T14:08:41.535396+0000","last_clean":"2024-03-28T14:08:41.535396+0000","last_became_active":"2024-03-28T14:08:41.532934+0000","last_became_peered":"2024-03-28T14:08:41.532934+0000","last_unstale":"2024-03-28T14:08:41.535396+0000","last_undegraded":"2024-03-28T14:08:41.535396+0000","last_fullsized":"2024-03-28T14:08:41.535396+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:08:40.465288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:08:40.465288+0000","last_clean_scrub_stamp":"2024-03-28T14:08:40.465288+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:59:08.216187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.3e","version":"0'0","reported_seq":9,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-28T14:08:41.509903+0000","last_change":"2024-03-28T14:08:41.504770+0000","last_active":"2024-03-28T14:08:41.509903+0000","last_peered":"2024-03-28T14:08:41.509903+0000","last_clean":"2024-03-28T14:08:41.509903+0000","last_became_active":"2024-03-28T14:08:41.504477+0000","last_became_peered":"2024-03-28T14:08:41.504477+0000","last_unstale":"2024-03-28T14:08:41.509903+0000","last_undegraded":"2024-03-28T14:08:41.509903+0000","last_fullsized":"2024-03-28T14:08:41.509903+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:08:40.465288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:08:40.465288+0000","last_clean_scrub_stamp":"2024-03-28T14:08:40.465288+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:56:54.080557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.3d","version":"0'0","reported_seq":9,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-28T14:08:41.542038+0000","last_change":"2024-03-28T14:08:41.528015+0000","last_active":"2024-03-28T14:08:41.542038+0000","last_peered":"2024-03-28T14:08:41.542038+0000","last_clean":"2024-03-28T14:08:41.542038+0000","last_became_active":"2024-03-28T14:08:41.526802+0000","last_became_peered":"2024-03-28T14:08:41.526802+0000","last_unstale":"2024-03-28T14:08:41.542038+0000","last_undegraded":"2024-03-28T14:08:41.542038+0000","last_fullsized":"2024-03-28T14:08:41.542038+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:08:40.465288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:08:40.465288+0000","last_clean_scrub_stamp":"2024-03-28T14:08:40.465288+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T16:19:05.943365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3c","version":"0'0","reported_seq":9,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-28T14:08:41.514524+0000","last_change":"2024-03-28T14:08:41.505145+0000","last_active":"2024-03-28T14:08:41.514524+0000","last_peered":"2024-03-28T14:08:41.514524+0000","last_clean":"2024-03-28T14:08:41.514524+0000","last_became_active":"2024-03-28T14:08:41.504901+0000","last_became_peered":"2024-03-28T14:08:41.504901+0000","last_unstale":"2024-03-28T14:08:41.514524+0000","last_undegraded":"2024-03-28T14:08:41.514524+0000","last_fullsized":"2024-03-28T14:08:41.514524+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:08:40.465288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:08:40.465288+0000","last_clean_scrub_stamp":"2024-03-28T14:08:40.465288+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T15:29:19.342685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3b","version":"0'0","reported_seq":9,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-28T14:08:41.505644+0000","last_change":"2024-03-28T14:08:41.503669+0000","last_active":"2024-03-28T14:08:41.505644+0000","last_peered":"2024-03-28T14:08:41.505644+0000","last_clean":"2024-03-28T14:08:41.505644+0000","last_became_active":"2024-03-28T14:08:41.503273+0000","last_became_peered":"2024-03-28T14:08:41.503273+0000","last_unstale":"2024-03-28T14:08:41.505644+0000","last_undegraded":"2024-03-28T14:08:41.505644+0000","last_fullsized":"2024-03-28T14:08:41.505644+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:08:40.465288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:08:40.465288+0000","last_clean_scrub_stamp":"2024-03-28T14:08:40.465288+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:35:29.396001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.3a","version":"0'0","reported_seq":9,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-28T14:08:41.532474+0000","last_change":"2024-03-28T14:08:41.529164+0000","last_active":"2024-03-28T14:08:41.532474+0000","last_peered":"2024-03-28T14:08:41.532474+0000","last_clean":"2024-03-28T14:08:41.532474+0000","last_became_active":"2024-03-28T14:08:41.528845+0000","last_became_peered":"2024-03-28T14:08:41.528845+0000","last_unstale":"2024-03-28T14:08:41.532474+0000","last_undegraded":"2024-03-28T14:08:41.532474+0000","last_fullsized":"2024-03-28T14:08:41.532474+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:08:40.465288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:08:40.465288+0000","last_clean_scrub_stamp":"2024-03-28T14:08:40.465288+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:49:54.840279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.39","version":"0'0","reported_seq":9,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-28T14:08:41.539164+0000","last_change":"2024-03-28T14:08:41.524620+0000","last_active":"2024-03-28T14:08:41.539164+0000","last_peered":"2024-03-28T14:08:41.539164+0000","last_clean":"2024-03-28T14:08:41.539164+0000","last_became_active":"2024-03-28T14:08:41.522895+0000","last_became_peered":"2024-03-28T14:08:41.522895+0000","last_unstale":"2024-03-28T14:08:41.539164+0000","last_undegraded":"2024-03-28T14:08:41.539164+0000","last_fullsized":"2024-03-28T14:08:41.539164+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:08:40.465288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:08:40.465288+0000","last_clean_scrub_stamp":"2024-03-28T14:08:40.465288+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T16:19:05.943365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.38","version":"0'0","reported_seq":9,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-28T14:08:41.540347+0000","last_change":"2024-03-28T14:08:41.517882+0000","last_active":"2024-03-28T14:08:41.540347+0000","last_peered":"2024-03-28T14:08:41.540347+0000","last_clean":"2024-03-28T14:08:41.540347+0000","last_became_active":"2024-03-28T14:08:41.517145+0000","last_became_peered":"2024-03-28T14:08:41.517145+0000","last_unstale":"2024-03-28T14:08:41.540347+0000","last_undegraded":"2024-03-28T14:08:41.540347+0000","last_fullsized":"2024-03-28T14:08:41.540347+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:08:40.465288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:08:40.465288+0000","last_clean_scrub_stamp":"2024-03-28T14:08:40.465288+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T16:51:22.768085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.37","version":"0'0","reported_seq":9,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-28T14:08:41.513044+0000","last_change":"2024-03-28T14:08:41.491910+0000","last_active":"2024-03-28T14:08:41.513044+0000","last_peered":"2024-03-28T14:08:41.513044+0000","last_clean":"2024-03-28T14:08:41.513044+0000","last_became_active":"2024-03-28T14:08:41.491064+0000","last_became_peered":"2024-03-28T14:08:41.491064+0000","last_unstale":"2024-03-28T14:08:41.513044+0000","last_undegraded":"2024-03-28T14:08:41.513044+0000","last_fullsized":"2024-03-28T14:08:41.513044+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:08:40.465288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:08:40.465288+0000","last_clean_scrub_stamp":"2024-03-28T14:08:40.465288+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:59:08.216187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.36","version":"0'0","reported_seq":9,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-28T14:08:41.514114+0000","last_change":"2024-03-28T14:08:41.507695+0000","last_active":"2024-03-28T14:08:41.514114+0000","last_peered":"2024-03-28T14:08:41.514114+0000","last_clean":"2024-03-28T14:08:41.514114+0000","last_became_active":"2024-03-28T14:08:41.507228+0000","last_became_peered":"2024-03-28T14:08:41.507228+0000","last_unstale":"2024-03-28T14:08:41.514114+0000","last_undegraded":"2024-03-28T14:08:41.514114+0000","last_fullsized":"2024-03-28T14:08:41.514114+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:08:40.465288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:08:40.465288+0000","last_clean_scrub_stamp":"2024-03-28T14:08:40.465288+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:36:46.949506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.35","version":"0'0","reported_seq":9,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-28T14:08:41.538674+0000","last_change":"2024-03-28T14:08:41.535048+0000","last_active":"2024-03-28T14:08:41.538674+0000","last_peered":"2024-03-28T14:08:41.538674+0000","last_clean":"2024-03-28T14:08:41.538674+0000","last_became_active":"2024-03-28T14:08:41.534360+0000","last_became_peered":"2024-03-28T14:08:41.534360+0000","last_unstale":"2024-03-28T14:08:41.538674+0000","last_undegraded":"2024-03-28T14:08:41.538674+0000","last_fullsized":"2024-03-28T14:08:41.538674+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:08:40.465288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:08:40.465288+0000","last_clean_scrub_stamp":"2024-03-28T14:08:40.465288+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:36:46.949506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.34","version":"0'0","reported_seq":9,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-28T14:08:41.543162+0000","last_change":"2024-03-28T14:08:41.531326+0000","last_active":"2024-03-28T14:08:41.543162+0000","last_peered":"2024-03-28T14:08:41.543162+0000","last_clean":"2024-03-28T14:08:41.543162+0000","last_became_active":"2024-03-28T14:08:41.530414+0000","last_became_peered":"2024-03-28T14:08:41.530414+0000","last_unstale":"2024-03-28T14:08:41.543162+0000","last_undegraded":"2024-03-28T14:08:41.543162+0000","last_fullsized":"2024-03-28T14:08:41.543162+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:08:40.465288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:08:40.465288+0000","last_clean_scrub_stamp":"2024-03-28T14:08:40.465288+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T16:14:54.560822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.33","version":"0'0","reported_seq":9,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-28T14:08:41.508551+0000","last_change":"2024-03-28T14:08:41.507821+0000","last_active":"2024-03-28T14:08:41.508551+0000","last_peered":"2024-03-28T14:08:41.508551+0000","last_clean":"2024-03-28T14:08:41.508551+0000","last_became_active":"2024-03-28T14:08:41.507485+0000","last_became_peered":"2024-03-28T14:08:41.507485+0000","last_unstale":"2024-03-28T14:08:41.508551+0000","last_undegraded":"2024-03-28T14:08:41.508551+0000","last_fullsized":"2024-03-28T14:08:41.508551+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:08:40.465288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:08:40.465288+0000","last_clean_scrub_stamp":"2024-03-28T14:08:40.465288+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:03:29.841049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.32","version":"0'0","reported_seq":9,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-28T14:08:41.509568+0000","last_change":"2024-03-28T14:08:41.503262+0000","last_active":"2024-03-28T14:08:41.509568+0000","last_peered":"2024-03-28T14:08:41.509568+0000","last_clean":"2024-03-28T14:08:41.509568+0000","last_became_active":"2024-03-28T14:08:41.502877+0000","last_became_peered":"2024-03-28T14:08:41.502877+0000","last_unstale":"2024-03-28T14:08:41.509568+0000","last_undegraded":"2024-03-28T14:08:41.509568+0000","last_fullsized":"2024-03-28T14:08:41.509568+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:08:40.465288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:08:40.465288+0000","last_clean_scrub_stamp":"2024-03-28T14:08:40.465288+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T14:33:01.059177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.31","version":"0'0","reported_seq":9,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-28T14:08:41.543811+0000","last_change":"2024-03-28T14:08:41.529966+0000","last_active":"2024-03-28T14:08:41.543811+0000","last_peered":"2024-03-28T14:08:41.543811+0000","last_clean":"2024-03-28T14:08:41.543811+0000","last_became_active":"2024-03-28T14:08:41.529034+0000","last_became_peered":"2024-03-28T14:08:41.529034+0000","last_unstale":"2024-03-28T14:08:41.543811+0000","last_undegraded":"2024-03-28T14:08:41.543811+0000","last_fullsized":"2024-03-28T14:08:41.543811+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:08:40.465288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:08:40.465288+0000","last_clean_scrub_stamp":"2024-03-28T14:08:40.465288+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:02:48.110646+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.30","version":"0'0","reported_seq":9,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-28T14:08:41.540304+0000","last_change":"2024-03-28T14:08:41.524413+0000","last_active":"2024-03-28T14:08:41.540304+0000","last_peered":"2024-03-28T14:08:41.540304+0000","last_clean":"2024-03-28T14:08:41.540304+0000","last_became_active":"2024-03-28T14:08:41.517290+0000","last_became_peered":"2024-03-28T14:08:41.517290+0000","last_unstale":"2024-03-28T14:08:41.540304+0000","last_undegraded":"2024-03-28T14:08:41.540304+0000","last_fullsized":"2024-03-28T14:08:41.540304+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:08:40.465288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:08:40.465288+0000","last_clean_scrub_stamp":"2024-03-28T14:08:40.465288+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T15:29:19.342685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2f","version":"0'0","reported_seq":9,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-28T14:08:41.538613+0000","last_change":"2024-03-28T14:08:41.523393+0000","last_active":"2024-03-28T14:08:41.538613+0000","last_peered":"2024-03-28T14:08:41.538613+0000","last_clean":"2024-03-28T14:08:41.538613+0000","last_became_active":"2024-03-28T14:08:41.522278+0000","last_became_peered":"2024-03-28T14:08:41.522278+0000","last_unstale":"2024-03-28T14:08:41.538613+0000","last_undegraded":"2024-03-28T14:08:41.538613+0000","last_fullsized":"2024-03-28T14:08:41.538613+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:08:40.465288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:08:40.465288+0000","last_clean_scrub_stamp":"2024-03-28T14:08:40.465288+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T17:12:47.786885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2e","version":"0'0","reported_seq":9,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-28T14:08:41.516902+0000","last_change":"2024-03-28T14:08:41.512212+0000","last_active":"2024-03-28T14:08:41.516902+0000","last_peered":"2024-03-28T14:08:41.516902+0000","last_clean":"2024-03-28T14:08:41.516902+0000","last_became_active":"2024-03-28T14:08:41.511884+0000","last_became_peered":"2024-03-28T14:08:41.511884+0000","last_unstale":"2024-03-28T14:08:41.516902+0000","last_undegraded":"2024-03-28T14:08:41.516902+0000","last_fullsized":"2024-03-28T14:08:41.516902+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:08:40.465288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:08:40.465288+0000","last_clean_scrub_stamp":"2024-03-28T14:08:40.465288+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:27:50.434958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2d","version":"0'0","reported_seq":9,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-28T14:08:41.516970+0000","last_change":"2024-03-28T14:08:41.512567+0000","last_active":"2024-03-28T14:08:41.516970+0000","last_peered":"2024-03-28T14:08:41.516970+0000","last_clean":"2024-03-28T14:08:41.516970+0000","last_became_active":"2024-03-28T14:08:41.512173+0000","last_became_peered":"2024-03-28T14:08:41.512173+0000","last_unstale":"2024-03-28T14:08:41.516970+0000","last_undegraded":"2024-03-28T14:08:41.516970+0000","last_fullsized":"2024-03-28T14:08:41.516970+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:08:40.465288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:08:40.465288+0000","last_clean_scrub_stamp":"2024-03-28T14:08:40.465288+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:04:24.834273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2c","version":"0'0","reported_seq":9,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-28T14:08:41.535125+0000","last_change":"2024-03-28T14:08:41.530715+0000","last_active":"2024-03-28T14:08:41.535125+0000","last_peered":"2024-03-28T14:08:41.535125+0000","last_clean":"2024-03-28T14:08:41.535125+0000","last_became_active":"2024-03-28T14:08:41.530416+0000","last_became_peered":"2024-03-28T14:08:41.530416+0000","last_unstale":"2024-03-28T14:08:41.535125+0000","last_undegraded":"2024-03-28T14:08:41.535125+0000","last_fullsized":"2024-03-28T14:08:41.535125+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:08:40.465288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:08:40.465288+0000","last_clean_scrub_stamp":"2024-03-28T14:08:40.465288+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:27:13.776391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.2b","version":"0'0","reported_seq":9,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-28T14:08:41.535767+0000","last_change":"2024-03-28T14:08:41.522747+0000","last_active":"2024-03-28T14:08:41.535767+0000","last_peered":"2024-03-28T14:08:41.535767+0000","last_clean":"2024-03-28T14:08:41.535767+0000","last_became_active":"2024-03-28T14:08:41.520721+0000","last_became_peered":"2024-03-28T14:08:41.520721+0000","last_unstale":"2024-03-28T14:08:41.535767+0000","last_undegraded":"2024-03-28T14:08:41.535767+0000","last_fullsized":"2024-03-28T14:08:41.535767+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:08:40.465288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:08:40.465288+0000","last_clean_scrub_stamp":"2024-03-28T14:08:40.465288+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T17:12:47.786885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.2a","version":"0'0","reported_seq":9,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-28T14:08:41.530823+0000","last_change":"2024-03-28T14:08:41.520125+0000","last_active":"2024-03-28T14:08:41.530823+0000","last_peered":"2024-03-28T14:08:41.530823+0000","last_clean":"2024-03-28T14:08:41.530823+0000","last_became_active":"2024-03-28T14:08:41.519384+0000","last_became_peered":"2024-03-28T14:08:41.519384+0000","last_unstale":"2024-03-28T14:08:41.530823+0000","last_undegraded":"2024-03-28T14:08:41.530823+0000","last_fullsized":"2024-03-28T14:08:41.530823+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:08:40.465288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:08:40.465288+0000","last_clean_scrub_stamp":"2024-03-28T14:08:40.465288+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T18:29:52.676969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.29","version":"0'0","reported_seq":9,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-28T14:08:41.519976+0000","last_change":"2024-03-28T14:08:41.517020+0000","last_active":"2024-03-28T14:08:41.519976+0000","last_peered":"2024-03-28T14:08:41.519976+0000","last_clean":"2024-03-28T14:08:41.519976+0000","last_became_active":"2024-03-28T14:08:41.516686+0000","last_became_peered":"2024-03-28T14:08:41.516686+0000","last_unstale":"2024-03-28T14:08:41.519976+0000","last_undegraded":"2024-03-28T14:08:41.519976+0000","last_fullsized":"2024-03-28T14:08:41.519976+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:08:40.465288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:08:40.465288+0000","last_clean_scrub_stamp":"2024-03-28T14:08:40.465288+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T16:19:05.943365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.28","version":"0'0","reported_seq":9,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-28T14:08:41.541762+0000","last_change":"2024-03-28T14:08:41.527726+0000","last_active":"2024-03-28T14:08:41.541762+0000","last_peered":"2024-03-28T14:08:41.541762+0000","last_clean":"2024-03-28T14:08:41.541762+0000","last_became_active":"2024-03-28T14:08:41.526721+0000","last_became_peered":"2024-03-28T14:08:41.526721+0000","last_unstale":"2024-03-28T14:08:41.541762+0000","last_undegraded":"2024-03-28T14:08:41.541762+0000","last_fullsized":"2024-03-28T14:08:41.541762+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:08:40.465288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:08:40.465288+0000","last_clean_scrub_stamp":"2024-03-28T14:08:40.465288+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:21:52.007093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.27","version":"0'0","reported_seq":9,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-28T14:08:41.514117+0000","last_change":"2024-03-28T14:08:41.504761+0000","last_active":"2024-03-28T14:08:41.514117+0000","last_peered":"2024-03-28T14:08:41.514117+0000","last_clean":"2024-03-28T14:08:41.514117+0000","last_became_active":"2024-03-28T14:08:41.504535+0000","last_became_peered":"2024-03-28T14:08:41.504535+0000","last_unstale":"2024-03-28T14:08:41.514117+0000","last_undegraded":"2024-03-28T14:08:41.514117+0000","last_fullsized":"2024-03-28T14:08:41.514117+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:08:40.465288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:08:40.465288+0000","last_clean_scrub_stamp":"2024-03-28T14:08:40.465288+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T16:51:22.768085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.26","version":"0'0","reported_seq":9,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-28T14:08:41.538261+0000","last_change":"2024-03-28T14:08:41.534067+0000","last_active":"2024-03-28T14:08:41.538261+0000","last_peered":"2024-03-28T14:08:41.538261+0000","last_clean":"2024-03-28T14:08:41.538261+0000","last_became_active":"2024-03-28T14:08:41.533740+0000","last_became_peered":"2024-03-28T14:08:41.533740+0000","last_unstale":"2024-03-28T14:08:41.538261+0000","last_undegraded":"2024-03-28T14:08:41.538261+0000","last_fullsized":"2024-03-28T14:08:41.538261+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:08:40.465288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:08:40.465288+0000","last_clean_scrub_stamp":"2024-03-28T14:08:40.465288+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T15:52:40.770832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.25","version":"0'0","reported_seq":9,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-28T14:08:41.510860+0000","last_change":"2024-03-28T14:08:41.498869+0000","last_active":"2024-03-28T14:08:41.510860+0000","last_peered":"2024-03-28T14:08:41.510860+0000","last_clean":"2024-03-28T14:08:41.510860+0000","last_became_active":"2024-03-28T14:08:41.498422+0000","last_became_peered":"2024-03-28T14:08:41.498422+0000","last_unstale":"2024-03-28T14:08:41.510860+0000","last_undegraded":"2024-03-28T14:08:41.510860+0000","last_fullsized":"2024-03-28T14:08:41.510860+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:08:40.465288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:08:40.465288+0000","last_clean_scrub_stamp":"2024-03-28T14:08:40.465288+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T16:19:05.943365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.24","version":"0'0","reported_seq":9,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-28T14:08:41.543119+0000","last_change":"2024-03-28T14:08:41.530599+0000","last_active":"2024-03-28T14:08:41.543119+0000","last_peered":"2024-03-28T14:08:41.543119+0000","last_clean":"2024-03-28T14:08:41.543119+0000","last_became_active":"2024-03-28T14:08:41.529682+0000","last_became_peered":"2024-03-28T14:08:41.529682+0000","last_unstale":"2024-03-28T14:08:41.543119+0000","last_undegraded":"2024-03-28T14:08:41.543119+0000","last_fullsized":"2024-03-28T14:08:41.543119+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:08:40.465288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:08:40.465288+0000","last_clean_scrub_stamp":"2024-03-28T14:08:40.465288+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:30:28.443913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.23","version":"0'0","reported_seq":9,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-28T14:08:41.536030+0000","last_change":"2024-03-28T14:08:41.524450+0000","last_active":"2024-03-28T14:08:41.536030+0000","last_peered":"2024-03-28T14:08:41.536030+0000","last_clean":"2024-03-28T14:08:41.536030+0000","last_became_active":"2024-03-28T14:08:41.522044+0000","last_became_peered":"2024-03-28T14:08:41.522044+0000","last_unstale":"2024-03-28T14:08:41.536030+0000","last_undegraded":"2024-03-28T14:08:41.536030+0000","last_fullsized":"2024-03-28T14:08:41.536030+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:08:40.465288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:08:40.465288+0000","last_clean_scrub_stamp":"2024-03-28T14:08:40.465288+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:21:52.007093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.22","version":"0'0","reported_seq":9,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-28T14:08:41.538668+0000","last_change":"2024-03-28T14:08:41.524549+0000","last_active":"2024-03-28T14:08:41.538668+0000","last_peered":"2024-03-28T14:08:41.538668+0000","last_clean":"2024-03-28T14:08:41.538668+0000","last_became_active":"2024-03-28T14:08:41.523781+0000","last_became_peered":"2024-03-28T14:08:41.523781+0000","last_unstale":"2024-03-28T14:08:41.538668+0000","last_undegraded":"2024-03-28T14:08:41.538668+0000","last_fullsized":"2024-03-28T14:08:41.538668+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:08:40.465288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:08:40.465288+0000","last_clean_scrub_stamp":"2024-03-28T14:08:40.465288+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:21:52.007093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.21","version":"0'0","reported_seq":9,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-28T14:08:41.518176+0000","last_change":"2024-03-28T14:08:41.513458+0000","last_active":"2024-03-28T14:08:41.518176+0000","last_peered":"2024-03-28T14:08:41.518176+0000","last_clean":"2024-03-28T14:08:41.518176+0000","last_became_active":"2024-03-28T14:08:41.512291+0000","last_became_peered":"2024-03-28T14:08:41.512291+0000","last_unstale":"2024-03-28T14:08:41.518176+0000","last_undegraded":"2024-03-28T14:08:41.518176+0000","last_fullsized":"2024-03-28T14:08:41.518176+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:08:40.465288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:08:40.465288+0000","last_clean_scrub_stamp":"2024-03-28T14:08:40.465288+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:32:42.783009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.20","version":"0'0","reported_seq":9,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-28T14:08:41.546365+0000","last_change":"2024-03-28T14:08:41.539577+0000","last_active":"2024-03-28T14:08:41.546365+0000","last_peered":"2024-03-28T14:08:41.546365+0000","last_clean":"2024-03-28T14:08:41.546365+0000","last_became_active":"2024-03-28T14:08:41.538345+0000","last_became_peered":"2024-03-28T14:08:41.538345+0000","last_unstale":"2024-03-28T14:08:41.546365+0000","last_undegraded":"2024-03-28T14:08:41.546365+0000","last_fullsized":"2024-03-28T14:08:41.546365+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:08:40.465288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:08:40.465288+0000","last_clean_scrub_stamp":"2024-03-28T14:08:40.465288+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:02:48.110646+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":9,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-28T14:08:41.509819+0000","last_change":"2024-03-28T14:08:41.507115+0000","last_active":"2024-03-28T14:08:41.509819+0000","last_peered":"2024-03-28T14:08:41.509819+0000","last_clean":"2024-03-28T14:08:41.509819+0000","last_became_active":"2024-03-28T14:08:41.506763+0000","last_became_peered":"2024-03-28T14:08:41.506763+0000","last_unstale":"2024-03-28T14:08:41.509819+0000","last_undegraded":"2024-03-28T14:08:41.509819+0000","last_fullsized":"2024-03-28T14:08:41.509819+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:08:40.465288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:08:40.465288+0000","last_clean_scrub_stamp":"2024-03-28T14:08:40.465288+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T17:12:47.786885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1f","version":"29'4","reported_seq":16,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-28T14:08:41.542766+0000","last_change":"2024-03-28T14:08:40.509298+0000","last_active":"2024-03-28T14:08:41.542766+0000","last_peered":"2024-03-28T14:08:41.542766+0000","last_clean":"2024-03-28T14:08:41.542766+0000","last_became_active":"2024-03-28T14:08:40.508347+0000","last_became_peered":"2024-03-28T14:08:40.508347+0000","last_unstale":"2024-03-28T14:08:41.542766+0000","last_undegraded":"2024-03-28T14:08:41.542766+0000","last_fullsized":"2024-03-28T14:08:41.542766+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:08:39.461488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:08:39.461488+0000","last_clean_scrub_stamp":"2024-03-28T14:08:39.461488+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:27:15.765153+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1f","version":"0'0","reported_seq":9,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-28T14:08:41.545858+0000","last_change":"2024-03-28T14:08:41.543753+0000","last_active":"2024-03-28T14:08:41.545858+0000","last_peered":"2024-03-28T14:08:41.545858+0000","last_clean":"2024-03-28T14:08:41.545858+0000","last_became_active":"2024-03-28T14:08:41.543201+0000","last_became_peered":"2024-03-28T14:08:41.543201+0000","last_unstale":"2024-03-28T14:08:41.545858+0000","last_undegraded":"2024-03-28T14:08:41.545858+0000","last_fullsized":"2024-03-28T14:08:41.545858+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:08:40.465288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:08:40.465288+0000","last_clean_scrub_stamp":"2024-03-28T14:08:40.465288+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T18:29:52.676969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1e","version":"29'1","reported_seq":13,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-28T14:08:41.526613+0000","last_change":"2024-03-28T14:08:40.522075+0000","last_active":"2024-03-28T14:08:41.526613+0000","last_peered":"2024-03-28T14:08:41.526613+0000","last_clean":"2024-03-28T14:08:41.526613+0000","last_became_active":"2024-03-28T14:08:40.521636+0000","last_became_peered":"2024-03-28T14:08:40.521636+0000","last_unstale":"2024-03-28T14:08:41.526613+0000","last_undegraded":"2024-03-28T14:08:41.526613+0000","last_fullsized":"2024-03-28T14:08:41.526613+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:08:39.461488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:08:39.461488+0000","last_clean_scrub_stamp":"2024-03-28T14:08:39.461488+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:21:17.037306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":9,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-28T14:08:41.521201+0000","last_change":"2024-03-28T14:08:41.518008+0000","last_active":"2024-03-28T14:08:41.521201+0000","last_peered":"2024-03-28T14:08:41.521201+0000","last_clean":"2024-03-28T14:08:41.521201+0000","last_became_active":"2024-03-28T14:08:41.516598+0000","last_became_peered":"2024-03-28T14:08:41.516598+0000","last_unstale":"2024-03-28T14:08:41.521201+0000","last_undegraded":"2024-03-28T14:08:41.521201+0000","last_fullsized":"2024-03-28T14:08:41.521201+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:08:40.465288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:08:40.465288+0000","last_clean_scrub_stamp":"2024-03-28T14:08:40.465288+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:19:23.400242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"29'2","reported_seq":14,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-28T14:08:41.514610+0000","last_change":"2024-03-28T14:08:40.495924+0000","last_active":"2024-03-28T14:08:41.514610+0000","last_peered":"2024-03-28T14:08:41.514610+0000","last_clean":"2024-03-28T14:08:41.514610+0000","last_became_active":"2024-03-28T14:08:40.495575+0000","last_became_peered":"2024-03-28T14:08:40.495575+0000","last_unstale":"2024-03-28T14:08:41.514610+0000","last_undegraded":"2024-03-28T14:08:41.514610+0000","last_fullsized":"2024-03-28T14:08:41.514610+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:08:39.461488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:08:39.461488+0000","last_clean_scrub_stamp":"2024-03-28T14:08:39.461488+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:30:27.440113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":9,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-28T14:08:41.510836+0000","last_change":"2024-03-28T14:08:41.498685+0000","last_active":"2024-03-28T14:08:41.510836+0000","last_peered":"2024-03-28T14:08:41.510836+0000","last_clean":"2024-03-28T14:08:41.510836+0000","last_became_active":"2024-03-28T14:08:41.498095+0000","last_became_peered":"2024-03-28T14:08:41.498095+0000","last_unstale":"2024-03-28T14:08:41.510836+0000","last_undegraded":"2024-03-28T14:08:41.510836+0000","last_fullsized":"2024-03-28T14:08:41.510836+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:08:40.465288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:08:40.465288+0000","last_clean_scrub_stamp":"2024-03-28T14:08:40.465288+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:21:52.007093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1c","version":"29'2","reported_seq":14,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-28T14:08:41.515262+0000","last_change":"2024-03-28T14:08:40.517365+0000","last_active":"2024-03-28T14:08:41.515262+0000","last_peered":"2024-03-28T14:08:41.515262+0000","last_clean":"2024-03-28T14:08:41.515262+0000","last_became_active":"2024-03-28T14:08:40.516963+0000","last_became_peered":"2024-03-28T14:08:40.516963+0000","last_unstale":"2024-03-28T14:08:41.515262+0000","last_undegraded":"2024-03-28T14:08:41.515262+0000","last_fullsized":"2024-03-28T14:08:41.515262+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:08:39.461488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:08:39.461488+0000","last_clean_scrub_stamp":"2024-03-28T14:08:39.461488+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:11:12.076772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":9,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-28T14:08:41.546389+0000","last_change":"2024-03-28T14:08:41.544034+0000","last_active":"2024-03-28T14:08:41.546389+0000","last_peered":"2024-03-28T14:08:41.546389+0000","last_clean":"2024-03-28T14:08:41.546389+0000","last_became_active":"2024-03-28T14:08:41.543317+0000","last_became_peered":"2024-03-28T14:08:41.543317+0000","last_unstale":"2024-03-28T14:08:41.546389+0000","last_undegraded":"2024-03-28T14:08:41.546389+0000","last_fullsized":"2024-03-28T14:08:41.546389+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:08:40.465288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:08:40.465288+0000","last_clean_scrub_stamp":"2024-03-28T14:08:40.465288+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T16:14:54.560822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1b","version":"29'1","reported_seq":13,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-28T14:08:41.535843+0000","last_change":"2024-03-28T14:08:40.508833+0000","last_active":"2024-03-28T14:08:41.535843+0000","last_peered":"2024-03-28T14:08:41.535843+0000","last_clean":"2024-03-28T14:08:41.535843+0000","last_became_active":"2024-03-28T14:08:40.508097+0000","last_became_peered":"2024-03-28T14:08:40.508097+0000","last_unstale":"2024-03-28T14:08:41.535843+0000","last_undegraded":"2024-03-28T14:08:41.535843+0000","last_fullsized":"2024-03-28T14:08:41.535843+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:08:39.461488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:08:39.461488+0000","last_clean_scrub_stamp":"2024-03-28T14:08:39.461488+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:47:33.379209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":9,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-28T14:08:41.542291+0000","last_change":"2024-03-28T14:08:41.528811+0000","last_active":"2024-03-28T14:08:41.542291+0000","last_peered":"2024-03-28T14:08:41.542291+0000","last_clean":"2024-03-28T14:08:41.542291+0000","last_became_active":"2024-03-28T14:08:41.527982+0000","last_became_peered":"2024-03-28T14:08:41.527982+0000","last_unstale":"2024-03-28T14:08:41.542291+0000","last_undegraded":"2024-03-28T14:08:41.542291+0000","last_fullsized":"2024-03-28T14:08:41.542291+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:08:40.465288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:08:40.465288+0000","last_clean_scrub_stamp":"2024-03-28T14:08:40.465288+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:36:46.949506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"29'1","reported_seq":13,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-28T14:08:41.530631+0000","last_change":"2024-03-28T14:08:40.496942+0000","last_active":"2024-03-28T14:08:41.530631+0000","last_peered":"2024-03-28T14:08:41.530631+0000","last_clean":"2024-03-28T14:08:41.530631+0000","last_became_active":"2024-03-28T14:08:40.496220+0000","last_became_peered":"2024-03-28T14:08:40.496220+0000","last_unstale":"2024-03-28T14:08:41.530631+0000","last_undegraded":"2024-03-28T14:08:41.530631+0000","last_fullsized":"2024-03-28T14:08:41.530631+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:08:39.461488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:08:39.461488+0000","last_clean_scrub_stamp":"2024-03-28T14:08:39.461488+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:40:36.997030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":9,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-28T14:08:41.542467+0000","last_change":"2024-03-28T14:08:41.538113+0000","last_active":"2024-03-28T14:08:41.542467+0000","last_peered":"2024-03-28T14:08:41.542467+0000","last_clean":"2024-03-28T14:08:41.542467+0000","last_became_active":"2024-03-28T14:08:41.537473+0000","last_became_peered":"2024-03-28T14:08:41.537473+0000","last_unstale":"2024-03-28T14:08:41.542467+0000","last_undegraded":"2024-03-28T14:08:41.542467+0000","last_fullsized":"2024-03-28T14:08:41.542467+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:08:40.465288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:08:40.465288+0000","last_clean_scrub_stamp":"2024-03-28T14:08:40.465288+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T15:29:19.342685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":12,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-28T14:08:41.497897+0000","last_change":"2024-03-28T14:08:40.506135+0000","last_active":"2024-03-28T14:08:41.497897+0000","last_peered":"2024-03-28T14:08:41.497897+0000","last_clean":"2024-03-28T14:08:41.497897+0000","last_became_active":"2024-03-28T14:08:40.505705+0000","last_became_peered":"2024-03-28T14:08:40.505705+0000","last_unstale":"2024-03-28T14:08:41.497897+0000","last_undegraded":"2024-03-28T14:08:41.497897+0000","last_fullsized":"2024-03-28T14:08:41.497897+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:08:39.461488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:08:39.461488+0000","last_clean_scrub_stamp":"2024-03-28T14:08:39.461488+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:36:45.945706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":9,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-28T14:08:41.538114+0000","last_change":"2024-03-28T14:08:41.531930+0000","last_active":"2024-03-28T14:08:41.538114+0000","last_peered":"2024-03-28T14:08:41.538114+0000","last_clean":"2024-03-28T14:08:41.538114+0000","last_became_active":"2024-03-28T14:08:41.531651+0000","last_became_peered":"2024-03-28T14:08:41.531651+0000","last_unstale":"2024-03-28T14:08:41.538114+0000","last_undegraded":"2024-03-28T14:08:41.538114+0000","last_fullsized":"2024-03-28T14:08:41.538114+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:08:40.465288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:08:40.465288+0000","last_clean_scrub_stamp":"2024-03-28T14:08:40.465288+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T16:14:54.560822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":12,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-28T14:08:41.485979+0000","last_change":"2024-03-28T14:08:40.508061+0000","last_active":"2024-03-28T14:08:41.485979+0000","last_peered":"2024-03-28T14:08:41.485979+0000","last_clean":"2024-03-28T14:08:41.485979+0000","last_became_active":"2024-03-28T14:08:40.507264+0000","last_became_peered":"2024-03-28T14:08:40.507264+0000","last_unstale":"2024-03-28T14:08:41.485979+0000","last_undegraded":"2024-03-28T14:08:41.485979+0000","last_fullsized":"2024-03-28T14:08:41.485979+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:08:39.461488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:08:39.461488+0000","last_clean_scrub_stamp":"2024-03-28T14:08:39.461488+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:27:12.772591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":9,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-28T14:08:41.541821+0000","last_change":"2024-03-28T14:08:41.528643+0000","last_active":"2024-03-28T14:08:41.541821+0000","last_peered":"2024-03-28T14:08:41.541821+0000","last_clean":"2024-03-28T14:08:41.541821+0000","last_became_active":"2024-03-28T14:08:41.527599+0000","last_became_peered":"2024-03-28T14:08:41.527599+0000","last_unstale":"2024-03-28T14:08:41.541821+0000","last_undegraded":"2024-03-28T14:08:41.541821+0000","last_fullsized":"2024-03-28T14:08:41.541821+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:08:40.465288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:08:40.465288+0000","last_clean_scrub_stamp":"2024-03-28T14:08:40.465288+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T16:51:22.768085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":12,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-28T14:08:41.499979+0000","last_change":"2024-03-28T14:08:40.524054+0000","last_active":"2024-03-28T14:08:41.499979+0000","last_peered":"2024-03-28T14:08:41.499979+0000","last_clean":"2024-03-28T14:08:41.499979+0000","last_became_active":"2024-03-28T14:08:40.523618+0000","last_became_peered":"2024-03-28T14:08:40.523618+0000","last_unstale":"2024-03-28T14:08:41.499979+0000","last_undegraded":"2024-03-28T14:08:41.499979+0000","last_fullsized":"2024-03-28T14:08:41.499979+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:08:39.461488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:08:39.461488+0000","last_clean_scrub_stamp":"2024-03-28T14:08:39.461488+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T15:46:07.679056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":9,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-28T14:08:41.516953+0000","last_change":"2024-03-28T14:08:41.512531+0000","last_active":"2024-03-28T14:08:41.516953+0000","last_peered":"2024-03-28T14:08:41.516953+0000","last_clean":"2024-03-28T14:08:41.516953+0000","last_became_active":"2024-03-28T14:08:41.512196+0000","last_became_peered":"2024-03-28T14:08:41.512196+0000","last_unstale":"2024-03-28T14:08:41.516953+0000","last_undegraded":"2024-03-28T14:08:41.516953+0000","last_fullsized":"2024-03-28T14:08:41.516953+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:08:40.465288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:08:40.465288+0000","last_clean_scrub_stamp":"2024-03-28T14:08:40.465288+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:40:18.761368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":12,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-28T14:08:41.490390+0000","last_change":"2024-03-28T14:08:40.509986+0000","last_active":"2024-03-28T14:08:41.490390+0000","last_peered":"2024-03-28T14:08:41.490390+0000","last_clean":"2024-03-28T14:08:41.490390+0000","last_became_active":"2024-03-28T14:08:40.509535+0000","last_became_peered":"2024-03-28T14:08:40.509535+0000","last_unstale":"2024-03-28T14:08:41.490390+0000","last_undegraded":"2024-03-28T14:08:41.490390+0000","last_fullsized":"2024-03-28T14:08:41.490390+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:08:39.461488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:08:39.461488+0000","last_clean_scrub_stamp":"2024-03-28T14:08:39.461488+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:32:41.779209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"0'0","reported_seq":9,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-28T14:08:41.538220+0000","last_change":"2024-03-28T14:08:41.533404+0000","last_active":"2024-03-28T14:08:41.538220+0000","last_peered":"2024-03-28T14:08:41.538220+0000","last_clean":"2024-03-28T14:08:41.538220+0000","last_became_active":"2024-03-28T14:08:41.533141+0000","last_became_peered":"2024-03-28T14:08:41.533141+0000","last_unstale":"2024-03-28T14:08:41.538220+0000","last_undegraded":"2024-03-28T14:08:41.538220+0000","last_fullsized":"2024-03-28T14:08:41.538220+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:08:40.465288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:08:40.465288+0000","last_clean_scrub_stamp":"2024-03-28T14:08:40.465288+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T15:22:58.145356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.15","version":"29'3","reported_seq":15,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-28T14:08:41.542152+0000","last_change":"2024-03-28T14:08:40.523797+0000","last_active":"2024-03-28T14:08:41.542152+0000","last_peered":"2024-03-28T14:08:41.542152+0000","last_clean":"2024-03-28T14:08:41.542152+0000","last_became_active":"2024-03-28T14:08:40.523127+0000","last_became_peered":"2024-03-28T14:08:40.523127+0000","last_unstale":"2024-03-28T14:08:41.542152+0000","last_undegraded":"2024-03-28T14:08:41.542152+0000","last_fullsized":"2024-03-28T14:08:41.542152+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:08:39.461488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:08:39.461488+0000","last_clean_scrub_stamp":"2024-03-28T14:08:39.461488+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T17:12:46.783085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"0'0","reported_seq":9,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-28T14:08:41.542080+0000","last_change":"2024-03-28T14:08:41.528460+0000","last_active":"2024-03-28T14:08:41.542080+0000","last_peered":"2024-03-28T14:08:41.542080+0000","last_clean":"2024-03-28T14:08:41.542080+0000","last_became_active":"2024-03-28T14:08:41.527740+0000","last_became_peered":"2024-03-28T14:08:41.527740+0000","last_unstale":"2024-03-28T14:08:41.542080+0000","last_undegraded":"2024-03-28T14:08:41.542080+0000","last_fullsized":"2024-03-28T14:08:41.542080+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:08:40.465288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:08:40.465288+0000","last_clean_scrub_stamp":"2024-03-28T14:08:40.465288+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T15:29:19.342685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"29'6","reported_seq":18,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-28T14:08:41.538564+0000","last_change":"2024-03-28T14:08:40.495856+0000","last_active":"2024-03-28T14:08:41.538564+0000","last_peered":"2024-03-28T14:08:41.538564+0000","last_clean":"2024-03-28T14:08:41.538564+0000","last_became_active":"2024-03-28T14:08:40.495479+0000","last_became_peered":"2024-03-28T14:08:40.495479+0000","last_unstale":"2024-03-28T14:08:41.538564+0000","last_undegraded":"2024-03-28T14:08:41.538564+0000","last_fullsized":"2024-03-28T14:08:41.538564+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:08:39.461488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:08:39.461488+0000","last_clean_scrub_stamp":"2024-03-28T14:08:39.461488+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T16:14:53.557022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":9,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-28T14:08:41.534955+0000","last_change":"2024-03-28T14:08:41.529102+0000","last_active":"2024-03-28T14:08:41.534955+0000","last_peered":"2024-03-28T14:08:41.534955+0000","last_clean":"2024-03-28T14:08:41.534955+0000","last_became_active":"2024-03-28T14:08:41.528836+0000","last_became_peered":"2024-03-28T14:08:41.528836+0000","last_unstale":"2024-03-28T14:08:41.534955+0000","last_undegraded":"2024-03-28T14:08:41.534955+0000","last_fullsized":"2024-03-28T14:08:41.534955+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:08:40.465288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:08:40.465288+0000","last_clean_scrub_stamp":"2024-03-28T14:08:40.465288+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:30:28.443913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"29'2","reported_seq":14,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-28T14:08:41.546144+0000","last_change":"2024-03-28T14:08:40.515762+0000","last_active":"2024-03-28T14:08:41.546144+0000","last_peered":"2024-03-28T14:08:41.546144+0000","last_clean":"2024-03-28T14:08:41.546144+0000","last_became_active":"2024-03-28T14:08:40.514769+0000","last_became_peered":"2024-03-28T14:08:40.514769+0000","last_unstale":"2024-03-28T14:08:41.546144+0000","last_undegraded":"2024-03-28T14:08:41.546144+0000","last_fullsized":"2024-03-28T14:08:41.546144+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:08:39.461488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:08:39.461488+0000","last_clean_scrub_stamp":"2024-03-28T14:08:39.461488+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:49:14.011336+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":9,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-28T14:08:41.516873+0000","last_change":"2024-03-28T14:08:41.511329+0000","last_active":"2024-03-28T14:08:41.516873+0000","last_peered":"2024-03-28T14:08:41.516873+0000","last_clean":"2024-03-28T14:08:41.516873+0000","last_became_active":"2024-03-28T14:08:41.510972+0000","last_became_peered":"2024-03-28T14:08:41.510972+0000","last_unstale":"2024-03-28T14:08:41.516873+0000","last_undegraded":"2024-03-28T14:08:41.516873+0000","last_fullsized":"2024-03-28T14:08:41.516873+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:08:40.465288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:08:40.465288+0000","last_clean_scrub_stamp":"2024-03-28T14:08:40.465288+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:12:57.613588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":12,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-28T14:08:41.483578+0000","last_change":"2024-03-28T14:08:40.501882+0000","last_active":"2024-03-28T14:08:41.483578+0000","last_peered":"2024-03-28T14:08:41.483578+0000","last_clean":"2024-03-28T14:08:41.483578+0000","last_became_active":"2024-03-28T14:08:40.501346+0000","last_became_peered":"2024-03-28T14:08:40.501346+0000","last_unstale":"2024-03-28T14:08:41.483578+0000","last_undegraded":"2024-03-28T14:08:41.483578+0000","last_fullsized":"2024-03-28T14:08:41.483578+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:08:39.461488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:08:39.461488+0000","last_clean_scrub_stamp":"2024-03-28T14:08:39.461488+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:59:07.212387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":9,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-28T14:08:41.546410+0000","last_change":"2024-03-28T14:08:41.540015+0000","last_active":"2024-03-28T14:08:41.546410+0000","last_peered":"2024-03-28T14:08:41.546410+0000","last_clean":"2024-03-28T14:08:41.546410+0000","last_became_active":"2024-03-28T14:08:41.539044+0000","last_became_peered":"2024-03-28T14:08:41.539044+0000","last_unstale":"2024-03-28T14:08:41.546410+0000","last_undegraded":"2024-03-28T14:08:41.546410+0000","last_fullsized":"2024-03-28T14:08:41.546410+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:08:40.465288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:08:40.465288+0000","last_clean_scrub_stamp":"2024-03-28T14:08:40.465288+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:30:28.443913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":12,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-28T14:08:41.498040+0000","last_change":"2024-03-28T14:08:40.522631+0000","last_active":"2024-03-28T14:08:41.498040+0000","last_peered":"2024-03-28T14:08:41.498040+0000","last_clean":"2024-03-28T14:08:41.498040+0000","last_became_active":"2024-03-28T14:08:40.522368+0000","last_became_peered":"2024-03-28T14:08:40.522368+0000","last_unstale":"2024-03-28T14:08:41.498040+0000","last_undegraded":"2024-03-28T14:08:41.498040+0000","last_fullsized":"2024-03-28T14:08:41.498040+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:08:39.461488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:08:39.461488+0000","last_clean_scrub_stamp":"2024-03-28T14:08:39.461488+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:02:47.106846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":9,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-28T14:08:41.544936+0000","last_change":"2024-03-28T14:08:41.540916+0000","last_active":"2024-03-28T14:08:41.544936+0000","last_peered":"2024-03-28T14:08:41.544936+0000","last_clean":"2024-03-28T14:08:41.544936+0000","last_became_active":"2024-03-28T14:08:41.540414+0000","last_became_peered":"2024-03-28T14:08:41.540414+0000","last_unstale":"2024-03-28T14:08:41.544936+0000","last_undegraded":"2024-03-28T14:08:41.544936+0000","last_fullsized":"2024-03-28T14:08:41.544936+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:08:40.465288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:08:40.465288+0000","last_clean_scrub_stamp":"2024-03-28T14:08:40.465288+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:19:23.400242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.10","version":"29'8","reported_seq":20,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-28T14:08:41.514972+0000","last_change":"2024-03-28T14:08:40.495790+0000","last_active":"2024-03-28T14:08:41.514972+0000","last_peered":"2024-03-28T14:08:41.514972+0000","last_clean":"2024-03-28T14:08:41.514972+0000","last_became_active":"2024-03-28T14:08:40.495441+0000","last_became_peered":"2024-03-28T14:08:40.495441+0000","last_unstale":"2024-03-28T14:08:41.514972+0000","last_undegraded":"2024-03-28T14:08:41.514972+0000","last_fullsized":"2024-03-28T14:08:41.514972+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:08:39.461488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:08:39.461488+0000","last_clean_scrub_stamp":"2024-03-28T14:08:39.461488+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T15:29:18.338885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"0'0","reported_seq":9,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-28T14:08:41.535815+0000","last_change":"2024-03-28T14:08:41.530295+0000","last_active":"2024-03-28T14:08:41.535815+0000","last_peered":"2024-03-28T14:08:41.535815+0000","last_clean":"2024-03-28T14:08:41.535815+0000","last_became_active":"2024-03-28T14:08:41.529583+0000","last_became_peered":"2024-03-28T14:08:41.529583+0000","last_unstale":"2024-03-28T14:08:41.535815+0000","last_undegraded":"2024-03-28T14:08:41.535815+0000","last_fullsized":"2024-03-28T14:08:41.535815+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:08:40.465288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:08:40.465288+0000","last_clean_scrub_stamp":"2024-03-28T14:08:40.465288+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T16:19:05.943365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.f","version":"29'1","reported_seq":13,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-28T14:08:41.503214+0000","last_change":"2024-03-28T14:08:40.503423+0000","last_active":"2024-03-28T14:08:41.503214+0000","last_peered":"2024-03-28T14:08:41.503214+0000","last_clean":"2024-03-28T14:08:41.503214+0000","last_became_active":"2024-03-28T14:08:40.503039+0000","last_became_peered":"2024-03-28T14:08:40.503039+0000","last_unstale":"2024-03-28T14:08:41.503214+0000","last_undegraded":"2024-03-28T14:08:41.503214+0000","last_fullsized":"2024-03-28T14:08:41.503214+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:08:39.461488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:08:39.461488+0000","last_clean_scrub_stamp":"2024-03-28T14:08:39.461488+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:19:22.396442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":9,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-28T14:08:41.543673+0000","last_change":"2024-03-28T14:08:41.532505+0000","last_active":"2024-03-28T14:08:41.543673+0000","last_peered":"2024-03-28T14:08:41.543673+0000","last_clean":"2024-03-28T14:08:41.543673+0000","last_became_active":"2024-03-28T14:08:41.530486+0000","last_became_peered":"2024-03-28T14:08:41.530486+0000","last_unstale":"2024-03-28T14:08:41.543673+0000","last_undegraded":"2024-03-28T14:08:41.543673+0000","last_fullsized":"2024-03-28T14:08:41.543673+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:08:40.465288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:08:40.465288+0000","last_clean_scrub_stamp":"2024-03-28T14:08:40.465288+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T14:33:01.059177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.e","version":"29'2","reported_seq":14,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-28T14:08:41.545482+0000","last_change":"2024-03-28T14:08:40.513533+0000","last_active":"2024-03-28T14:08:41.545482+0000","last_peered":"2024-03-28T14:08:41.545482+0000","last_clean":"2024-03-28T14:08:41.545482+0000","last_became_active":"2024-03-28T14:08:40.512865+0000","last_became_peered":"2024-03-28T14:08:40.512865+0000","last_unstale":"2024-03-28T14:08:41.545482+0000","last_undegraded":"2024-03-28T14:08:41.545482+0000","last_fullsized":"2024-03-28T14:08:41.545482+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:08:39.461488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:08:39.461488+0000","last_clean_scrub_stamp":"2024-03-28T14:08:39.461488+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:32:41.779209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":9,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-28T14:08:41.520058+0000","last_change":"2024-03-28T14:08:41.518443+0000","last_active":"2024-03-28T14:08:41.520058+0000","last_peered":"2024-03-28T14:08:41.520058+0000","last_clean":"2024-03-28T14:08:41.520058+0000","last_became_active":"2024-03-28T14:08:41.518226+0000","last_became_peered":"2024-03-28T14:08:41.518226+0000","last_unstale":"2024-03-28T14:08:41.520058+0000","last_undegraded":"2024-03-28T14:08:41.520058+0000","last_fullsized":"2024-03-28T14:08:41.520058+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:08:40.465288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:08:40.465288+0000","last_clean_scrub_stamp":"2024-03-28T14:08:40.465288+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:02:48.110646+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.d","version":"29'1","reported_seq":13,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-28T14:08:41.521487+0000","last_change":"2024-03-28T14:08:40.508324+0000","last_active":"2024-03-28T14:08:41.521487+0000","last_peered":"2024-03-28T14:08:41.521487+0000","last_clean":"2024-03-28T14:08:41.521487+0000","last_became_active":"2024-03-28T14:08:40.507410+0000","last_became_peered":"2024-03-28T14:08:40.507410+0000","last_unstale":"2024-03-28T14:08:41.521487+0000","last_undegraded":"2024-03-28T14:08:41.521487+0000","last_fullsized":"2024-03-28T14:08:41.521487+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:08:39.461488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:08:39.461488+0000","last_clean_scrub_stamp":"2024-03-28T14:08:39.461488+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:27:15.765153+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"0'0","reported_seq":9,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-28T14:08:41.526009+0000","last_change":"2024-03-28T14:08:41.524631+0000","last_active":"2024-03-28T14:08:41.526009+0000","last_peered":"2024-03-28T14:08:41.526009+0000","last_clean":"2024-03-28T14:08:41.526009+0000","last_became_active":"2024-03-28T14:08:41.524356+0000","last_became_peered":"2024-03-28T14:08:41.524356+0000","last_unstale":"2024-03-28T14:08:41.526009+0000","last_undegraded":"2024-03-28T14:08:41.526009+0000","last_fullsized":"2024-03-28T14:08:41.526009+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:08:40.465288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:08:40.465288+0000","last_clean_scrub_stamp":"2024-03-28T14:08:40.465288+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:54:36.883461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":12,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-28T14:08:41.486790+0000","last_change":"2024-03-28T14:08:40.509241+0000","last_active":"2024-03-28T14:08:41.486790+0000","last_peered":"2024-03-28T14:08:41.486790+0000","last_clean":"2024-03-28T14:08:41.486790+0000","last_became_active":"2024-03-28T14:08:40.507984+0000","last_became_peered":"2024-03-28T14:08:40.507984+0000","last_unstale":"2024-03-28T14:08:41.486790+0000","last_undegraded":"2024-03-28T14:08:41.486790+0000","last_fullsized":"2024-03-28T14:08:41.486790+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:08:39.461488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:08:39.461488+0000","last_clean_scrub_stamp":"2024-03-28T14:08:39.461488+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:37:19.550407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":9,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-28T14:08:41.522657+0000","last_change":"2024-03-28T14:08:41.520093+0000","last_active":"2024-03-28T14:08:41.522657+0000","last_peered":"2024-03-28T14:08:41.522657+0000","last_clean":"2024-03-28T14:08:41.522657+0000","last_became_active":"2024-03-28T14:08:41.519746+0000","last_became_peered":"2024-03-28T14:08:41.519746+0000","last_unstale":"2024-03-28T14:08:41.522657+0000","last_undegraded":"2024-03-28T14:08:41.522657+0000","last_fullsized":"2024-03-28T14:08:41.522657+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:08:40.465288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:08:40.465288+0000","last_clean_scrub_stamp":"2024-03-28T14:08:40.465288+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:00:07.685993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.b","version":"29'2","reported_seq":14,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-28T14:08:41.514745+0000","last_change":"2024-03-28T14:08:40.502639+0000","last_active":"2024-03-28T14:08:41.514745+0000","last_peered":"2024-03-28T14:08:41.514745+0000","last_clean":"2024-03-28T14:08:41.514745+0000","last_became_active":"2024-03-28T14:08:40.502250+0000","last_became_peered":"2024-03-28T14:08:40.502250+0000","last_unstale":"2024-03-28T14:08:41.514745+0000","last_undegraded":"2024-03-28T14:08:41.514745+0000","last_fullsized":"2024-03-28T14:08:41.514745+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:08:39.461488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:08:39.461488+0000","last_clean_scrub_stamp":"2024-03-28T14:08:39.461488+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:19:22.396442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":9,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-28T14:08:41.514052+0000","last_change":"2024-03-28T14:08:41.511022+0000","last_active":"2024-03-28T14:08:41.514052+0000","last_peered":"2024-03-28T14:08:41.514052+0000","last_clean":"2024-03-28T14:08:41.514052+0000","last_became_active":"2024-03-28T14:08:41.510604+0000","last_became_peered":"2024-03-28T14:08:41.510604+0000","last_unstale":"2024-03-28T14:08:41.514052+0000","last_undegraded":"2024-03-28T14:08:41.514052+0000","last_fullsized":"2024-03-28T14:08:41.514052+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:08:40.465288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:08:40.465288+0000","last_clean_scrub_stamp":"2024-03-28T14:08:40.465288+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:52:05.146099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":12,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-28T14:08:41.500718+0000","last_change":"2024-03-28T14:08:40.503771+0000","last_active":"2024-03-28T14:08:41.500718+0000","last_peered":"2024-03-28T14:08:41.500718+0000","last_clean":"2024-03-28T14:08:41.500718+0000","last_became_active":"2024-03-28T14:08:40.502947+0000","last_became_peered":"2024-03-28T14:08:40.502947+0000","last_unstale":"2024-03-28T14:08:41.500718+0000","last_undegraded":"2024-03-28T14:08:41.500718+0000","last_fullsized":"2024-03-28T14:08:41.500718+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:08:39.461488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:08:39.461488+0000","last_clean_scrub_stamp":"2024-03-28T14:08:39.461488+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:49:14.011336+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":9,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-28T14:08:41.501380+0000","last_change":"2024-03-28T14:08:41.498351+0000","last_active":"2024-03-28T14:08:41.501380+0000","last_peered":"2024-03-28T14:08:41.501380+0000","last_clean":"2024-03-28T14:08:41.501380+0000","last_became_active":"2024-03-28T14:08:41.497530+0000","last_became_peered":"2024-03-28T14:08:41.497530+0000","last_unstale":"2024-03-28T14:08:41.501380+0000","last_undegraded":"2024-03-28T14:08:41.501380+0000","last_fullsized":"2024-03-28T14:08:41.501380+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:08:40.465288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:08:40.465288+0000","last_clean_scrub_stamp":"2024-03-28T14:08:40.465288+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:50:39.985024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":12,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-28T14:08:41.486476+0000","last_change":"2024-03-28T14:08:40.510590+0000","last_active":"2024-03-28T14:08:41.486476+0000","last_peered":"2024-03-28T14:08:41.486476+0000","last_clean":"2024-03-28T14:08:41.486476+0000","last_became_active":"2024-03-28T14:08:40.509879+0000","last_became_peered":"2024-03-28T14:08:40.509879+0000","last_unstale":"2024-03-28T14:08:41.486476+0000","last_undegraded":"2024-03-28T14:08:41.486476+0000","last_fullsized":"2024-03-28T14:08:41.486476+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:08:39.461488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:08:39.461488+0000","last_clean_scrub_stamp":"2024-03-28T14:08:39.461488+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:54:15.294534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":9,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-28T14:08:41.539118+0000","last_change":"2024-03-28T14:08:41.518804+0000","last_active":"2024-03-28T14:08:41.539118+0000","last_peered":"2024-03-28T14:08:41.539118+0000","last_clean":"2024-03-28T14:08:41.539118+0000","last_became_active":"2024-03-28T14:08:41.517961+0000","last_became_peered":"2024-03-28T14:08:41.517961+0000","last_unstale":"2024-03-28T14:08:41.539118+0000","last_undegraded":"2024-03-28T14:08:41.539118+0000","last_fullsized":"2024-03-28T14:08:41.539118+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:08:40.465288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:08:40.465288+0000","last_clean_scrub_stamp":"2024-03-28T14:08:40.465288+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:19:23.400242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":12,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-28T14:08:41.497880+0000","last_change":"2024-03-28T14:08:40.526487+0000","last_active":"2024-03-28T14:08:41.497880+0000","last_peered":"2024-03-28T14:08:41.497880+0000","last_clean":"2024-03-28T14:08:41.497880+0000","last_became_active":"2024-03-28T14:08:40.525646+0000","last_became_peered":"2024-03-28T14:08:40.525646+0000","last_unstale":"2024-03-28T14:08:41.497880+0000","last_undegraded":"2024-03-28T14:08:41.497880+0000","last_fullsized":"2024-03-28T14:08:41.497880+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:08:39.461488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:08:39.461488+0000","last_clean_scrub_stamp":"2024-03-28T14:08:39.461488+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T18:29:51.673169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":9,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-28T14:08:41.532168+0000","last_change":"2024-03-28T14:08:41.528486+0000","last_active":"2024-03-28T14:08:41.532168+0000","last_peered":"2024-03-28T14:08:41.532168+0000","last_clean":"2024-03-28T14:08:41.532168+0000","last_became_active":"2024-03-28T14:08:41.528180+0000","last_became_peered":"2024-03-28T14:08:41.528180+0000","last_unstale":"2024-03-28T14:08:41.532168+0000","last_undegraded":"2024-03-28T14:08:41.532168+0000","last_fullsized":"2024-03-28T14:08:41.532168+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:08:40.465288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:08:40.465288+0000","last_clean_scrub_stamp":"2024-03-28T14:08:40.465288+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T14:18:16.544200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"29'2","reported_seq":14,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-28T14:08:41.520878+0000","last_change":"2024-03-28T14:08:40.517457+0000","last_active":"2024-03-28T14:08:41.520878+0000","last_peered":"2024-03-28T14:08:41.520878+0000","last_clean":"2024-03-28T14:08:41.520878+0000","last_became_active":"2024-03-28T14:08:40.516812+0000","last_became_peered":"2024-03-28T14:08:40.516812+0000","last_unstale":"2024-03-28T14:08:41.520878+0000","last_undegraded":"2024-03-28T14:08:41.520878+0000","last_fullsized":"2024-03-28T14:08:41.520878+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:08:39.461488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:08:39.461488+0000","last_clean_scrub_stamp":"2024-03-28T14:08:39.461488+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T18:29:51.673169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":9,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-28T14:08:41.543624+0000","last_change":"2024-03-28T14:08:41.531881+0000","last_active":"2024-03-28T14:08:41.543624+0000","last_peered":"2024-03-28T14:08:41.543624+0000","last_clean":"2024-03-28T14:08:41.543624+0000","last_became_active":"2024-03-28T14:08:41.529671+0000","last_became_peered":"2024-03-28T14:08:41.529671+0000","last_unstale":"2024-03-28T14:08:41.543624+0000","last_undegraded":"2024-03-28T14:08:41.543624+0000","last_fullsized":"2024-03-28T14:08:41.543624+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:08:40.465288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:08:40.465288+0000","last_clean_scrub_stamp":"2024-03-28T14:08:40.465288+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T15:46:08.682856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":12,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-28T14:08:41.490689+0000","last_change":"2024-03-28T14:08:40.502674+0000","last_active":"2024-03-28T14:08:41.490689+0000","last_peered":"2024-03-28T14:08:41.490689+0000","last_clean":"2024-03-28T14:08:41.490689+0000","last_became_active":"2024-03-28T14:08:40.502316+0000","last_became_peered":"2024-03-28T14:08:40.502316+0000","last_unstale":"2024-03-28T14:08:41.490689+0000","last_undegraded":"2024-03-28T14:08:41.490689+0000","last_fullsized":"2024-03-28T14:08:41.490689+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:08:39.461488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:08:39.461488+0000","last_clean_scrub_stamp":"2024-03-28T14:08:39.461488+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:59:07.212387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":9,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-28T14:08:41.501593+0000","last_change":"2024-03-28T14:08:41.496026+0000","last_active":"2024-03-28T14:08:41.501593+0000","last_peered":"2024-03-28T14:08:41.501593+0000","last_clean":"2024-03-28T14:08:41.501593+0000","last_became_active":"2024-03-28T14:08:41.495649+0000","last_became_peered":"2024-03-28T14:08:41.495649+0000","last_unstale":"2024-03-28T14:08:41.501593+0000","last_undegraded":"2024-03-28T14:08:41.501593+0000","last_fullsized":"2024-03-28T14:08:41.501593+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:08:40.465288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:08:40.465288+0000","last_clean_scrub_stamp":"2024-03-28T14:08:40.465288+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T17:00:12.873136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":12,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-28T14:08:41.484542+0000","last_change":"2024-03-28T14:08:40.507920+0000","last_active":"2024-03-28T14:08:41.484542+0000","last_peered":"2024-03-28T14:08:41.484542+0000","last_clean":"2024-03-28T14:08:41.484542+0000","last_became_active":"2024-03-28T14:08:40.507479+0000","last_became_peered":"2024-03-28T14:08:40.507479+0000","last_unstale":"2024-03-28T14:08:41.484542+0000","last_undegraded":"2024-03-28T14:08:41.484542+0000","last_fullsized":"2024-03-28T14:08:41.484542+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:08:39.461488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:08:39.461488+0000","last_clean_scrub_stamp":"2024-03-28T14:08:39.461488+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T17:12:46.783085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":9,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-28T14:08:41.524113+0000","last_change":"2024-03-28T14:08:41.511743+0000","last_active":"2024-03-28T14:08:41.524113+0000","last_peered":"2024-03-28T14:08:41.524113+0000","last_clean":"2024-03-28T14:08:41.524113+0000","last_became_active":"2024-03-28T14:08:41.510984+0000","last_became_peered":"2024-03-28T14:08:41.510984+0000","last_unstale":"2024-03-28T14:08:41.524113+0000","last_undegraded":"2024-03-28T14:08:41.524113+0000","last_fullsized":"2024-03-28T14:08:41.524113+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:08:40.465288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:08:40.465288+0000","last_clean_scrub_stamp":"2024-03-28T14:08:40.465288+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:21:18.041106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":12,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-28T14:08:41.504372+0000","last_change":"2024-03-28T14:08:40.504570+0000","last_active":"2024-03-28T14:08:41.504372+0000","last_peered":"2024-03-28T14:08:41.504372+0000","last_clean":"2024-03-28T14:08:41.504372+0000","last_became_active":"2024-03-28T14:08:40.504198+0000","last_became_peered":"2024-03-28T14:08:40.504198+0000","last_unstale":"2024-03-28T14:08:41.504372+0000","last_undegraded":"2024-03-28T14:08:41.504372+0000","last_fullsized":"2024-03-28T14:08:41.504372+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:08:39.461488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:08:39.461488+0000","last_clean_scrub_stamp":"2024-03-28T14:08:39.461488+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T18:29:51.673169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":9,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-28T14:08:41.516693+0000","last_change":"2024-03-28T14:08:41.511129+0000","last_active":"2024-03-28T14:08:41.516693+0000","last_peered":"2024-03-28T14:08:41.516693+0000","last_clean":"2024-03-28T14:08:41.516693+0000","last_became_active":"2024-03-28T14:08:41.510286+0000","last_became_peered":"2024-03-28T14:08:41.510286+0000","last_unstale":"2024-03-28T14:08:41.516693+0000","last_undegraded":"2024-03-28T14:08:41.516693+0000","last_fullsized":"2024-03-28T14:08:41.516693+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:08:40.465288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:08:40.465288+0000","last_clean_scrub_stamp":"2024-03-28T14:08:40.465288+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:21:52.007093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":12,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-28T14:08:41.483610+0000","last_change":"2024-03-28T14:08:40.508674+0000","last_active":"2024-03-28T14:08:41.483610+0000","last_peered":"2024-03-28T14:08:41.483610+0000","last_clean":"2024-03-28T14:08:41.483610+0000","last_became_active":"2024-03-28T14:08:40.508240+0000","last_became_peered":"2024-03-28T14:08:40.508240+0000","last_unstale":"2024-03-28T14:08:41.483610+0000","last_undegraded":"2024-03-28T14:08:41.483610+0000","last_fullsized":"2024-03-28T14:08:41.483610+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:08:39.461488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:08:39.461488+0000","last_clean_scrub_stamp":"2024-03-28T14:08:39.461488+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T17:00:11.869336+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":9,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-28T14:08:41.516984+0000","last_change":"2024-03-28T14:08:41.512590+0000","last_active":"2024-03-28T14:08:41.516984+0000","last_peered":"2024-03-28T14:08:41.516984+0000","last_clean":"2024-03-28T14:08:41.516984+0000","last_became_active":"2024-03-28T14:08:41.512351+0000","last_became_peered":"2024-03-28T14:08:41.512351+0000","last_unstale":"2024-03-28T14:08:41.516984+0000","last_undegraded":"2024-03-28T14:08:41.516984+0000","last_fullsized":"2024-03-28T14:08:41.516984+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:08:40.465288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:08:40.465288+0000","last_clean_scrub_stamp":"2024-03-28T14:08:40.465288+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T17:13:46.853110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":12,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-28T14:08:41.486921+0000","last_change":"2024-03-28T14:08:40.516523+0000","last_active":"2024-03-28T14:08:41.486921+0000","last_peered":"2024-03-28T14:08:41.486921+0000","last_clean":"2024-03-28T14:08:41.486921+0000","last_became_active":"2024-03-28T14:08:40.515289+0000","last_became_peered":"2024-03-28T14:08:40.515289+0000","last_unstale":"2024-03-28T14:08:41.486921+0000","last_undegraded":"2024-03-28T14:08:41.486921+0000","last_fullsized":"2024-03-28T14:08:41.486921+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:08:39.461488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:08:39.461488+0000","last_clean_scrub_stamp":"2024-03-28T14:08:39.461488+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:21:17.037306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":9,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-28T14:08:41.521289+0000","last_change":"2024-03-28T14:08:41.506696+0000","last_active":"2024-03-28T14:08:41.521289+0000","last_peered":"2024-03-28T14:08:41.521289+0000","last_clean":"2024-03-28T14:08:41.521289+0000","last_became_active":"2024-03-28T14:08:41.505946+0000","last_became_peered":"2024-03-28T14:08:41.505946+0000","last_unstale":"2024-03-28T14:08:41.521289+0000","last_undegraded":"2024-03-28T14:08:41.521289+0000","last_fullsized":"2024-03-28T14:08:41.521289+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:08:40.465288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:08:40.465288+0000","last_clean_scrub_stamp":"2024-03-28T14:08:40.465288+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T17:00:12.873136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":12,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-28T14:08:41.497600+0000","last_change":"2024-03-28T14:08:40.525713+0000","last_active":"2024-03-28T14:08:41.497600+0000","last_peered":"2024-03-28T14:08:41.497600+0000","last_clean":"2024-03-28T14:08:41.497600+0000","last_became_active":"2024-03-28T14:08:40.525183+0000","last_became_peered":"2024-03-28T14:08:40.525183+0000","last_unstale":"2024-03-28T14:08:41.497600+0000","last_undegraded":"2024-03-28T14:08:41.497600+0000","last_fullsized":"2024-03-28T14:08:41.497600+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:08:39.461488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:08:39.461488+0000","last_clean_scrub_stamp":"2024-03-28T14:08:39.461488+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T16:51:21.764285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":9,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-28T14:08:41.538482+0000","last_change":"2024-03-28T14:08:41.522760+0000","last_active":"2024-03-28T14:08:41.538482+0000","last_peered":"2024-03-28T14:08:41.538482+0000","last_clean":"2024-03-28T14:08:41.538482+0000","last_became_active":"2024-03-28T14:08:41.522046+0000","last_became_peered":"2024-03-28T14:08:41.522046+0000","last_unstale":"2024-03-28T14:08:41.538482+0000","last_undegraded":"2024-03-28T14:08:41.538482+0000","last_fullsized":"2024-03-28T14:08:41.538482+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:08:40.465288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:08:40.465288+0000","last_clean_scrub_stamp":"2024-03-28T14:08:40.465288+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T18:29:52.676969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":12,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-28T14:08:41.500774+0000","last_change":"2024-03-28T14:08:40.511404+0000","last_active":"2024-03-28T14:08:41.500774+0000","last_peered":"2024-03-28T14:08:41.500774+0000","last_clean":"2024-03-28T14:08:41.500774+0000","last_became_active":"2024-03-28T14:08:40.510675+0000","last_became_peered":"2024-03-28T14:08:40.510675+0000","last_unstale":"2024-03-28T14:08:41.500774+0000","last_undegraded":"2024-03-28T14:08:41.500774+0000","last_fullsized":"2024-03-28T14:08:41.500774+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:08:39.461488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:08:39.461488+0000","last_clean_scrub_stamp":"2024-03-28T14:08:39.461488+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:37:19.550407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":184,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-28T14:08:41.496713+0000","last_change":"2024-03-28T14:07:49.234448+0000","last_active":"2024-03-28T14:08:41.496713+0000","last_peered":"2024-03-28T14:08:41.496713+0000","last_clean":"2024-03-28T14:08:41.496713+0000","last_became_active":"2024-03-28T14:07:49.233895+0000","last_became_peered":"2024-03-28T14:07:49.233895+0000","last_unstale":"2024-03-28T14:08:41.496713+0000","last_undegraded":"2024-03-28T14:08:41.496713+0000","last_fullsized":"2024-03-28T14:08:41.496713+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:07:48.069157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:07:48.069157+0000","last_clean_scrub_stamp":"2024-03-28T14:07:48.069157+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:11:39.977557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574863,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27928,"kb_used_data":900,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343912,"statfs":{"total":96636764160,"available":96608165888,"internally_reserved":0,"allocated":921600,"data_stored":712841,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":11,"apply_latency_ms":11,"commit_latency_ns":11000000,"apply_latency_ns":11000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574864,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27344,"kb_used_data":324,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344496,"statfs":{"total":96636764160,"available":96608763904,"internally_reserved":0,"allocated":331776,"data_stored":122999,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574862,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27344,"kb_used_data":324,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344496,"statfs":{"total":96636764160,"available":96608763904,"internally_reserved":0,"allocated":331776,"data_stored":123055,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":11,"apply_latency_ms":11,"commit_latency_ns":11000000,"apply_latency_ns":11000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574863,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27336,"kb_used_data":316,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344504,"statfs":{"total":96636764160,"available":96608772096,"internally_reserved":0,"allocated":323584,"data_stored":122383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574865,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27352,"kb_used_data":332,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344488,"statfs":{"total":96636764160,"available":96608755712,"internally_reserved":0,"allocated":339968,"data_stored":124017,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574863,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27340,"kb_used_data":320,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344500,"statfs":{"total":96636764160,"available":96608768000,"internally_reserved":0,"allocated":327680,"data_stored":122453,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574861,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27352,"kb_used_data":332,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344488,"statfs":{"total":96636764160,"available":96608755712,"internally_reserved":0,"allocated":339968,"data_stored":124017,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574864,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27924,"kb_used_data":896,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343916,"statfs":{"total":96636764160,"available":96608169984,"internally_reserved":0,"allocated":917504,"data_stored":712751,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-28T14:08:44.916 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-03-28T14:08:44.916 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-03-28T14:08:45.084 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:45.082+0000 7f5a779d5640 1 Processor -- start 2024-03-28T14:08:45.085 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:45.082+0000 7f5a779d5640 1 -- start start 2024-03-28T14:08:45.087 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:45.082+0000 7f5a779d5640 1 --2- >> v2:172.21.15.171:3300/0 conn(0x7f5a700b1e80 0x7f5a70071be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:45.087 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:45.082+0000 7f5a779d5640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5a70072fd0 con 0x7f5a70145ce0 2024-03-28T14:08:45.087 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:45.082+0000 7f5a779d5640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5a700734a0 con 0x7f5a70072bf0 2024-03-28T14:08:45.087 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:45.082+0000 7f5a779d5640 1 -- --> v2:172.21.15.171:3300/0 -- mon_getmap magic: 0 -- 0x7f5a70073250 con 0x7f5a700b1e80 2024-03-28T14:08:45.087 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:45.082+0000 7f5a75f72640 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f5a70145ce0 0x7f5a7014fe90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:59858/0 (socket says 172.21.15.104:59858) 2024-03-28T14:08:45.087 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:45.082+0000 7f5a75f72640 1 -- 172.21.15.104:0/2675916199 learned_addr learned my addr 172.21.15.104:0/2675916199 (peer_addr_for_me v1:172.21.15.104:0/0) 2024-03-28T14:08:45.087 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:45.082+0000 7f5a74f70640 1 --2- 172.21.15.104:0/2675916199 >> v2:172.21.15.171:3300/0 conn(0x7f5a700b1e80 0x7f5a70071be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:45.087 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:45.082+0000 7f5a67fff640 1 -- 172.21.15.104:0/2675916199 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2126415798 0 0) 0x7f5a70072fd0 con 0x7f5a70145ce0 2024-03-28T14:08:45.087 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:45.082+0000 7f5a67fff640 1 -- 172.21.15.104:0/2675916199 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5a5c003530 con 0x7f5a70145ce0 2024-03-28T14:08:45.087 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:45.082+0000 7f5a67fff640 1 -- 172.21.15.104:0/2675916199 <== mon.0 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4156665252 0 0) 0x7f5a5c003530 con 0x7f5a70145ce0 2024-03-28T14:08:45.088 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:45.082+0000 7f5a67fff640 1 -- 172.21.15.104:0/2675916199 >> v2:172.21.15.171:3300/0 conn(0x7f5a700b1e80 msgr2=0x7f5a70071be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T14:08:45.088 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:45.082+0000 7f5a67fff640 1 --2- 172.21.15.104:0/2675916199 >> v2:172.21.15.171:3300/0 conn(0x7f5a700b1e80 0x7f5a70071be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:45.088 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:45.082+0000 7f5a67fff640 1 -- 172.21.15.104:0/2675916199 >> v1:172.21.15.171:6789/0 conn(0x7f5a70072bf0 legacy=0x7f5a700b12f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:45.088 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:45.082+0000 7f5a67fff640 1 -- 172.21.15.104:0/2675916199 --> v1:172.21.15.104:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5a70073780 con 0x7f5a70145ce0 2024-03-28T14:08:45.088 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:45.082+0000 7f5a67fff640 1 -- 172.21.15.104:0/2675916199 <== mon.0 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3745580849 0 0) 0x7f5a6c001a00 con 0x7f5a70145ce0 2024-03-28T14:08:45.088 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:45.082+0000 7f5a67fff640 1 -- 172.21.15.104:0/2675916199 >> v1:172.21.15.104:6789/0 conn(0x7f5a70145ce0 legacy=0x7f5a7014fe90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:45.088 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:45.082+0000 7f5a67fff640 1 --2- 172.21.15.104:0/2675916199 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f5a5c0039c0 0x7f5a5c003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:45.088 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:45.082+0000 7f5a67fff640 1 --2- 172.21.15.104:0/2675916199 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5a5c004810 0x7f5a5c004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:45.088 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:45.082+0000 7f5a67fff640 1 --2- 172.21.15.104:0/2675916199 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f5a5c0052f0 0x7f5a5c00b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:45.088 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:45.082+0000 7f5a67fff640 1 -- 172.21.15.104:0/2675916199 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f5a70073250 con 0x7f5a5c004810 2024-03-28T14:08:45.088 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:45.082+0000 7f5a67fff640 1 -- 172.21.15.104:0/2675916199 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f5a5c00c1c0 con 0x7f5a5c0052f0 2024-03-28T14:08:45.088 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:45.082+0000 7f5a67fff640 1 -- 172.21.15.104:0/2675916199 --> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] -- mon_getmap magic: 0 -- 0x7f5a5c00c3c0 con 0x7f5a5c0039c0 2024-03-28T14:08:45.088 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:45.082+0000 7f5a74f70640 1 --2- 172.21.15.104:0/2675916199 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5a5c004810 0x7f5a5c004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:45.088 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:45.082+0000 7f5a75771640 1 --2- 172.21.15.104:0/2675916199 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f5a5c0039c0 0x7f5a5c003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:45.089 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:45.082+0000 7f5a74f70640 1 -- 172.21.15.104:0/2675916199 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f5a5c0039c0 msgr2=0x7f5a5c003e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T14:08:45.089 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:45.082+0000 7f5a74f70640 1 --2- 172.21.15.104:0/2675916199 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f5a5c0039c0 0x7f5a5c003e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:45.089 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:45.082+0000 7f5a74f70640 1 -- 172.21.15.104:0/2675916199 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f5a5c0052f0 msgr2=0x7f5a5c00b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T14:08:45.089 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:45.082+0000 7f5a74f70640 1 --2- 172.21.15.104:0/2675916199 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f5a5c0052f0 0x7f5a5c00b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:45.089 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:45.082+0000 7f5a74f70640 1 -- 172.21.15.104:0/2675916199 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5a5c003530 con 0x7f5a5c004810 2024-03-28T14:08:45.089 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:45.082+0000 7f5a75f72640 1 -- 172.21.15.104:0/2675916199 reap_dead start 2024-03-28T14:08:45.090 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:45.086+0000 7f5a74f70640 1 --2- 172.21.15.104:0/2675916199 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5a5c004810 0x7f5a5c004c90 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f5a60009ae0 tx=0x7f5a6001ca90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f4cec602131c0fd3 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:08:45.090 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:45.086+0000 7f5a67fff640 1 -- 172.21.15.104:0/2675916199 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a600096b0 con 0x7f5a5c004810 2024-03-28T14:08:45.090 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:45.086+0000 7f5a67fff640 1 -- 172.21.15.104:0/2675916199 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5a600a6470 con 0x7f5a5c004810 2024-03-28T14:08:45.090 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:45.086+0000 7f5a67fff640 1 -- 172.21.15.104:0/2675916199 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a600a6860 con 0x7f5a5c004810 2024-03-28T14:08:45.090 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:45.086+0000 7f5a779d5640 1 -- 172.21.15.104:0/2675916199 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5a5c004810 msgr2=0x7f5a5c004c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:45.090 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:45.086+0000 7f5a779d5640 1 --2- 172.21.15.104:0/2675916199 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5a5c004810 0x7f5a5c004c90 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f5a60009ae0 tx=0x7f5a6001ca90 comp rx=0 tx=0).stop 2024-03-28T14:08:45.091 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:45.086+0000 7f5a779d5640 1 -- 172.21.15.104:0/2675916199 shutdown_connections 2024-03-28T14:08:45.091 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:45.086+0000 7f5a779d5640 1 --2- 172.21.15.104:0/2675916199 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5a5c004810 0x7f5a5c004c90 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:45.091 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:45.086+0000 7f5a779d5640 1 -- 172.21.15.104:0/2675916199 >> 172.21.15.104:0/2675916199 conn(0x7f5a700b7b10 msgr2=0x7f5a700b7f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T14:08:45.091 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:45.086+0000 7f5a779d5640 1 -- 172.21.15.104:0/2675916199 shutdown_connections 2024-03-28T14:08:45.091 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:45.086+0000 7f5a779d5640 1 -- 172.21.15.104:0/2675916199 wait complete. 2024-03-28T14:08:45.093 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:45.086+0000 7f5a779d5640 1 Processor -- start 2024-03-28T14:08:45.093 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:45.086+0000 7f5a779d5640 1 -- start start 2024-03-28T14:08:45.093 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:45.090+0000 7f5a779d5640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f5a5c004810 0x7f5a70147880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:45.093 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:45.090+0000 7f5a779d5640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5a701e07b0 0x7f5a7009ee20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:45.093 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:45.090+0000 7f5a779d5640 1 --2- >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f5a701e0c30 0x7f5a7009f360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:45.093 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:45.090+0000 7f5a779d5640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f5a70148ce0 con 0x7f5a701e07b0 2024-03-28T14:08:45.093 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:45.090+0000 7f5a779d5640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f5a70148b60 con 0x7f5a5c004810 2024-03-28T14:08:45.093 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:45.090+0000 7f5a779d5640 1 -- --> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] -- mon_getmap magic: 0 -- 0x7f5a70148e60 con 0x7f5a701e0c30 2024-03-28T14:08:45.093 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:45.090+0000 7f5a74f70640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5a701e07b0 0x7f5a7009ee20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:45.093 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:45.090+0000 7f5a74f70640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5a701e07b0 0x7f5a7009ee20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:50482/0 (socket says 172.21.15.104:50482) 2024-03-28T14:08:45.094 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:45.090+0000 7f5a74f70640 1 -- 172.21.15.104:0/908528171 learned_addr learned my addr 172.21.15.104:0/908528171 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-03-28T14:08:45.094 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:45.090+0000 7f5a75f72640 1 --2- 172.21.15.104:0/908528171 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f5a701e0c30 0x7f5a7009f360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:45.094 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:45.090+0000 7f5a74f70640 1 -- 172.21.15.104:0/908528171 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f5a701e0c30 msgr2=0x7f5a7009f360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:45.094 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:45.090+0000 7f5a75771640 1 --2- 172.21.15.104:0/908528171 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f5a5c004810 0x7f5a70147880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:45.094 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:45.090+0000 7f5a74f70640 1 --2- 172.21.15.104:0/908528171 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f5a701e0c30 0x7f5a7009f360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:45.094 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:45.090+0000 7f5a74f70640 1 -- 172.21.15.104:0/908528171 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f5a5c004810 msgr2=0x7f5a70147880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:45.094 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:45.090+0000 7f5a74f70640 1 --2- 172.21.15.104:0/908528171 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f5a5c004810 0x7f5a70147880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:45.094 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:45.090+0000 7f5a74f70640 1 -- 172.21.15.104:0/908528171 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5a7006d5b0 con 0x7f5a701e07b0 2024-03-28T14:08:45.094 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:45.090+0000 7f5a74f70640 1 --2- 172.21.15.104:0/908528171 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5a701e07b0 0x7f5a7009ee20 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f5a60009ae0 tx=0x7f5a60019830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:08:45.094 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:45.090+0000 7f5a65ffb640 1 -- 172.21.15.104:0/908528171 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a600a7450 con 0x7f5a701e07b0 2024-03-28T14:08:45.094 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:45.090+0000 7f5a65ffb640 1 -- 172.21.15.104:0/908528171 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5a600a75f0 con 0x7f5a701e07b0 2024-03-28T14:08:45.094 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:45.090+0000 7f5a779d5640 1 -- 172.21.15.104:0/908528171 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5a7009ff00 con 0x7f5a701e07b0 2024-03-28T14:08:45.094 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:45.090+0000 7f5a65ffb640 1 -- 172.21.15.104:0/908528171 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a600a79c0 con 0x7f5a701e07b0 2024-03-28T14:08:45.095 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:45.090+0000 7f5a779d5640 1 -- 172.21.15.104:0/908528171 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5a70143250 con 0x7f5a701e07b0 2024-03-28T14:08:45.095 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:45.090+0000 7f5a779d5640 1 -- 172.21.15.104:0/908528171 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5a700687c0 con 0x7f5a701e07b0 2024-03-28T14:08:45.095 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:45.090+0000 7f5a65ffb640 1 -- 172.21.15.104:0/908528171 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) ==== 87803+0+0 (secure 0 0 0) 0x7f5a600c1060 con 0x7f5a701e07b0 2024-03-28T14:08:45.095 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:45.090+0000 7f5a65ffb640 1 --2- 172.21.15.104:0/908528171 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f5a54067fc0 0x7f5a5406a480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:45.095 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:45.090+0000 7f5a65ffb640 1 -- 172.21.15.104:0/908528171 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 5709+0+0 (secure 0 0 0) 0x7f5a60120600 con 0x7f5a701e07b0 2024-03-28T14:08:45.096 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:45.090+0000 7f5a75771640 1 --2- 172.21.15.104:0/908528171 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f5a54067fc0 0x7f5a5406a480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:45.096 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:45.094+0000 7f5a75771640 1 --2- 172.21.15.104:0/908528171 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f5a54067fc0 0x7f5a5406a480 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f5a68007fd0 tx=0x7f5a68009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:08:45.102 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:45.098+0000 7f5a65ffb640 1 -- 172.21.15.104:0/908528171 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179117 (secure 0 0 0) 0x7f5a600b5070 con 0x7f5a701e07b0 2024-03-28T14:08:45.308 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:45.302+0000 7f5a779d5640 1 -- 172.21.15.104:0/908528171 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) -- 0x7f5a70072fd0 con 0x7f5a701e07b0 2024-03-28T14:08:45.310 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:45.306+0000 7f5a65ffb640 1 -- 172.21.15.104:0/908528171 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v11) ==== 190+0+147 (secure 0 0 0) 0x7f5a600ac020 con 0x7f5a701e07b0 2024-03-28T14:08:45.311 INFO:teuthology.orchestra.run.smithi104.stdout:[client.0] 2024-03-28T14:08:45.311 INFO:teuthology.orchestra.run.smithi104.stdout: key = AQCFeQVmkAgiGhAAUVl6HwWo1xt2L4bwVt1VWw== 2024-03-28T14:08:45.311 INFO:teuthology.orchestra.run.smithi104.stdout: caps mds = "allow" 2024-03-28T14:08:45.311 INFO:teuthology.orchestra.run.smithi104.stdout: caps mon = "allow r" 2024-03-28T14:08:45.311 INFO:teuthology.orchestra.run.smithi104.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2024-03-28T14:08:45.311 INFO:teuthology.orchestra.run.smithi104.stderr:updated caps for client.0 2024-03-28T14:08:45.313 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:45.310+0000 7f5a779d5640 1 -- 172.21.15.104:0/908528171 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f5a54067fc0 msgr2=0x7f5a5406a480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:45.313 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:45.310+0000 7f5a779d5640 1 --2- 172.21.15.104:0/908528171 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f5a54067fc0 0x7f5a5406a480 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f5a68007fd0 tx=0x7f5a68009040 comp rx=0 tx=0).stop 2024-03-28T14:08:45.313 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:45.310+0000 7f5a779d5640 1 -- 172.21.15.104:0/908528171 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5a701e07b0 msgr2=0x7f5a7009ee20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:45.313 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:45.310+0000 7f5a779d5640 1 --2- 172.21.15.104:0/908528171 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5a701e07b0 0x7f5a7009ee20 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f5a60009ae0 tx=0x7f5a60019830 comp rx=0 tx=0).stop 2024-03-28T14:08:45.313 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:45.310+0000 7f5a779d5640 1 -- 172.21.15.104:0/908528171 shutdown_connections 2024-03-28T14:08:45.313 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:45.310+0000 7f5a779d5640 1 --2- 172.21.15.104:0/908528171 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f5a701e0c30 0x7f5a7009f360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:45.313 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:45.310+0000 7f5a779d5640 1 --2- 172.21.15.104:0/908528171 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f5a54067fc0 0x7f5a5406a480 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:45.313 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:45.310+0000 7f5a779d5640 1 --2- 172.21.15.104:0/908528171 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5a701e07b0 0x7f5a7009ee20 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:45.313 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:45.310+0000 7f5a779d5640 1 --2- 172.21.15.104:0/908528171 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f5a5c004810 0x7f5a70147880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:45.313 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:45.310+0000 7f5a779d5640 1 -- 172.21.15.104:0/908528171 >> 172.21.15.104:0/908528171 conn(0x7f5a700b7b10 msgr2=0x7f5a7006e9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T14:08:45.314 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:45.310+0000 7f5a779d5640 1 -- 172.21.15.104:0/908528171 shutdown_connections 2024-03-28T14:08:45.314 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:45.310+0000 7f5a779d5640 1 -- 172.21.15.104:0/908528171 wait complete. 2024-03-28T14:08:45.328 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T14:08:45.436 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:45.434+0000 7f31c4c1c640 1 Processor -- start 2024-03-28T14:08:45.436 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:45.434+0000 7f31c4c1c640 1 -- start start 2024-03-28T14:08:45.437 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:45.434+0000 7f31c4c1c640 1 --2- >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f31c00b6330 0x7f31c0144110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:45.437 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:45.434+0000 7f31c4c1c640 1 --2- >> v2:172.21.15.171:3300/0 conn(0x7f31c00b9180 0x7f31c0144650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:45.437 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:45.434+0000 7f31c4c1c640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f31c00b6710 con 0x7f31c01432c0 2024-03-28T14:08:45.437 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:45.434+0000 7f31c4c1c640 1 -- --> v2:172.21.15.171:3300/0 -- mon_getmap magic: 0 -- 0x7f31c00b4190 con 0x7f31c00b9180 2024-03-28T14:08:45.438 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:45.434+0000 7f31c4c1c640 1 -- --> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] -- mon_getmap magic: 0 -- 0x7f31c00b4f40 con 0x7f31c00b6330 2024-03-28T14:08:45.438 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:45.434+0000 7f31bed9d640 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f31c01432c0 0x7f31c0143720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:59864/0 (socket says 172.21.15.104:59864) 2024-03-28T14:08:45.438 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:45.434+0000 7f31bed9d640 1 -- 172.21.15.104:0/172367394 learned_addr learned my addr 172.21.15.104:0/172367394 (peer_addr_for_me v1:172.21.15.104:0/0) 2024-03-28T14:08:45.438 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:45.434+0000 7f31be59c640 1 --2- 172.21.15.104:0/172367394 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f31c00b6330 0x7f31c0144110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:45.438 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:45.434+0000 7f31bd59a640 1 -- 172.21.15.104:0/172367394 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3815567697 0 0) 0x7f31c00b6710 con 0x7f31c01432c0 2024-03-28T14:08:45.438 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:45.434+0000 7f31bdd9b640 1 --2- 172.21.15.104:0/172367394 >> v2:172.21.15.171:3300/0 conn(0x7f31c00b9180 0x7f31c0144650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:45.438 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:45.434+0000 7f31bd59a640 1 -- 172.21.15.104:0/172367394 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f31a8003530 con 0x7f31c01432c0 2024-03-28T14:08:45.439 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:45.434+0000 7f31bd59a640 1 -- 172.21.15.104:0/172367394 <== mon.0 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1698466094 0 0) 0x7f31a8003530 con 0x7f31c01432c0 2024-03-28T14:08:45.440 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:45.434+0000 7f31bd59a640 1 -- 172.21.15.104:0/172367394 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f31c00b6330 msgr2=0x7f31c0144110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T14:08:45.440 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:45.434+0000 7f31bd59a640 1 --2- 172.21.15.104:0/172367394 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f31c00b6330 0x7f31c0144110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:45.440 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:45.434+0000 7f31bd59a640 1 -- 172.21.15.104:0/172367394 >> v2:172.21.15.171:3300/0 conn(0x7f31c00b9180 msgr2=0x7f31c0144650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T14:08:45.440 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:45.434+0000 7f31bd59a640 1 --2- 172.21.15.104:0/172367394 >> v2:172.21.15.171:3300/0 conn(0x7f31c00b9180 0x7f31c0144650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:45.440 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:45.434+0000 7f31bd59a640 1 -- 172.21.15.104:0/172367394 --> v1:172.21.15.104:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f31c00b5220 con 0x7f31c01432c0 2024-03-28T14:08:45.440 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:45.434+0000 7f31bd59a640 1 -- 172.21.15.104:0/172367394 <== mon.0 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3745580849 0 0) 0x7f31b8001680 con 0x7f31c01432c0 2024-03-28T14:08:45.440 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:45.434+0000 7f31bd59a640 1 -- 172.21.15.104:0/172367394 >> v1:172.21.15.104:6789/0 conn(0x7f31c01432c0 legacy=0x7f31c0143720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:45.440 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:45.434+0000 7f31bd59a640 1 --2- 172.21.15.104:0/172367394 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f31a80039f0 0x7f31a8003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:45.440 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:45.434+0000 7f31bd59a640 1 --2- 172.21.15.104:0/172367394 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f31c00b6330 0x7f31a8004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:45.440 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:45.434+0000 7f31bd59a640 1 --2- 172.21.15.104:0/172367394 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f31a8004d50 0x7f31a80051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:45.440 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:45.434+0000 7f31bd59a640 1 -- 172.21.15.104:0/172367394 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f31c00b4190 con 0x7f31a8004d50 2024-03-28T14:08:45.440 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:45.434+0000 7f31bd59a640 1 -- 172.21.15.104:0/172367394 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f31c00b4f40 con 0x7f31a80039f0 2024-03-28T14:08:45.440 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:45.434+0000 7f31bd59a640 1 -- 172.21.15.104:0/172367394 --> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] -- mon_getmap magic: 0 -- 0x7f31a800ba00 con 0x7f31c00b6330 2024-03-28T14:08:45.441 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:45.434+0000 7f31bed9d640 1 --2- 172.21.15.104:0/172367394 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f31a8004d50 0x7f31a80051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:45.441 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:45.434+0000 7f31be59c640 1 --2- 172.21.15.104:0/172367394 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f31a80039f0 0x7f31a8003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:45.441 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:45.434+0000 7f31bdd9b640 1 --2- 172.21.15.104:0/172367394 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f31c00b6330 0x7f31a8004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:45.441 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:45.434+0000 7f31bed9d640 1 -- 172.21.15.104:0/172367394 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f31c00b6330 msgr2=0x7f31a8004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T14:08:45.441 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:45.434+0000 7f31bed9d640 1 --2- 172.21.15.104:0/172367394 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f31c00b6330 0x7f31a8004810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:45.441 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:45.434+0000 7f31bed9d640 1 -- 172.21.15.104:0/172367394 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f31a80039f0 msgr2=0x7f31a8003e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T14:08:45.441 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:45.434+0000 7f31bed9d640 1 --2- 172.21.15.104:0/172367394 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f31a80039f0 0x7f31a8003e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:45.441 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:45.434+0000 7f31bed9d640 1 -- 172.21.15.104:0/172367394 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f31a8003530 con 0x7f31a8004d50 2024-03-28T14:08:45.441 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:45.434+0000 7f31bdd9b640 1 --2- 172.21.15.104:0/172367394 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f31c00b6330 0x7f31a8004810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T14:08:45.444 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:45.438+0000 7f31bed9d640 1 --2- 172.21.15.104:0/172367394 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f31a8004d50 0x7f31a80051d0 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f31b800c210 tx=0x7f31b801f3f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8d91580bcc13f43a server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:08:45.444 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:45.438+0000 7f31bd59a640 1 -- 172.21.15.104:0/172367394 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31b8007d40 con 0x7f31a8004d50 2024-03-28T14:08:45.444 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:45.438+0000 7f31bd59a640 1 -- 172.21.15.104:0/172367394 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f31b8005b00 con 0x7f31a8004d50 2024-03-28T14:08:45.444 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:45.438+0000 7f31bd59a640 1 -- 172.21.15.104:0/172367394 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31b8007070 con 0x7f31a8004d50 2024-03-28T14:08:45.444 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:45.438+0000 7f31c4c1c640 1 -- 172.21.15.104:0/172367394 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f31a8004d50 msgr2=0x7f31a80051d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:45.444 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:45.438+0000 7f31c4c1c640 1 --2- 172.21.15.104:0/172367394 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f31a8004d50 0x7f31a80051d0 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f31b800c210 tx=0x7f31b801f3f0 comp rx=0 tx=0).stop 2024-03-28T14:08:45.444 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:45.438+0000 7f31bed9d640 1 -- 172.21.15.104:0/172367394 reap_dead start 2024-03-28T14:08:45.444 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:45.438+0000 7f31c4c1c640 1 -- 172.21.15.104:0/172367394 shutdown_connections 2024-03-28T14:08:45.444 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:45.438+0000 7f31c4c1c640 1 -- 172.21.15.104:0/172367394 >> 172.21.15.104:0/172367394 conn(0x7f31c00b9660 msgr2=0x7f31c00b9a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T14:08:45.444 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:45.438+0000 7f31c4c1c640 1 -- 172.21.15.104:0/172367394 shutdown_connections 2024-03-28T14:08:45.445 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:45.438+0000 7f31c4c1c640 1 -- 172.21.15.104:0/172367394 wait complete. 2024-03-28T14:08:45.445 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:45.438+0000 7f31c4c1c640 1 Processor -- start 2024-03-28T14:08:45.445 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:45.438+0000 7f31c4c1c640 1 -- start start 2024-03-28T14:08:45.445 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:45.438+0000 7f31c4c1c640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f31a8004d50 0x7f31c013fb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:45.445 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:45.438+0000 7f31c4c1c640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f31c01dfd60 0x7f31c00b2a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:45.445 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:45.438+0000 7f31c4c1c640 1 --2- >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f31c01e01e0 0x7f31c00b2fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:45.445 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:45.438+0000 7f31c4c1c640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f31c009ebc0 con 0x7f31a8004d50 2024-03-28T14:08:45.445 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:45.438+0000 7f31c4c1c640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f31c009ea40 con 0x7f31c01dfd60 2024-03-28T14:08:45.445 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:45.438+0000 7f31c4c1c640 1 -- --> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] -- mon_getmap magic: 0 -- 0x7f31c00b7460 con 0x7f31c01e01e0 2024-03-28T14:08:45.445 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:45.438+0000 7f31be59c640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f31a8004d50 0x7f31c013fb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:45.445 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:45.438+0000 7f31be59c640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f31a8004d50 0x7f31c013fb30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:50508/0 (socket says 172.21.15.104:50508) 2024-03-28T14:08:45.445 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:45.438+0000 7f31be59c640 1 -- 172.21.15.104:0/1504986019 learned_addr learned my addr 172.21.15.104:0/1504986019 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-03-28T14:08:45.445 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:45.438+0000 7f31be59c640 1 -- 172.21.15.104:0/1504986019 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f31c01e01e0 msgr2=0x7f31c00b2fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:45.445 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:45.438+0000 7f31be59c640 1 --2- 172.21.15.104:0/1504986019 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f31c01e01e0 0x7f31c00b2fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:45.446 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:45.438+0000 7f31be59c640 1 -- 172.21.15.104:0/1504986019 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f31c01dfd60 msgr2=0x7f31c00b2a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:45.446 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:45.438+0000 7f31be59c640 1 --2- 172.21.15.104:0/1504986019 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f31c01dfd60 0x7f31c00b2a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:45.446 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:45.438+0000 7f31be59c640 1 -- 172.21.15.104:0/1504986019 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f31c0072750 con 0x7f31a8004d50 2024-03-28T14:08:45.446 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:45.438+0000 7f31be59c640 1 --2- 172.21.15.104:0/1504986019 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f31a8004d50 0x7f31c013fb30 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f31b000f380 tx=0x7f31b000f840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:08:45.446 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:45.442+0000 7f31aeffd640 1 -- 172.21.15.104:0/1504986019 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31b001c070 con 0x7f31a8004d50 2024-03-28T14:08:45.446 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:45.442+0000 7f31aeffd640 1 -- 172.21.15.104:0/1504986019 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f31b0017620 con 0x7f31a8004d50 2024-03-28T14:08:45.446 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:45.442+0000 7f31c4c1c640 1 -- 172.21.15.104:0/1504986019 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f31c01401c0 con 0x7f31a8004d50 2024-03-28T14:08:45.446 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:45.442+0000 7f31aeffd640 1 -- 172.21.15.104:0/1504986019 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31b0017a10 con 0x7f31a8004d50 2024-03-28T14:08:45.446 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:45.442+0000 7f31c4c1c640 1 -- 172.21.15.104:0/1504986019 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f31c01454b0 con 0x7f31a8004d50 2024-03-28T14:08:45.447 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:45.442+0000 7f31c4c1c640 1 -- 172.21.15.104:0/1504986019 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f31c00687c0 con 0x7f31a8004d50 2024-03-28T14:08:45.447 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:45.442+0000 7f31aeffd640 1 -- 172.21.15.104:0/1504986019 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) ==== 87803+0+0 (secure 0 0 0) 0x7f31b001f070 con 0x7f31a8004d50 2024-03-28T14:08:45.447 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:45.442+0000 7f31aeffd640 1 --2- 172.21.15.104:0/1504986019 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f31a0067f30 0x7f31a006a3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:45.447 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:45.442+0000 7f31aeffd640 1 -- 172.21.15.104:0/1504986019 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 5709+0+0 (secure 0 0 0) 0x7f31b0086a70 con 0x7f31a8004d50 2024-03-28T14:08:45.447 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:45.442+0000 7f31bdd9b640 1 --2- 172.21.15.104:0/1504986019 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f31a0067f30 0x7f31a006a3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:45.447 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:45.442+0000 7f31bdd9b640 1 --2- 172.21.15.104:0/1504986019 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f31a0067f30 0x7f31a006a3f0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f31c00b7120 tx=0x7f31b4008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:08:45.450 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:45.446+0000 7f31aeffd640 1 -- 172.21.15.104:0/1504986019 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179117 (secure 0 0 0) 0x7f31b0012070 con 0x7f31a8004d50 2024-03-28T14:08:45.642 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:45.638+0000 7f31c4c1c640 1 -- 172.21.15.104:0/1504986019 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f31c00b84c0 con 0x7f31a8004d50 2024-03-28T14:08:45.643 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:45.638+0000 7f31aeffd640 1 -- 172.21.15.104:0/1504986019 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 14 v14) ==== 94+0+3537 (secure 0 0 0) 0x7f31b005d030 con 0x7f31a8004d50 2024-03-28T14:08:45.643 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-03-28T14:08:45.644 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":14,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4550,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.171:6835/3418793776","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6834","nonce":3418793776},{"type":"v1","addr":"172.21.15.171:6835","nonce":3418793776}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11}],"filesystems":[{"mdsmap":{"epoch":14,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T14:08:40.854108+0000","modified":"2024-03-28T14:08:42.859657+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4959},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4959":{"gid":4959,"name":"a","rank":0,"incarnation":12,"state":"up:active","state_seq":3,"addr":"172.21.15.104:6833/1094990707","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":1094990707},{"type":"v1","addr":"172.21.15.104:6833","nonce":1094990707}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4959,"qdb_cluster":[4959]},"id":2}]} 2024-03-28T14:08:45.644 INFO:teuthology.orchestra.run.smithi104.stderr:dumped fsmap epoch 14 2024-03-28T14:08:45.645 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:45.642+0000 7f31c4c1c640 1 -- 172.21.15.104:0/1504986019 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f31a0067f30 msgr2=0x7f31a006a3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:45.645 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:45.642+0000 7f31c4c1c640 1 --2- 172.21.15.104:0/1504986019 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f31a0067f30 0x7f31a006a3f0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f31c00b7120 tx=0x7f31b4008040 comp rx=0 tx=0).stop 2024-03-28T14:08:45.646 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:45.642+0000 7f31c4c1c640 1 -- 172.21.15.104:0/1504986019 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f31a8004d50 msgr2=0x7f31c013fb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:45.646 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:45.642+0000 7f31c4c1c640 1 --2- 172.21.15.104:0/1504986019 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f31a8004d50 0x7f31c013fb30 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f31b000f380 tx=0x7f31b000f840 comp rx=0 tx=0).stop 2024-03-28T14:08:45.646 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:45.642+0000 7f31c4c1c640 1 -- 172.21.15.104:0/1504986019 shutdown_connections 2024-03-28T14:08:45.646 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:45.642+0000 7f31c4c1c640 1 --2- 172.21.15.104:0/1504986019 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f31c01e01e0 0x7f31c00b2fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:45.646 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:45.642+0000 7f31c4c1c640 1 --2- 172.21.15.104:0/1504986019 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f31c01dfd60 0x7f31c00b2a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:45.646 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:45.642+0000 7f31c4c1c640 1 --2- 172.21.15.104:0/1504986019 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f31a0067f30 0x7f31a006a3f0 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:45.646 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:45.642+0000 7f31c4c1c640 1 --2- 172.21.15.104:0/1504986019 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f31a8004d50 0x7f31c013fb30 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:45.646 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:45.642+0000 7f31c4c1c640 1 -- 172.21.15.104:0/1504986019 >> 172.21.15.104:0/1504986019 conn(0x7f31c00b9660 msgr2=0x7f31c00b7790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T14:08:45.646 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:45.642+0000 7f31c4c1c640 1 -- 172.21.15.104:0/1504986019 shutdown_connections 2024-03-28T14:08:45.646 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:45.642+0000 7f31c4c1c640 1 -- 172.21.15.104:0/1504986019 wait complete. 2024-03-28T14:08:45.861 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 14, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-28T14:08:40.854108+0000', 'modified': '2024-03-28T14:08:42.859657+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 4959}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4959': {'gid': 4959, 'name': 'a', 'rank': 0, 'incarnation': 12, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.104:6833/1094990707', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.104:6832', 'nonce': 1094990707}, {'type': 'v1', 'addr': '172.21.15.104:6833', 'nonce': 1094990707}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [5], 'metadata_pool': 4, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4959, 'qdb_cluster': [4959]} 2024-03-28T14:08:45.861 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-03-28T14:08:45.862 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2024-03-28T14:08:45.983 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:45.978+0000 7fd30b974640 1 Processor -- start 2024-03-28T14:08:45.984 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:45.978+0000 7fd30b974640 1 -- start start 2024-03-28T14:08:45.984 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:45.978+0000 7fd30b974640 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fd3040b51e0 0x7fd30409c450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:45.984 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:45.978+0000 7fd30b974640 1 --2- >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fd30409aa10 0x7fd304073180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:45.984 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:45.978+0000 7fd30b974640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd30406e0f0 con 0x7fd30406dd10 2024-03-28T14:08:45.984 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:45.978+0000 7fd30b974640 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 -- 0x7fd3040b6370 con 0x7fd3040b51e0 2024-03-28T14:08:45.984 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:45.978+0000 7fd30b974640 1 -- --> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] -- mon_getmap magic: 0 -- 0x7fd30409ae80 con 0x7fd30409aa10 2024-03-28T14:08:45.984 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:45.978+0000 7fd308f0f640 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fd3040b51e0 0x7fd30409c450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:45.985 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:45.978+0000 7fd309710640 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7fd30406dd10 0x7fd30413f6e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:59866/0 (socket says 172.21.15.104:59866) 2024-03-28T14:08:45.985 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:45.978+0000 7fd309710640 1 -- 172.21.15.104:0/712542985 learned_addr learned my addr 172.21.15.104:0/712542985 (peer_addr_for_me v1:172.21.15.104:0/0) 2024-03-28T14:08:45.985 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:45.978+0000 7fd309f11640 1 --2- 172.21.15.104:0/712542985 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fd30409aa10 0x7fd304073180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:45.985 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:45.978+0000 7fd308f0f640 1 -- 172.21.15.104:0/712542985 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fd30409aa10 msgr2=0x7fd304073180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T14:08:45.985 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:45.978+0000 7fd308f0f640 1 --2- 172.21.15.104:0/712542985 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fd30409aa10 0x7fd304073180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:45.985 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:45.978+0000 7fd308f0f640 1 -- 172.21.15.104:0/712542985 >> v1:172.21.15.104:6789/0 conn(0x7fd30406dd10 legacy=0x7fd30413f6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:45.985 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:45.978+0000 7fd308f0f640 1 -- 172.21.15.104:0/712542985 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd3040b5d20 con 0x7fd3040b51e0 2024-03-28T14:08:45.985 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:45.978+0000 7fd2f7fff640 1 -- 172.21.15.104:0/712542985 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3987912495 0 0) 0x7fd30406e0f0 con 0x7fd30406dd10 2024-03-28T14:08:45.985 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:45.978+0000 7fd309f11640 1 --2- 172.21.15.104:0/712542985 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fd30409aa10 0x7fd304073180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T14:08:45.989 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:45.982+0000 7fd308f0f640 1 --2- 172.21.15.104:0/712542985 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd3040b51e0 0x7fd30409c450 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7fd3000097c0 tx=0x7fd30001c410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6f2f7a32ef432609 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:08:45.989 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:45.982+0000 7fd2f7fff640 1 -- 172.21.15.104:0/712542985 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3000b0070 con 0x7fd3040b51e0 2024-03-28T14:08:45.989 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:45.982+0000 7fd2f7fff640 1 -- 172.21.15.104:0/712542985 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd3000ab600 con 0x7fd3040b51e0 2024-03-28T14:08:45.989 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:45.982+0000 7fd2f7fff640 1 -- 172.21.15.104:0/712542985 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3000ab9d0 con 0x7fd3040b51e0 2024-03-28T14:08:45.989 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:45.982+0000 7fd30b974640 1 -- 172.21.15.104:0/712542985 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd3040b51e0 msgr2=0x7fd30409c450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:45.989 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:45.982+0000 7fd30b974640 1 --2- 172.21.15.104:0/712542985 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd3040b51e0 0x7fd30409c450 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7fd3000097c0 tx=0x7fd30001c410 comp rx=0 tx=0).stop 2024-03-28T14:08:45.989 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:45.982+0000 7fd30b974640 1 -- 172.21.15.104:0/712542985 shutdown_connections 2024-03-28T14:08:45.989 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:45.982+0000 7fd30b974640 1 --2- 172.21.15.104:0/712542985 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fd30409aa10 0x7fd304073180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:45.990 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:45.982+0000 7fd30b974640 1 --2- 172.21.15.104:0/712542985 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd3040b51e0 0x7fd30409c450 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:45.990 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:45.982+0000 7fd30b974640 1 -- 172.21.15.104:0/712542985 >> 172.21.15.104:0/712542985 conn(0x7fd3040ba390 msgr2=0x7fd3040ba7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T14:08:45.990 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:45.982+0000 7fd30b974640 1 -- 172.21.15.104:0/712542985 shutdown_connections 2024-03-28T14:08:45.990 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:45.982+0000 7fd30b974640 1 -- 172.21.15.104:0/712542985 wait complete. 2024-03-28T14:08:45.990 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:45.982+0000 7fd30b974640 1 Processor -- start 2024-03-28T14:08:45.990 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:45.982+0000 7fd30b974640 1 -- start start 2024-03-28T14:08:45.990 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:45.982+0000 7fd30b974640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fd30406dd10 0x7fd3040af880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:45.990 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:45.982+0000 7fd30b974640 1 --2- >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fd30409aa10 0x7fd30413c1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:45.990 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:45.982+0000 7fd30b974640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd3040b51e0 0x7fd3041dfd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:45.990 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:45.982+0000 7fd30b974640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7fd304152b10 con 0x7fd3040b51e0 2024-03-28T14:08:45.990 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:45.982+0000 7fd30b974640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7fd304140000 con 0x7fd30406dd10 2024-03-28T14:08:45.991 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:45.982+0000 7fd30b974640 1 -- --> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] -- mon_getmap magic: 0 -- 0x7fd304152c90 con 0x7fd30409aa10 2024-03-28T14:08:45.991 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:45.982+0000 7fd309f11640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd3040b51e0 0x7fd3041dfd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:45.991 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:45.982+0000 7fd309f11640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd3040b51e0 0x7fd3041dfd30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:50518/0 (socket says 172.21.15.104:50518) 2024-03-28T14:08:45.991 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:45.982+0000 7fd309f11640 1 -- 172.21.15.104:0/435257380 learned_addr learned my addr 172.21.15.104:0/435257380 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-03-28T14:08:45.991 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:45.982+0000 7fd308f0f640 1 --2- 172.21.15.104:0/435257380 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fd30409aa10 0x7fd30413c1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:45.991 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:45.982+0000 7fd309f11640 1 -- 172.21.15.104:0/435257380 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fd30409aa10 msgr2=0x7fd30413c1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:45.991 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:45.982+0000 7fd309f11640 1 --2- 172.21.15.104:0/435257380 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fd30409aa10 0x7fd30413c1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:45.991 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:45.982+0000 7fd309f11640 1 -- 172.21.15.104:0/435257380 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fd30406dd10 msgr2=0x7fd3040af880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:45.991 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:45.982+0000 7fd309f11640 1 --2- 172.21.15.104:0/435257380 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fd30406dd10 0x7fd3040af880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:45.991 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:45.982+0000 7fd309f11640 1 -- 172.21.15.104:0/435257380 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd30414fc70 con 0x7fd3040b51e0 2024-03-28T14:08:45.991 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:45.986+0000 7fd309f11640 1 --2- 172.21.15.104:0/435257380 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd3040b51e0 0x7fd3041dfd30 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7fd2fc00cf20 tx=0x7fd2fc00a7d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:08:45.991 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:45.986+0000 7fd2f5ffb640 1 -- 172.21.15.104:0/435257380 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2fc01a070 con 0x7fd3040b51e0 2024-03-28T14:08:45.992 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:45.986+0000 7fd2f5ffb640 1 -- 172.21.15.104:0/435257380 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd2fc00d840 con 0x7fd3040b51e0 2024-03-28T14:08:45.992 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:45.986+0000 7fd30b974640 1 -- 172.21.15.104:0/435257380 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd3040b0310 con 0x7fd3040b51e0 2024-03-28T14:08:45.992 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:45.986+0000 7fd2f5ffb640 1 -- 172.21.15.104:0/435257380 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2fc00dc30 con 0x7fd3040b51e0 2024-03-28T14:08:45.992 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:45.986+0000 7fd30b974640 1 -- 172.21.15.104:0/435257380 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd30414f9f0 con 0x7fd3040b51e0 2024-03-28T14:08:45.992 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:45.986+0000 7fd2f5ffb640 1 -- 172.21.15.104:0/435257380 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) ==== 87803+0+0 (secure 0 0 0) 0x7fd2fc01d070 con 0x7fd3040b51e0 2024-03-28T14:08:45.999 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:45.986+0000 7fd2f5ffb640 1 --2- 172.21.15.104:0/435257380 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7fd2e8067fc0 0x7fd2e806a480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:45.999 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:45.986+0000 7fd2f5ffb640 1 -- 172.21.15.104:0/435257380 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 5709+0+0 (secure 0 0 0) 0x7fd2fc086dd0 con 0x7fd3040b51e0 2024-03-28T14:08:45.999 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:45.986+0000 7fd309710640 1 --2- 172.21.15.104:0/435257380 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7fd2e8067fc0 0x7fd2e806a480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:45.999 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:45.986+0000 7fd30eaf3000 1 Processor -- start 2024-03-28T14:08:45.999 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:45.986+0000 7fd30eaf3000 1 -- start start 2024-03-28T14:08:45.999 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:45.986+0000 7fd309710640 1 --2- 172.21.15.104:0/435257380 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7fd2e8067fc0 0x7fd2e806a480 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fd2f80015a0 tx=0x7fd2f8007970 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:08:45.999 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:45.986+0000 7fd30eaf3000 1 --2- >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x558986190e00 0x5589861b9fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:46.000 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:45.986+0000 7fd30eaf3000 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5589861b4e10 0x5589861b51f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:46.000 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:45.986+0000 7fd30eaf3000 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5589861b5730 0x5589861c6d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:46.000 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:45.986+0000 7fd30eaf3000 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x558985fdb950 con 0x5589861b4e10 2024-03-28T14:08:46.000 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:45.986+0000 7fd30eaf3000 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x558985f78990 con 0x5589861b5730 2024-03-28T14:08:46.000 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:45.986+0000 7fd30eaf3000 1 -- --> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] -- mon_getmap magic: 0 -- 0x558985fd7900 con 0x558986190e00 2024-03-28T14:08:46.000 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:45.986+0000 7fd308f0f640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5589861b4e10 0x5589861b51f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:46.000 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:45.986+0000 7fd308f0f640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5589861b4e10 0x5589861b51f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:50530/0 (socket says 172.21.15.104:50530) 2024-03-28T14:08:46.000 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:45.986+0000 7fd308f0f640 1 -- 172.21.15.104:0/244143760 learned_addr learned my addr 172.21.15.104:0/244143760 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-03-28T14:08:46.000 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:45.986+0000 7fd309710640 1 --2- 172.21.15.104:0/244143760 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x558986190e00 0x5589861b9fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:46.000 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:45.986+0000 7fd309f11640 1 --2- 172.21.15.104:0/244143760 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5589861b5730 0x5589861c6d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:46.000 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:45.986+0000 7fd308f0f640 1 -- 172.21.15.104:0/244143760 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x558986190e00 msgr2=0x5589861b9fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T14:08:46.001 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:45.986+0000 7fd308f0f640 1 --2- 172.21.15.104:0/244143760 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x558986190e00 0x5589861b9fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:46.001 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:45.986+0000 7fd308f0f640 1 -- 172.21.15.104:0/244143760 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5589861b5730 msgr2=0x5589861c6d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T14:08:46.001 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:45.986+0000 7fd308f0f640 1 --2- 172.21.15.104:0/244143760 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5589861b5730 0x5589861c6d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:46.001 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:45.986+0000 7fd308f0f640 1 -- 172.21.15.104:0/244143760 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558986020100 con 0x5589861b4e10 2024-03-28T14:08:46.001 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:45.986+0000 7fd308f0f640 1 --2- 172.21.15.104:0/244143760 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5589861b4e10 0x5589861b51f0 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7fd300000c10 tx=0x7fd300004740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=52bd7c4d6162f25f server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:08:46.001 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:45.990+0000 7fd2eeffd640 1 -- 172.21.15.104:0/244143760 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3000b0040 con 0x5589861b4e10 2024-03-28T14:08:46.001 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:45.990+0000 7fd2eeffd640 1 -- 172.21.15.104:0/244143760 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd3000ab700 con 0x5589861b4e10 2024-03-28T14:08:46.001 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:45.990+0000 7fd2eeffd640 1 -- 172.21.15.104:0/244143760 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3000abab0 con 0x5589861b4e10 2024-03-28T14:08:46.001 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:45.990+0000 7fd30eaf3000 1 -- 172.21.15.104:0/244143760 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5589861b4e10 msgr2=0x5589861b51f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:46.001 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:45.990+0000 7fd30eaf3000 1 --2- 172.21.15.104:0/244143760 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5589861b4e10 0x5589861b51f0 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7fd300000c10 tx=0x7fd300004740 comp rx=0 tx=0).stop 2024-03-28T14:08:46.001 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:45.990+0000 7fd30eaf3000 1 -- 172.21.15.104:0/244143760 shutdown_connections 2024-03-28T14:08:46.002 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:45.990+0000 7fd30eaf3000 1 --2- 172.21.15.104:0/244143760 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5589861b5730 0x5589861c6d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:46.002 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:45.990+0000 7fd30eaf3000 1 --2- 172.21.15.104:0/244143760 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5589861b4e10 0x5589861b51f0 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:46.002 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:45.990+0000 7fd30eaf3000 1 --2- 172.21.15.104:0/244143760 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x558986190e00 0x5589861b9fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:46.002 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:45.990+0000 7fd30eaf3000 1 -- 172.21.15.104:0/244143760 >> 172.21.15.104:0/244143760 conn(0x558986073b80 msgr2=0x5589861b9890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T14:08:46.002 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:45.990+0000 7fd30eaf3000 1 -- 172.21.15.104:0/244143760 shutdown_connections 2024-03-28T14:08:46.002 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:45.990+0000 7fd30eaf3000 1 -- 172.21.15.104:0/244143760 wait complete. 2024-03-28T14:08:46.002 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:45.990+0000 7fd30eaf3000 1 -- start start 2024-03-28T14:08:46.002 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:45.990+0000 7fd30eaf3000 1 Processor -- start 2024-03-28T14:08:46.002 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:45.990+0000 7fd30eaf3000 1 --2- >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x558986190e00 0x5589861c6d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:46.002 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:45.990+0000 7fd30eaf3000 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5589861b4e10 0x5589861c72a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:46.002 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:45.990+0000 7fd30eaf3000 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5589861b5730 0x55898618bc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:46.002 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:45.990+0000 7fd30eaf3000 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x558985fdff80 con 0x5589861b4e10 2024-03-28T14:08:46.003 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:45.990+0000 7fd30eaf3000 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x558985f993e0 con 0x5589861b5730 2024-03-28T14:08:46.003 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:45.990+0000 7fd30eaf3000 1 -- --> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] -- mon_getmap magic: 0 -- 0x558985fe8060 con 0x558986190e00 2024-03-28T14:08:46.003 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:45.990+0000 7fd30eaf3000 10 client.0 _ll_register_callbacks cb 0x55898603c0f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T14:08:46.003 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:45.990+0000 7fd308f0f640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5589861b4e10 0x5589861c72a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:46.003 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:45.990+0000 7fd308f0f640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5589861b4e10 0x5589861c72a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:50538/0 (socket says 172.21.15.104:50538) 2024-03-28T14:08:46.003 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:45.990+0000 7fd308f0f640 1 -- 172.21.15.104:0/1939338652 learned_addr learned my addr 172.21.15.104:0/1939338652 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-03-28T14:08:46.003 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:45.990+0000 7fd309710640 1 --2- 172.21.15.104:0/1939338652 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x558986190e00 0x5589861c6d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:46.003 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:45.990+0000 7fd308f0f640 1 -- 172.21.15.104:0/1939338652 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x558986190e00 msgr2=0x5589861c6d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T14:08:46.003 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:45.990+0000 7fd309f11640 1 --2- 172.21.15.104:0/1939338652 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5589861b5730 0x55898618bc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:46.003 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:45.990+0000 7fd308f0f640 1 --2- 172.21.15.104:0/1939338652 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x558986190e00 0x5589861c6d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:46.003 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:45.990+0000 7fd308f0f640 1 -- 172.21.15.104:0/1939338652 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5589861b5730 msgr2=0x55898618bc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T14:08:46.004 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:45.990+0000 7fd308f0f640 1 --2- 172.21.15.104:0/1939338652 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5589861b5730 0x55898618bc30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:46.004 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:45.990+0000 7fd308f0f640 1 -- 172.21.15.104:0/1939338652 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558985fa58c0 con 0x5589861b4e10 2024-03-28T14:08:46.004 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:45.990+0000 7fd308f0f640 1 --2- 172.21.15.104:0/1939338652 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5589861b4e10 0x5589861c72a0 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7fd30001a040 tx=0x7fd30001ac50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bdb3d10559f30bde server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:08:46.004 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:45.990+0000 7fd2e3fff640 10 client.0 ms_handle_connect on v2:172.21.15.104:3300/0 2024-03-28T14:08:46.004 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:45.990+0000 7fd2e3fff640 1 -- 172.21.15.104:0/1939338652 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3000b0040 con 0x5589861b4e10 2024-03-28T14:08:46.004 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:45.990+0000 7fd2e3fff640 1 -- 172.21.15.104:0/1939338652 <== mon.0 v2:172.21.15.104:3300/0 2 ==== osd_map(30..30 src has 1..30) ==== 5709+0+0 (secure 0 0 0) 0x7fd3000a4ca0 con 0x5589861b4e10 2024-03-28T14:08:46.004 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:45.990+0000 7fd30eaf3000 1 -- 172.21.15.104:0/1939338652 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd3000ab700 con 0x5589861b4e10 2024-03-28T14:08:46.004 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:45.990+0000 7fd2e3fff640 1 -- 172.21.15.104:0/1939338652 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_get_version_reply(handle=1 version=14) ==== 24+0+0 (secure 0 0 0) 0x7fd3000cf050 con 0x5589861b4e10 2024-03-28T14:08:46.004 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:45.990+0000 7fd30eaf3000 10 client.5142 fetch_fsmap learned FSMap version 14 2024-03-28T14:08:46.004 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:45.990+0000 7fd30eaf3000 1 -- 172.21.15.104:0/1939338652 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,fsmap=14,monmap=2+}) -- 0x558985f63e80 con 0x5589861b4e10 2024-03-28T14:08:46.005 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:45.990+0000 7fd2e3fff640 1 -- 172.21.15.104:0/1939338652 <== mon.0 v2:172.21.15.104:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd3000ab700 con 0x5589861b4e10 2024-03-28T14:08:46.005 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:45.990+0000 7fd2e3fff640 1 -- 172.21.15.104:0/1939338652 <== mon.0 v2:172.21.15.104:3300/0 5 ==== fsmap(e 14) ==== 2284+0+0 (secure 0 0 0) 0x7fd3000d2020 con 0x5589861b4e10 2024-03-28T14:08:46.005 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:45.990+0000 7fd30eaf3000 10 client.5142 fetch_fsmap finished waiting for FSMap version 14 2024-03-28T14:08:46.005 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:45.990+0000 7fd30eaf3000 10 client.5142 resolve_mds: resolved name 'a' to daemon mds.a 2024-03-28T14:08:46.005 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:45.990+0000 7fd30eaf3000 20 client.5142 populate_metadata read hostname 'smithi104' 2024-03-28T14:08:46.005 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:45.990+0000 7fd30eaf3000 1 --2- 172.21.15.104:0/1939338652 >> [v2:172.21.15.104:6832/1094990707,v1:172.21.15.104:6833/1094990707] conn(0x55898618c9a0 0x55898618cd80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:46.005 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:45.990+0000 7fd30eaf3000 4 client.5142 mds_command: new command op to 4959 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T14:08:46.006 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:45.990+0000 7fd30eaf3000 1 -- 172.21.15.104:0/1939338652 --> [v2:172.21.15.104:6832/1094990707,v1:172.21.15.104:6833/1094990707] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x558985f76bc0 con 0x55898618c9a0 2024-03-28T14:08:46.006 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:45.990+0000 7fd309710640 1 --2- 172.21.15.104:0/1939338652 >> [v2:172.21.15.104:6832/1094990707,v1:172.21.15.104:6833/1094990707] conn(0x55898618c9a0 0x55898618cd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:46.006 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:45.990+0000 7fd2e3fff640 10 client.5142 ms_handle_connect on v2:172.21.15.104:6832/1094990707 2024-03-28T14:08:46.006 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:45.990+0000 7fd309710640 1 --2- 172.21.15.104:0/1939338652 >> [v2:172.21.15.104:6832/1094990707,v1:172.21.15.104:6833/1094990707] conn(0x55898618c9a0 0x55898618cd80 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4c8a7bbdfd5b5408 server_cookie=ab74a549c04f20ce in_seq=0 out_seq=0 2024-03-28T14:08:46.006 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:45.994+0000 7fd2e3fff640 1 -- 172.21.15.104:0/1939338652 <== mds.0 v2:172.21.15.104:6832/1094990707 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x558985f76bc0 con 0x55898618c9a0 2024-03-28T14:08:46.006 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:45.994+0000 7fd2e3fff640 10 client.5142 handle_command_reply: tid=0 2024-03-28T14:08:46.007 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:45.994+0000 7fd30eaf3000 1 client.5142 shutdown 2024-03-28T14:08:46.007 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:45.994+0000 7fd30eaf3000 1 -- 172.21.15.104:0/1939338652 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5589861b4e10 msgr2=0x5589861c72a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:46.007 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:45.994+0000 7fd30eaf3000 1 --2- 172.21.15.104:0/1939338652 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5589861b4e10 0x5589861c72a0 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7fd30001a040 tx=0x7fd30001ac50 comp rx=0 tx=0).stop 2024-03-28T14:08:46.007 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:45.994+0000 7fd30eaf3000 1 -- 172.21.15.104:0/1939338652 shutdown_connections 2024-03-28T14:08:46.007 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:45.994+0000 7fd30eaf3000 1 --2- 172.21.15.104:0/1939338652 >> [v2:172.21.15.104:6832/1094990707,v1:172.21.15.104:6833/1094990707] conn(0x55898618c9a0 0x55898618cd80 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:46.007 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:45.994+0000 7fd30eaf3000 1 --2- 172.21.15.104:0/1939338652 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5589861b5730 0x55898618bc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:46.008 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:45.994+0000 7fd30eaf3000 1 --2- 172.21.15.104:0/1939338652 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5589861b4e10 0x5589861c72a0 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:46.008 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:45.994+0000 7fd2e3fff640 0 client.5142 ms_handle_reset on v2:172.21.15.104:6832/1094990707 2024-03-28T14:08:46.008 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:45.994+0000 7fd30eaf3000 1 --2- 172.21.15.104:0/1939338652 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x558986190e00 0x5589861c6d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:46.008 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:45.994+0000 7fd30eaf3000 1 -- 172.21.15.104:0/1939338652 >> 172.21.15.104:0/1939338652 conn(0x558986073b80 msgr2=0x5589861b9890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T14:08:46.008 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:45.994+0000 7fd30eaf3000 1 -- 172.21.15.104:0/1939338652 shutdown_connections 2024-03-28T14:08:46.008 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:45.994+0000 7fd30eaf3000 1 -- 172.21.15.104:0/1939338652 wait complete. 2024-03-28T14:08:46.009 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:45.994+0000 7fd30eaf3000 20 client.5142 trim_cache size 0 max 16384 2024-03-28T14:08:46.013 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:46.010+0000 7fd30eaf3000 1 Processor -- start 2024-03-28T14:08:46.014 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:46.010+0000 7fd30eaf3000 1 -- start start 2024-03-28T14:08:46.027 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:46.010+0000 7fd30eaf3000 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x558986190e00 0x55898618b800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:46.027 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:46.010+0000 7fd30eaf3000 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5589861b4e10 0x5589861ccbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:46.027 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:46.010+0000 7fd30eaf3000 1 --2- >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x5589861b5730 0x5589861cd110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:46.027 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:46.010+0000 7fd30eaf3000 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7fd3041ec6d0 con 0x5589861b4e10 2024-03-28T14:08:46.027 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:46.010+0000 7fd30eaf3000 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7fd3041eb860 con 0x558986190e00 2024-03-28T14:08:46.028 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:46.010+0000 7fd30eaf3000 1 -- --> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] -- mon_getmap magic: 0 -- 0x558985f737b0 con 0x5589861b5730 2024-03-28T14:08:46.028 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:46.010+0000 7fd308f0f640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5589861b4e10 0x5589861ccbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:46.028 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:46.010+0000 7fd308f0f640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5589861b4e10 0x5589861ccbd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:50544/0 (socket says 172.21.15.104:50544) 2024-03-28T14:08:46.028 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:46.010+0000 7fd308f0f640 1 -- 172.21.15.104:0/1552470592 learned_addr learned my addr 172.21.15.104:0/1552470592 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-03-28T14:08:46.028 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:46.010+0000 7fd308f0f640 1 -- 172.21.15.104:0/1552470592 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x5589861b5730 msgr2=0x5589861cd110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T14:08:46.028 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:46.010+0000 7fd308f0f640 1 --2- 172.21.15.104:0/1552470592 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x5589861b5730 0x5589861cd110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:46.029 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:46.010+0000 7fd308f0f640 1 -- 172.21.15.104:0/1552470592 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x558986190e00 msgr2=0x55898618b800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T14:08:46.029 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:46.010+0000 7fd309710640 1 --2- 172.21.15.104:0/1552470592 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x558986190e00 0x55898618b800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:46.029 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:46.010+0000 7fd308f0f640 1 --2- 172.21.15.104:0/1552470592 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x558986190e00 0x55898618b800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:46.029 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:46.010+0000 7fd308f0f640 1 -- 172.21.15.104:0/1552470592 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5589861b5560 con 0x5589861b4e10 2024-03-28T14:08:46.029 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:46.010+0000 7fd308f0f640 1 --2- 172.21.15.104:0/1552470592 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5589861b4e10 0x5589861ccbd0 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7fd300019020 tx=0x7fd30000b050 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6e3fe4cf90900bd0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:08:46.030 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:46.010+0000 7fd2ee7fc640 1 -- 172.21.15.104:0/1552470592 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3000ab600 con 0x5589861b4e10 2024-03-28T14:08:46.030 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:46.010+0000 7fd2ee7fc640 1 -- 172.21.15.104:0/1552470592 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd300000e50 con 0x5589861b4e10 2024-03-28T14:08:46.030 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:46.010+0000 7fd2ee7fc640 1 -- 172.21.15.104:0/1552470592 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd300004240 con 0x5589861b4e10 2024-03-28T14:08:46.030 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:46.014+0000 7fd30eaf3000 1 -- 172.21.15.104:0/1552470592 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5589861b4e10 msgr2=0x5589861ccbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:46.030 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:46.014+0000 7fd30eaf3000 1 --2- 172.21.15.104:0/1552470592 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5589861b4e10 0x5589861ccbd0 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7fd300019020 tx=0x7fd30000b050 comp rx=0 tx=0).stop 2024-03-28T14:08:46.030 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:46.014+0000 7fd30eaf3000 1 -- 172.21.15.104:0/1552470592 shutdown_connections 2024-03-28T14:08:46.031 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:46.014+0000 7fd30eaf3000 1 --2- 172.21.15.104:0/1552470592 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x5589861b5730 0x5589861cd110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:46.031 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:46.014+0000 7fd30eaf3000 1 --2- 172.21.15.104:0/1552470592 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5589861b4e10 0x5589861ccbd0 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:46.031 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:46.014+0000 7fd30eaf3000 1 --2- 172.21.15.104:0/1552470592 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x558986190e00 0x55898618b800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:46.031 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:46.014+0000 7fd30eaf3000 1 -- 172.21.15.104:0/1552470592 >> 172.21.15.104:0/1552470592 conn(0x558986073b80 msgr2=0x5589861d5ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T14:08:46.031 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:46.014+0000 7fd30eaf3000 1 -- 172.21.15.104:0/1552470592 shutdown_connections 2024-03-28T14:08:46.031 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:46.014+0000 7fd30eaf3000 1 -- 172.21.15.104:0/1552470592 wait complete. 2024-03-28T14:08:46.032 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:46.014+0000 7fd30eaf3000 1 -- start start 2024-03-28T14:08:46.032 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:46.014+0000 7fd30eaf3000 1 Processor -- start 2024-03-28T14:08:46.032 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:46.014+0000 7fd30eaf3000 1 --2- >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x558986190e00 0x5589861c9be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:46.032 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:46.014+0000 7fd30eaf3000 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5589861b4e10 0x5589861ca120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:46.032 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:46.014+0000 7fd30eaf3000 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5589861b5730 0x5589861ca660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:46.032 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:46.014+0000 7fd30eaf3000 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x558985fe8060 con 0x5589861b5730 2024-03-28T14:08:46.033 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:46.014+0000 7fd30eaf3000 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x558985fdac60 con 0x5589861b4e10 2024-03-28T14:08:46.033 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:46.014+0000 7fd30eaf3000 1 -- --> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] -- mon_getmap magic: 0 -- 0x558985f95c90 con 0x558986190e00 2024-03-28T14:08:46.033 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:46.014+0000 7fd30eaf3000 10 client.0 _ll_register_callbacks cb 0x7fd300009640 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T14:08:46.033 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:46.014+0000 7fd309f11640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5589861b5730 0x5589861ca660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:46.034 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:46.014+0000 7fd309f11640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5589861b5730 0x5589861ca660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:50550/0 (socket says 172.21.15.104:50550) 2024-03-28T14:08:46.034 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:46.014+0000 7fd309f11640 1 -- 172.21.15.104:0/3435239419 learned_addr learned my addr 172.21.15.104:0/3435239419 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-03-28T14:08:46.034 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:46.014+0000 7fd309710640 1 --2- 172.21.15.104:0/3435239419 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x558986190e00 0x5589861c9be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:46.034 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:46.014+0000 7fd308f0f640 1 --2- 172.21.15.104:0/3435239419 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5589861b4e10 0x5589861ca120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:46.034 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:46.014+0000 7fd309f11640 1 -- 172.21.15.104:0/3435239419 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x558986190e00 msgr2=0x5589861c9be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T14:08:46.035 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:46.014+0000 7fd309f11640 1 --2- 172.21.15.104:0/3435239419 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x558986190e00 0x5589861c9be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:46.035 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:46.014+0000 7fd309f11640 1 -- 172.21.15.104:0/3435239419 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5589861b4e10 msgr2=0x5589861ca120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T14:08:46.035 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:46.014+0000 7fd309f11640 1 --2- 172.21.15.104:0/3435239419 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5589861b4e10 0x5589861ca120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:46.035 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:46.014+0000 7fd309f11640 1 -- 172.21.15.104:0/3435239419 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558985f96490 con 0x5589861b5730 2024-03-28T14:08:46.035 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:46.014+0000 7fd309710640 1 --2- 172.21.15.104:0/3435239419 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x558986190e00 0x5589861c9be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T14:08:46.036 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:46.014+0000 7fd308f0f640 1 --2- 172.21.15.104:0/3435239419 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5589861b4e10 0x5589861ca120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T14:08:46.036 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:46.014+0000 7fd309f11640 1 --2- 172.21.15.104:0/3435239419 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5589861b5730 0x5589861ca660 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7fd2fc0532d0 tx=0x7fd2fc056e70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b095bbd6aa69d1d3 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:08:46.036 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:46.014+0000 7fd2e3fff640 10 client.0 ms_handle_connect on v2:172.21.15.104:3300/0 2024-03-28T14:08:46.036 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:46.014+0000 7fd2e3fff640 1 -- 172.21.15.104:0/3435239419 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2fc06f020 con 0x5589861b5730 2024-03-28T14:08:46.036 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:46.014+0000 7fd30eaf3000 1 -- 172.21.15.104:0/3435239419 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd300000e50 con 0x5589861b5730 2024-03-28T14:08:46.036 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:46.014+0000 7fd2e3fff640 1 -- 172.21.15.104:0/3435239419 <== mon.0 v2:172.21.15.104:3300/0 2 ==== osd_map(30..30 src has 1..30) ==== 5709+0+0 (secure 0 0 0) 0x7fd2fc055170 con 0x5589861b5730 2024-03-28T14:08:46.037 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:46.014+0000 7fd2e3fff640 1 -- 172.21.15.104:0/3435239419 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_get_version_reply(handle=1 version=14) ==== 24+0+0 (secure 0 0 0) 0x7fd2fc05fa50 con 0x5589861b5730 2024-03-28T14:08:46.037 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:46.014+0000 7fd30eaf3000 10 client.5148 fetch_fsmap learned FSMap version 14 2024-03-28T14:08:46.037 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:46.014+0000 7fd30eaf3000 1 -- 172.21.15.104:0/3435239419 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,fsmap=14,monmap=2+}) -- 0x558985f94e80 con 0x5589861b5730 2024-03-28T14:08:46.037 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:46.014+0000 7fd2e3fff640 1 -- 172.21.15.104:0/3435239419 <== mon.0 v2:172.21.15.104:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd300000e50 con 0x5589861b5730 2024-03-28T14:08:46.037 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:46.014+0000 7fd2e3fff640 1 -- 172.21.15.104:0/3435239419 <== mon.0 v2:172.21.15.104:3300/0 5 ==== fsmap(e 14) ==== 2284+0+0 (secure 0 0 0) 0x7fd2fc05f5c0 con 0x5589861b5730 2024-03-28T14:08:46.038 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:46.014+0000 7fd30eaf3000 10 client.5148 fetch_fsmap finished waiting for FSMap version 14 2024-03-28T14:08:46.038 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:46.014+0000 7fd30eaf3000 10 client.5148 resolve_mds: resolved name 'a' to daemon mds.a 2024-03-28T14:08:46.038 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:46.014+0000 7fd30eaf3000 20 client.5148 populate_metadata read hostname 'smithi104' 2024-03-28T14:08:46.038 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:46.014+0000 7fd30eaf3000 1 --2- 172.21.15.104:0/3435239419 >> [v2:172.21.15.104:6832/1094990707,v1:172.21.15.104:6833/1094990707] conn(0x5589861caba0 0x5589862378e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:46.038 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:46.014+0000 7fd30eaf3000 4 client.5148 mds_command: new command op to 4959 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-28T14:08:46.039 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:46.014+0000 7fd30eaf3000 1 -- 172.21.15.104:0/3435239419 --> [v2:172.21.15.104:6832/1094990707,v1:172.21.15.104:6833/1094990707] -- command(tid 0: {"prefix": "status"}) -- 0x558985fe8e90 con 0x5589861caba0 2024-03-28T14:08:46.039 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:46.014+0000 7fd309710640 1 --2- 172.21.15.104:0/3435239419 >> [v2:172.21.15.104:6832/1094990707,v1:172.21.15.104:6833/1094990707] conn(0x5589861caba0 0x5589862378e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:46.039 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:46.014+0000 7fd2e3fff640 10 client.5148 ms_handle_connect on v2:172.21.15.104:6832/1094990707 2024-03-28T14:08:46.039 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:46.014+0000 7fd309710640 1 --2- 172.21.15.104:0/3435239419 >> [v2:172.21.15.104:6832/1094990707,v1:172.21.15.104:6833/1094990707] conn(0x5589861caba0 0x5589862378e0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=dc9d97e26a4a41c2 server_cookie=489468206c300bf1 in_seq=0 out_seq=0 2024-03-28T14:08:46.039 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:46.014+0000 7fd2e3fff640 1 -- 172.21.15.104:0/3435239419 <== mds.0 v2:172.21.15.104:6832/1094990707 1 ==== command_reply(tid 0: 0 ) ==== 8+0+331 (crc 0 0 0) 0x558985fe8e90 con 0x5589861caba0 2024-03-28T14:08:46.040 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:46.014+0000 7fd2e3fff640 10 client.5148 handle_command_reply: tid=0 2024-03-28T14:08:46.040 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:46.014+0000 7fd30eaf3000 1 client.5148 shutdown 2024-03-28T14:08:46.040 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:46.014+0000 7fd30eaf3000 1 -- 172.21.15.104:0/3435239419 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5589861b5730 msgr2=0x5589861ca660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:46.040 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:46.014+0000 7fd30eaf3000 1 --2- 172.21.15.104:0/3435239419 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5589861b5730 0x5589861ca660 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7fd2fc0532d0 tx=0x7fd2fc056e70 comp rx=0 tx=0).stop 2024-03-28T14:08:46.040 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:46.014+0000 7fd30eaf3000 1 -- 172.21.15.104:0/3435239419 shutdown_connections 2024-03-28T14:08:46.040 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:46.014+0000 7fd30eaf3000 1 --2- 172.21.15.104:0/3435239419 >> [v2:172.21.15.104:6832/1094990707,v1:172.21.15.104:6833/1094990707] conn(0x5589861caba0 0x5589862378e0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:46.041 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:46.014+0000 7fd30eaf3000 1 --2- 172.21.15.104:0/3435239419 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5589861b5730 0x5589861ca660 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:46.041 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:46.014+0000 7fd30eaf3000 1 --2- 172.21.15.104:0/3435239419 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5589861b4e10 0x5589861ca120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:46.041 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:46.014+0000 7fd2e3fff640 0 client.5148 ms_handle_reset on v2:172.21.15.104:6832/1094990707 2024-03-28T14:08:46.041 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:46.014+0000 7fd30eaf3000 1 --2- 172.21.15.104:0/3435239419 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x558986190e00 0x5589861c9be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:46.041 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:46.014+0000 7fd30eaf3000 1 -- 172.21.15.104:0/3435239419 >> 172.21.15.104:0/3435239419 conn(0x558986073b80 msgr2=0x5589861d6780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T14:08:46.042 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:46.014+0000 7fd30eaf3000 1 -- 172.21.15.104:0/3435239419 shutdown_connections 2024-03-28T14:08:46.042 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:46.014+0000 7fd30eaf3000 1 -- 172.21.15.104:0/3435239419 wait complete. 2024-03-28T14:08:46.042 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:46.014+0000 7fd30eaf3000 20 client.5148 trim_cache size 0 max 16384 2024-03-28T14:08:46.043 INFO:teuthology.orchestra.run.smithi104.stdout:{ 2024-03-28T14:08:46.043 INFO:teuthology.orchestra.run.smithi104.stdout: "cluster_fsid": "428a90f2-3351-435f-8fcf-f7e66c78a13e", 2024-03-28T14:08:46.043 INFO:teuthology.orchestra.run.smithi104.stdout: "whoami": 0, 2024-03-28T14:08:46.043 INFO:teuthology.orchestra.run.smithi104.stdout: "id": 4959, 2024-03-28T14:08:46.043 INFO:teuthology.orchestra.run.smithi104.stdout: "want_state": "up:active", 2024-03-28T14:08:46.044 INFO:teuthology.orchestra.run.smithi104.stdout: "state": "up:active", 2024-03-28T14:08:46.044 INFO:teuthology.orchestra.run.smithi104.stdout: "fs_name": "cephfs", 2024-03-28T14:08:46.044 INFO:teuthology.orchestra.run.smithi104.stdout: "rank_uptime": 4.5418238579999999, 2024-03-28T14:08:46.044 INFO:teuthology.orchestra.run.smithi104.stdout: "mdsmap_epoch": 14, 2024-03-28T14:08:46.044 INFO:teuthology.orchestra.run.smithi104.stdout: "osdmap_epoch": 30, 2024-03-28T14:08:46.045 INFO:teuthology.orchestra.run.smithi104.stdout: "osdmap_epoch_barrier": 30, 2024-03-28T14:08:46.045 INFO:teuthology.orchestra.run.smithi104.stdout: "uptime": 11.991992226000001 2024-03-28T14:08:46.045 INFO:teuthology.orchestra.run.smithi104.stdout:} 2024-03-28T14:08:46.048 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:46.018+0000 7fd30b974640 1 -- 172.21.15.104:0/435257380 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7fd2e8067fc0 msgr2=0x7fd2e806a480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:46.048 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:46.018+0000 7fd30b974640 1 --2- 172.21.15.104:0/435257380 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7fd2e8067fc0 0x7fd2e806a480 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fd2f80015a0 tx=0x7fd2f8007970 comp rx=0 tx=0).stop 2024-03-28T14:08:46.048 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:46.018+0000 7fd30b974640 1 -- 172.21.15.104:0/435257380 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd3040b51e0 msgr2=0x7fd3041dfd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:46.048 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:46.018+0000 7fd30b974640 1 --2- 172.21.15.104:0/435257380 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd3040b51e0 0x7fd3041dfd30 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7fd2fc00cf20 tx=0x7fd2fc00a7d0 comp rx=0 tx=0).stop 2024-03-28T14:08:46.048 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:46.018+0000 7fd30b974640 1 -- 172.21.15.104:0/435257380 shutdown_connections 2024-03-28T14:08:46.048 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:46.018+0000 7fd30b974640 1 --2- 172.21.15.104:0/435257380 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7fd2e8067fc0 0x7fd2e806a480 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:46.048 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:46.018+0000 7fd30b974640 1 --2- 172.21.15.104:0/435257380 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd3040b51e0 0x7fd3041dfd30 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:46.048 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:46.018+0000 7fd30b974640 1 --2- 172.21.15.104:0/435257380 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fd30409aa10 0x7fd30413c1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:46.048 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:46.018+0000 7fd30b974640 1 --2- 172.21.15.104:0/435257380 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fd30406dd10 0x7fd3040af880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:46.048 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:46.018+0000 7fd30b974640 1 -- 172.21.15.104:0/435257380 >> 172.21.15.104:0/435257380 conn(0x7fd3040ba390 msgr2=0x7fd3041457c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T14:08:46.048 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:46.018+0000 7fd30b974640 1 -- 172.21.15.104:0/435257380 shutdown_connections 2024-03-28T14:08:46.049 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:46.018+0000 7fd30b974640 1 -- 172.21.15.104:0/435257380 wait complete. 2024-03-28T14:08:46.049 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-03-28T14:08:46.155 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:46.150+0000 7f43ff109640 1 Processor -- start 2024-03-28T14:08:46.155 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:46.150+0000 7f43ff109640 1 -- start start 2024-03-28T14:08:46.156 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:46.154+0000 7f43ff109640 1 --2- >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f43f802c460 0x7f43f814afd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:46.156 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:46.154+0000 7f43ff109640 1 --2- >> v2:172.21.15.171:3300/0 conn(0x7f43f8148110 0x7f43f81485a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:46.156 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:46.154+0000 7f43ff109640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f43f802c840 con 0x7f43f80b5570 2024-03-28T14:08:46.156 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:46.154+0000 7f43ff109640 1 -- --> v2:172.21.15.171:3300/0 -- mon_getmap magic: 0 -- 0x7f43f80b4850 con 0x7f43f8148110 2024-03-28T14:08:46.156 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:46.154+0000 7f43ff109640 1 -- --> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] -- mon_getmap magic: 0 -- 0x7f43f80b4d70 con 0x7f43f802c460 2024-03-28T14:08:46.157 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:46.154+0000 7f43f7fff640 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f43f80b5570 0x7f43f80b3ff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:59868/0 (socket says 172.21.15.104:59868) 2024-03-28T14:08:46.157 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:46.154+0000 7f43f7fff640 1 -- 172.21.15.104:0/1831701472 learned_addr learned my addr 172.21.15.104:0/1831701472 (peer_addr_for_me v1:172.21.15.104:0/0) 2024-03-28T14:08:46.158 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:46.154+0000 7f43fcea5640 1 --2- 172.21.15.104:0/1831701472 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f43f802c460 0x7f43f814afd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:46.159 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:46.154+0000 7f43fd6a6640 1 --2- 172.21.15.104:0/1831701472 >> v2:172.21.15.171:3300/0 conn(0x7f43f8148110 0x7f43f81485a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:46.159 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:46.154+0000 7f43f77fe640 1 -- 172.21.15.104:0/1831701472 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1275098357 0 0) 0x7f43f802c840 con 0x7f43f80b5570 2024-03-28T14:08:46.159 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:46.154+0000 7f43f77fe640 1 -- 172.21.15.104:0/1831701472 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f43e4003530 con 0x7f43f80b5570 2024-03-28T14:08:46.159 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:46.154+0000 7f43f77fe640 1 -- 172.21.15.104:0/1831701472 <== mon.0 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3846272235 0 0) 0x7f43e4003530 con 0x7f43f80b5570 2024-03-28T14:08:46.159 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:46.154+0000 7f43f77fe640 1 -- 172.21.15.104:0/1831701472 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f43f802c460 msgr2=0x7f43f814afd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T14:08:46.159 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:46.154+0000 7f43f77fe640 1 --2- 172.21.15.104:0/1831701472 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f43f802c460 0x7f43f814afd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:46.159 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:46.154+0000 7f43f77fe640 1 -- 172.21.15.104:0/1831701472 >> v2:172.21.15.171:3300/0 conn(0x7f43f8148110 msgr2=0x7f43f81485a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T14:08:46.159 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:46.154+0000 7f43f77fe640 1 --2- 172.21.15.104:0/1831701472 >> v2:172.21.15.171:3300/0 conn(0x7f43f8148110 0x7f43f81485a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:46.159 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:46.154+0000 7f43f77fe640 1 -- 172.21.15.104:0/1831701472 --> v1:172.21.15.104:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f43f80b5050 con 0x7f43f80b5570 2024-03-28T14:08:46.159 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:46.154+0000 7f43f77fe640 1 -- 172.21.15.104:0/1831701472 <== mon.0 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3745580849 0 0) 0x7f43e8001680 con 0x7f43f80b5570 2024-03-28T14:08:46.160 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:46.154+0000 7f43f77fe640 1 -- 172.21.15.104:0/1831701472 >> v1:172.21.15.104:6789/0 conn(0x7f43f80b5570 legacy=0x7f43f80b3ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:46.160 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:46.154+0000 7f43f77fe640 1 --2- 172.21.15.104:0/1831701472 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f43e40039c0 0x7f43e4003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:46.160 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:46.154+0000 7f43f77fe640 1 --2- 172.21.15.104:0/1831701472 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f43f802c460 0x7f43e4004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:46.160 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:46.154+0000 7f43f77fe640 1 --2- 172.21.15.104:0/1831701472 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f43e4004d50 0x7f43e40051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:46.160 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:46.154+0000 7f43f77fe640 1 -- 172.21.15.104:0/1831701472 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f43f80b4850 con 0x7f43e4004d50 2024-03-28T14:08:46.160 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:46.154+0000 7f43f77fe640 1 -- 172.21.15.104:0/1831701472 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f43f80b4d70 con 0x7f43e40039c0 2024-03-28T14:08:46.160 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:46.154+0000 7f43f77fe640 1 -- 172.21.15.104:0/1831701472 --> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] -- mon_getmap magic: 0 -- 0x7f43e400b9d0 con 0x7f43f802c460 2024-03-28T14:08:46.160 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:46.154+0000 7f43fd6a6640 1 --2- 172.21.15.104:0/1831701472 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f43e4004d50 0x7f43e40051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:46.160 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:46.154+0000 7f43fcea5640 1 --2- 172.21.15.104:0/1831701472 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f43e40039c0 0x7f43e4003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:46.160 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:46.154+0000 7f43fd6a6640 1 -- 172.21.15.104:0/1831701472 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f43f802c460 msgr2=0x7f43e4004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T14:08:46.160 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:46.154+0000 7f43fd6a6640 1 --2- 172.21.15.104:0/1831701472 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f43f802c460 0x7f43e4004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:46.160 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:46.154+0000 7f43fd6a6640 1 -- 172.21.15.104:0/1831701472 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f43e40039c0 msgr2=0x7f43e4003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T14:08:46.161 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:46.154+0000 7f43fd6a6640 1 --2- 172.21.15.104:0/1831701472 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f43e40039c0 0x7f43e4003de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:46.161 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:46.154+0000 7f43fd6a6640 1 -- 172.21.15.104:0/1831701472 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f43e4003530 con 0x7f43e4004d50 2024-03-28T14:08:46.164 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:46.154+0000 7f43fd6a6640 1 --2- 172.21.15.104:0/1831701472 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f43e4004d50 0x7f43e40051d0 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f43f0009ae0 tx=0x7f43f001cab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a705e0564981935f server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:08:46.164 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:46.154+0000 7f43f77fe640 1 -- 172.21.15.104:0/1831701472 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43f00b2070 con 0x7f43e4004d50 2024-03-28T14:08:46.164 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:46.154+0000 7f43f77fe640 1 -- 172.21.15.104:0/1831701472 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f43f00a56a0 con 0x7f43e4004d50 2024-03-28T14:08:46.164 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:46.154+0000 7f43f77fe640 1 -- 172.21.15.104:0/1831701472 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43f00096b0 con 0x7f43e4004d50 2024-03-28T14:08:46.164 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:46.158+0000 7f43ff109640 1 -- 172.21.15.104:0/1831701472 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f43e4004d50 msgr2=0x7f43e40051d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:46.164 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:46.158+0000 7f43ff109640 1 --2- 172.21.15.104:0/1831701472 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f43e4004d50 0x7f43e40051d0 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f43f0009ae0 tx=0x7f43f001cab0 comp rx=0 tx=0).stop 2024-03-28T14:08:46.165 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:46.158+0000 7f43fd6a6640 1 -- 172.21.15.104:0/1831701472 reap_dead start 2024-03-28T14:08:46.165 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:46.158+0000 7f43ff109640 1 -- 172.21.15.104:0/1831701472 shutdown_connections 2024-03-28T14:08:46.165 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:46.158+0000 7f43ff109640 1 -- 172.21.15.104:0/1831701472 >> 172.21.15.104:0/1831701472 conn(0x7f43f80b95f0 msgr2=0x7f43f80b99d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T14:08:46.165 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:46.158+0000 7f43ff109640 1 -- 172.21.15.104:0/1831701472 shutdown_connections 2024-03-28T14:08:46.165 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:46.158+0000 7f43ff109640 1 -- 172.21.15.104:0/1831701472 wait complete. 2024-03-28T14:08:46.165 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:46.158+0000 7f43ff109640 1 Processor -- start 2024-03-28T14:08:46.165 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:46.158+0000 7f43ff109640 1 -- start start 2024-03-28T14:08:46.165 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:46.158+0000 7f43ff109640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f43e4004d50 0x7f43f8071370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:46.165 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:46.158+0000 7f43ff109640 1 --2- >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f43f81e0470 0x7f43f81d6e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:46.165 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:46.158+0000 7f43ff109640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f43f81e08f0 0x7f43f81dcf00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:46.165 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:46.158+0000 7f43ff109640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f43f809ea60 con 0x7f43f81e08f0 2024-03-28T14:08:46.165 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:46.158+0000 7f43ff109640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f43f8148ae0 con 0x7f43e4004d50 2024-03-28T14:08:46.165 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:46.158+0000 7f43ff109640 1 -- --> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] -- mon_getmap magic: 0 -- 0x7f43f809ebe0 con 0x7f43f81e0470 2024-03-28T14:08:46.165 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:46.158+0000 7f43fd6a6640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f43f81e08f0 0x7f43f81dcf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:46.165 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:46.158+0000 7f43fd6a6640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f43f81e08f0 0x7f43f81dcf00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:50582/0 (socket says 172.21.15.104:50582) 2024-03-28T14:08:46.166 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:46.158+0000 7f43fd6a6640 1 -- 172.21.15.104:0/199302480 learned_addr learned my addr 172.21.15.104:0/199302480 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-03-28T14:08:46.166 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:46.158+0000 7f43f7fff640 1 --2- 172.21.15.104:0/199302480 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f43f81e0470 0x7f43f81d6e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:46.166 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:46.158+0000 7f43fcea5640 1 --2- 172.21.15.104:0/199302480 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f43e4004d50 0x7f43f8071370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:46.166 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:46.158+0000 7f43fd6a6640 1 -- 172.21.15.104:0/199302480 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f43f81e0470 msgr2=0x7f43f81d6e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:46.166 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:46.158+0000 7f43fd6a6640 1 --2- 172.21.15.104:0/199302480 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f43f81e0470 0x7f43f81d6e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:46.166 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:46.158+0000 7f43fd6a6640 1 -- 172.21.15.104:0/199302480 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f43e4004d50 msgr2=0x7f43f8071370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:46.166 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:46.158+0000 7f43fd6a6640 1 --2- 172.21.15.104:0/199302480 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f43e4004d50 0x7f43f8071370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:46.166 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:46.158+0000 7f43fd6a6640 1 -- 172.21.15.104:0/199302480 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f43f8072220 con 0x7f43f81e08f0 2024-03-28T14:08:46.166 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:46.158+0000 7f43fcea5640 1 --2- 172.21.15.104:0/199302480 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f43e4004d50 0x7f43f8071370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T14:08:46.166 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:46.158+0000 7f43fd6a6640 1 --2- 172.21.15.104:0/199302480 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f43f81e08f0 0x7f43f81dcf00 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f43f00b0040 tx=0x7f43e4005840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:08:46.166 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:46.158+0000 7f43f57fa640 1 -- 172.21.15.104:0/199302480 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43f00bb070 con 0x7f43f81e08f0 2024-03-28T14:08:46.166 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:46.158+0000 7f43f57fa640 1 -- 172.21.15.104:0/199302480 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f43f0000e50 con 0x7f43f81e08f0 2024-03-28T14:08:46.166 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:46.158+0000 7f43ff109640 1 -- 172.21.15.104:0/199302480 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f43f81422e0 con 0x7f43f81e08f0 2024-03-28T14:08:46.167 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:46.158+0000 7f43f57fa640 1 -- 172.21.15.104:0/199302480 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43f00b2040 con 0x7f43f81e08f0 2024-03-28T14:08:46.167 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:46.158+0000 7f43ff109640 1 -- 172.21.15.104:0/199302480 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f43f8071d00 con 0x7f43f81e08f0 2024-03-28T14:08:46.167 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:46.158+0000 7f43ff109640 1 -- 172.21.15.104:0/199302480 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f43f80687c0 con 0x7f43f81e08f0 2024-03-28T14:08:46.167 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:46.162+0000 7f43f57fa640 1 -- 172.21.15.104:0/199302480 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) ==== 87803+0+0 (secure 0 0 0) 0x7f43f00be070 con 0x7f43f81e08f0 2024-03-28T14:08:46.167 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:46.162+0000 7f43f57fa640 1 --2- 172.21.15.104:0/199302480 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f43d8067ff0 0x7f43d806a4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:46.168 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:46.162+0000 7f43f57fa640 1 -- 172.21.15.104:0/199302480 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 5709+0+0 (secure 0 0 0) 0x7f43f011f730 con 0x7f43f81e08f0 2024-03-28T14:08:46.168 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:46.162+0000 7f43fcea5640 1 --2- 172.21.15.104:0/199302480 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f43d8067ff0 0x7f43d806a4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:46.168 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:46.162+0000 7f43fcea5640 1 --2- 172.21.15.104:0/199302480 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f43d8067ff0 0x7f43d806a4b0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f43f80986c0 tx=0x7f43e000d040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:08:46.168 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:46.166+0000 7f43f57fa640 1 -- 172.21.15.104:0/199302480 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179117 (secure 0 0 0) 0x7f43f00167c0 con 0x7f43f81e08f0 2024-03-28T14:08:46.351 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:46.346+0000 7f43ff109640 1 -- 172.21.15.104:0/199302480 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7f43f809c770 con 0x7f43f81e08f0 2024-03-28T14:08:46.352 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:46.350+0000 7f43f57fa640 1 -- 172.21.15.104:0/199302480 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v14) ==== 53+0+73 (secure 0 0 0) 0x7f43f00ef920 con 0x7f43f81e08f0 2024-03-28T14:08:46.352 INFO:teuthology.orchestra.run.smithi104.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-03-28T14:08:46.354 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:46.350+0000 7f43ff109640 1 -- 172.21.15.104:0/199302480 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f43d8067ff0 msgr2=0x7f43d806a4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:46.354 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:46.350+0000 7f43ff109640 1 --2- 172.21.15.104:0/199302480 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f43d8067ff0 0x7f43d806a4b0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f43f80986c0 tx=0x7f43e000d040 comp rx=0 tx=0).stop 2024-03-28T14:08:46.354 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:46.350+0000 7f43ff109640 1 -- 172.21.15.104:0/199302480 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f43f81e08f0 msgr2=0x7f43f81dcf00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:46.354 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:46.350+0000 7f43ff109640 1 --2- 172.21.15.104:0/199302480 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f43f81e08f0 0x7f43f81dcf00 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f43f00b0040 tx=0x7f43e4005840 comp rx=0 tx=0).stop 2024-03-28T14:08:46.354 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:46.350+0000 7f43ff109640 1 -- 172.21.15.104:0/199302480 shutdown_connections 2024-03-28T14:08:46.355 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:46.350+0000 7f43ff109640 1 --2- 172.21.15.104:0/199302480 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f43d8067ff0 0x7f43d806a4b0 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:46.355 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:46.350+0000 7f43ff109640 1 --2- 172.21.15.104:0/199302480 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f43f81e08f0 0x7f43f81dcf00 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:46.355 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:46.350+0000 7f43ff109640 1 --2- 172.21.15.104:0/199302480 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f43f81e0470 0x7f43f81d6e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:46.355 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:46.350+0000 7f43ff109640 1 --2- 172.21.15.104:0/199302480 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f43e4004d50 0x7f43f8071370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:46.355 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:46.350+0000 7f43ff109640 1 -- 172.21.15.104:0/199302480 >> 172.21.15.104:0/199302480 conn(0x7f43f80b95f0 msgr2=0x7f43f80b5f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T14:08:46.355 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:46.350+0000 7f43ff109640 1 -- 172.21.15.104:0/199302480 shutdown_connections 2024-03-28T14:08:46.355 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:46.350+0000 7f43ff109640 1 -- 172.21.15.104:0/199302480 wait complete. 2024-03-28T14:08:46.369 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-03-28T14:08:46.369 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-28T14:08:46.369 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi104.front.sepia.ceph.com 2024-03-28T14:08:46.369 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-03-28T14:08:46.369 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-03-28T14:08:46.369 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-03-28T14:08:46.370 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-03-28T14:08:46.370 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-28T14:08:46.370 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-28T14:08:46.370 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T14:08:46.370 DEBUG:teuthology.orchestra.run.smithi104:> ip addr 2024-03-28T14:08:46.376 INFO:teuthology.orchestra.run.smithi104.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-28T14:08:46.376 INFO:teuthology.orchestra.run.smithi104.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-28T14:08:46.376 INFO:teuthology.orchestra.run.smithi104.stdout: inet 127.0.0.1/8 scope host lo 2024-03-28T14:08:46.376 INFO:teuthology.orchestra.run.smithi104.stdout: valid_lft forever preferred_lft forever 2024-03-28T14:08:46.376 INFO:teuthology.orchestra.run.smithi104.stdout: inet6 ::1/128 scope host 2024-03-28T14:08:46.376 INFO:teuthology.orchestra.run.smithi104.stdout: valid_lft forever preferred_lft forever 2024-03-28T14:08:46.376 INFO:teuthology.orchestra.run.smithi104.stdout:2: eno1: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-03-28T14:08:46.376 INFO:teuthology.orchestra.run.smithi104.stdout: link/ether 0c:c4:7a:d8:a1:1c brd ff:ff:ff:ff:ff:ff 2024-03-28T14:08:46.376 INFO:teuthology.orchestra.run.smithi104.stdout: altname enp7s0f0 2024-03-28T14:08:46.377 INFO:teuthology.orchestra.run.smithi104.stdout:3: eno2: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-03-28T14:08:46.377 INFO:teuthology.orchestra.run.smithi104.stdout: link/ether 0c:c4:7a:d8:a1:1d brd ff:ff:ff:ff:ff:ff 2024-03-28T14:08:46.377 INFO:teuthology.orchestra.run.smithi104.stdout: altname enp7s0f1 2024-03-28T14:08:46.377 INFO:teuthology.orchestra.run.smithi104.stdout:4: enp3s0f0: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-03-28T14:08:46.377 INFO:teuthology.orchestra.run.smithi104.stdout: link/ether 0c:c4:7a:8f:ce:88 brd ff:ff:ff:ff:ff:ff 2024-03-28T14:08:46.377 INFO:teuthology.orchestra.run.smithi104.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-28T14:08:46.377 INFO:teuthology.orchestra.run.smithi104.stdout: link/ether 0c:c4:7a:8f:ce:89 brd ff:ff:ff:ff:ff:ff 2024-03-28T14:08:46.377 INFO:teuthology.orchestra.run.smithi104.stdout: inet 172.21.15.104/20 metric 100 brd 172.21.15.255 scope global dynamic enp3s0f1 2024-03-28T14:08:46.377 INFO:teuthology.orchestra.run.smithi104.stdout: valid_lft 42465sec preferred_lft 42465sec 2024-03-28T14:08:46.377 INFO:teuthology.orchestra.run.smithi104.stdout: inet6 fe80::ec4:7aff:fe8f:ce89/64 scope link 2024-03-28T14:08:46.377 INFO:teuthology.orchestra.run.smithi104.stdout: valid_lft forever preferred_lft forever 2024-03-28T14:08:46.377 INFO:teuthology.orchestra.run.smithi104.stdout:6: docker0: mtu 1500 qdisc noqueue state DOWN group default 2024-03-28T14:08:46.377 INFO:teuthology.orchestra.run.smithi104.stdout: link/ether 02:42:4f:83:9c:9e brd ff:ff:ff:ff:ff:ff 2024-03-28T14:08:46.377 INFO:teuthology.orchestra.run.smithi104.stdout: inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0 2024-03-28T14:08:46.378 INFO:teuthology.orchestra.run.smithi104.stdout: valid_lft forever preferred_lft forever 2024-03-28T14:08:46.378 INFO:teuthology.orchestra.run.smithi104.stdout:7: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-03-28T14:08:46.378 INFO:teuthology.orchestra.run.smithi104.stdout: link/ether 5e:1f:84:6d:79:ae brd ff:ff:ff:ff:ff:ff 2024-03-28T14:08:46.378 INFO:teuthology.orchestra.run.smithi104.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-03-28T14:08:46.378 INFO:teuthology.orchestra.run.smithi104.stdout: valid_lft forever preferred_lft forever 2024-03-28T14:08:46.378 INFO:teuthology.orchestra.run.smithi104.stdout: inet6 fe80::5c1f:84ff:fe6d:79ae/64 scope link 2024-03-28T14:08:46.378 INFO:teuthology.orchestra.run.smithi104.stdout: valid_lft forever preferred_lft forever 2024-03-28T14:08:46.378 INFO:teuthology.orchestra.run.smithi104.stdout:8: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-28T14:08:46.378 INFO:teuthology.orchestra.run.smithi104.stdout: link/ether 0e:5c:f6:1f:82:2f brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-28T14:08:46.378 INFO:teuthology.orchestra.run.smithi104.stdout: inet6 fe80::c5c:f6ff:fe1f:822f/64 scope link 2024-03-28T14:08:46.378 INFO:teuthology.orchestra.run.smithi104.stdout: valid_lft forever preferred_lft forever 2024-03-28T14:08:46.378 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T14:08:46.379 DEBUG:teuthology.orchestra.run.smithi104:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-28T14:08:46.430 INFO:teuthology.orchestra.run.smithi104.stdout:1 2024-03-28T14:08:46.431 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T14:08:46.432 DEBUG:teuthology.orchestra.run.smithi104:> route 2024-03-28T14:08:46.484 INFO:teuthology.orchestra.run.smithi104.stdout:Kernel IP routing table 2024-03-28T14:08:46.484 INFO:teuthology.orchestra.run.smithi104.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-28T14:08:46.484 INFO:teuthology.orchestra.run.smithi104.stdout:default _gateway 0.0.0.0 UG 0 0 0 enp3s0f1 2024-03-28T14:08:46.484 INFO:teuthology.orchestra.run.smithi104.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-03-28T14:08:46.485 INFO:teuthology.orchestra.run.smithi104.stdout:158.69.67.47 _gateway 255.255.255.255 UGH 100 0 0 enp3s0f1 2024-03-28T14:08:46.485 INFO:teuthology.orchestra.run.smithi104.stdout:172.17.0.0 0.0.0.0 255.255.0.0 U 0 0 0 docker0 2024-03-28T14:08:46.485 INFO:teuthology.orchestra.run.smithi104.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-03-28T14:08:46.485 INFO:teuthology.orchestra.run.smithi104.stdout:ns1.front.sepia 0.0.0.0 255.255.255.255 UH 100 0 0 enp3s0f1 2024-03-28T14:08:46.485 INFO:teuthology.orchestra.run.smithi104.stdout:172.21.0.2 0.0.0.0 255.255.255.255 UH 100 0 0 enp3s0f1 2024-03-28T14:08:46.485 INFO:teuthology.orchestra.run.smithi104.stdout:_gateway 0.0.0.0 255.255.255.255 UH 100 0 0 enp3s0f1 2024-03-28T14:08:46.485 INFO:teuthology.orchestra.run.smithi104.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-28T14:08:46.485 DEBUG:teuthology.orchestra.run.smithi104:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-28T14:08:46.485 DEBUG:teuthology.orchestra.run.smithi104:> set -e 2024-03-28T14:08:46.485 DEBUG:teuthology.orchestra.run.smithi104:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-03-28T14:08:46.485 DEBUG:teuthology.orchestra.run.smithi104:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-03-28T14:08:46.486 DEBUG:teuthology.orchestra.run.smithi104:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-03-28T14:08:46.486 DEBUG:teuthology.orchestra.run.smithi104:> ') 2024-03-28T14:08:46.573 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:46 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-28T14:08:46.599 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:46 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-28T14:08:46.605 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T14:08:46.605 DEBUG:teuthology.orchestra.run.smithi104:> ip netns list 2024-03-28T14:08:46.610 INFO:teuthology.orchestra.run.smithi104.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-03-28T14:08:46.610 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-28T14:08:46.610 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T14:08:46.610 DEBUG:teuthology.orchestra.run.smithi104:> sudo ip link set brx.0 up 2024-03-28T14:08:46.666 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2024-03-28T14:08:46.666 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-28T14:08:46.666 DEBUG:teuthology.orchestra.run.smithi104:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-03-28T14:08:46.712 INFO:teuthology.orchestra.run.smithi104.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-03-28T14:08:46.712 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-28T14:08:46.712 DEBUG:teuthology.orchestra.run.smithi104:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-03-28T14:08:46.759 DEBUG:teuthology.orchestra.run.smithi104:> sudo modprobe fuse 2024-03-28T14:08:46.817 DEBUG:teuthology.orchestra.run.smithi104:> cat /proc/self/mounts | awk '{print $2}' 2024-03-28T14:08:46.865 INFO:teuthology.orchestra.run.smithi104.stdout:/sys 2024-03-28T14:08:46.865 INFO:teuthology.orchestra.run.smithi104.stdout:/proc 2024-03-28T14:08:46.865 INFO:teuthology.orchestra.run.smithi104.stdout:/dev 2024-03-28T14:08:46.865 INFO:teuthology.orchestra.run.smithi104.stdout:/dev/pts 2024-03-28T14:08:46.865 INFO:teuthology.orchestra.run.smithi104.stdout:/run 2024-03-28T14:08:46.865 INFO:teuthology.orchestra.run.smithi104.stdout:/ 2024-03-28T14:08:46.865 INFO:teuthology.orchestra.run.smithi104.stdout:/sys/kernel/security 2024-03-28T14:08:46.865 INFO:teuthology.orchestra.run.smithi104.stdout:/dev/shm 2024-03-28T14:08:46.865 INFO:teuthology.orchestra.run.smithi104.stdout:/run/lock 2024-03-28T14:08:46.865 INFO:teuthology.orchestra.run.smithi104.stdout:/sys/fs/cgroup 2024-03-28T14:08:46.866 INFO:teuthology.orchestra.run.smithi104.stdout:/sys/fs/pstore 2024-03-28T14:08:46.866 INFO:teuthology.orchestra.run.smithi104.stdout:/sys/fs/bpf 2024-03-28T14:08:46.866 INFO:teuthology.orchestra.run.smithi104.stdout:/proc/sys/fs/binfmt_misc 2024-03-28T14:08:46.866 INFO:teuthology.orchestra.run.smithi104.stdout:/dev/hugepages 2024-03-28T14:08:46.866 INFO:teuthology.orchestra.run.smithi104.stdout:/dev/mqueue 2024-03-28T14:08:46.866 INFO:teuthology.orchestra.run.smithi104.stdout:/sys/kernel/debug 2024-03-28T14:08:46.866 INFO:teuthology.orchestra.run.smithi104.stdout:/sys/kernel/tracing 2024-03-28T14:08:46.866 INFO:teuthology.orchestra.run.smithi104.stdout:/proc/fs/nfsd 2024-03-28T14:08:46.866 INFO:teuthology.orchestra.run.smithi104.stdout:/sys/fs/fuse/connections 2024-03-28T14:08:46.866 INFO:teuthology.orchestra.run.smithi104.stdout:/sys/kernel/config 2024-03-28T14:08:46.866 INFO:teuthology.orchestra.run.smithi104.stdout:/run/credentials/systemd-sysusers.service 2024-03-28T14:08:46.866 INFO:teuthology.orchestra.run.smithi104.stdout:/run/rpc_pipefs 2024-03-28T14:08:46.866 INFO:teuthology.orchestra.run.smithi104.stdout:/run/user/1000 2024-03-28T14:08:46.866 INFO:teuthology.orchestra.run.smithi104.stdout:/var/lib/ceph 2024-03-28T14:08:46.867 INFO:teuthology.orchestra.run.smithi104.stdout:/var/lib/ceph/osd/ceph-0 2024-03-28T14:08:46.867 INFO:teuthology.orchestra.run.smithi104.stdout:/var/lib/ceph/osd/ceph-1 2024-03-28T14:08:46.867 INFO:teuthology.orchestra.run.smithi104.stdout:/var/lib/ceph/osd/ceph-2 2024-03-28T14:08:46.867 INFO:teuthology.orchestra.run.smithi104.stdout:/var/lib/ceph/osd/ceph-3 2024-03-28T14:08:46.867 INFO:teuthology.orchestra.run.smithi104.stdout:/run/netns 2024-03-28T14:08:46.867 INFO:teuthology.orchestra.run.smithi104.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-28T14:08:46.867 INFO:teuthology.orchestra.run.smithi104.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-28T14:08:46.868 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T14:08:46.868 DEBUG:teuthology.orchestra.run.smithi104:> ls /sys/fs/fuse/connections 2024-03-28T14:08:46.920 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-03-28T14:08:46.920 DEBUG:teuthology.orchestra.run.smithi104:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2024-03-28T14:08:46.966 DEBUG:teuthology.orchestra.run.smithi104:> sudo modprobe fuse 2024-03-28T14:08:46.977 DEBUG:teuthology.orchestra.run.smithi104:> cat /proc/self/mounts | awk '{print $2}' 2024-03-28T14:08:47.029 INFO:teuthology.orchestra.run.smithi104.stdout:/sys 2024-03-28T14:08:47.029 INFO:teuthology.orchestra.run.smithi104.stdout:/proc 2024-03-28T14:08:47.029 INFO:teuthology.orchestra.run.smithi104.stdout:/dev 2024-03-28T14:08:47.029 INFO:teuthology.orchestra.run.smithi104.stdout:/dev/pts 2024-03-28T14:08:47.029 INFO:teuthology.orchestra.run.smithi104.stdout:/run 2024-03-28T14:08:47.029 INFO:teuthology.orchestra.run.smithi104.stdout:/ 2024-03-28T14:08:47.029 INFO:teuthology.orchestra.run.smithi104.stdout:/sys/kernel/security 2024-03-28T14:08:47.029 INFO:teuthology.orchestra.run.smithi104.stdout:/dev/shm 2024-03-28T14:08:47.029 INFO:teuthology.orchestra.run.smithi104.stdout:/run/lock 2024-03-28T14:08:47.029 INFO:teuthology.orchestra.run.smithi104.stdout:/sys/fs/cgroup 2024-03-28T14:08:47.029 INFO:teuthology.orchestra.run.smithi104.stdout:/sys/fs/pstore 2024-03-28T14:08:47.029 INFO:teuthology.orchestra.run.smithi104.stdout:/sys/fs/bpf 2024-03-28T14:08:47.030 INFO:teuthology.orchestra.run.smithi104.stdout:/proc/sys/fs/binfmt_misc 2024-03-28T14:08:47.030 INFO:teuthology.orchestra.run.smithi104.stdout:/dev/hugepages 2024-03-28T14:08:47.030 INFO:teuthology.orchestra.run.smithi104.stdout:/dev/mqueue 2024-03-28T14:08:47.030 INFO:teuthology.orchestra.run.smithi104.stdout:/sys/kernel/debug 2024-03-28T14:08:47.030 INFO:teuthology.orchestra.run.smithi104.stdout:/sys/kernel/tracing 2024-03-28T14:08:47.030 INFO:teuthology.orchestra.run.smithi104.stdout:/proc/fs/nfsd 2024-03-28T14:08:47.030 INFO:teuthology.orchestra.run.smithi104.stdout:/sys/fs/fuse/connections 2024-03-28T14:08:47.030 INFO:teuthology.orchestra.run.smithi104.stdout:/sys/kernel/config 2024-03-28T14:08:47.030 INFO:teuthology.orchestra.run.smithi104.stdout:/run/credentials/systemd-sysusers.service 2024-03-28T14:08:47.030 INFO:teuthology.orchestra.run.smithi104.stdout:/run/rpc_pipefs 2024-03-28T14:08:47.030 INFO:teuthology.orchestra.run.smithi104.stdout:/run/user/1000 2024-03-28T14:08:47.030 INFO:teuthology.orchestra.run.smithi104.stdout:/var/lib/ceph 2024-03-28T14:08:47.030 INFO:teuthology.orchestra.run.smithi104.stdout:/var/lib/ceph/osd/ceph-0 2024-03-28T14:08:47.030 INFO:teuthology.orchestra.run.smithi104.stdout:/var/lib/ceph/osd/ceph-1 2024-03-28T14:08:47.031 INFO:teuthology.orchestra.run.smithi104.stdout:/var/lib/ceph/osd/ceph-2 2024-03-28T14:08:47.031 INFO:teuthology.orchestra.run.smithi104.stdout:/var/lib/ceph/osd/ceph-3 2024-03-28T14:08:47.031 INFO:teuthology.orchestra.run.smithi104.stdout:/run/netns 2024-03-28T14:08:47.031 INFO:teuthology.orchestra.run.smithi104.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-28T14:08:47.031 INFO:teuthology.orchestra.run.smithi104.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-28T14:08:47.032 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T14:08:47.032 DEBUG:teuthology.orchestra.run.smithi104:> ls /sys/fs/fuse/connections 2024-03-28T14:08:47.042 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi104.stderr:2024-03-28T14:08:47.038+0000 7f73900bd580 -1 init, newargv = 0x55f19cfc3230 newargc=15 2024-03-28T14:08:47.042 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi104.stderr:ceph-fuse[36837]: starting ceph client 2024-03-28T14:08:47.047 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi104.stderr:ceph-fuse[36837]: starting fuse 2024-03-28T14:08:48.148 DEBUG:teuthology.orchestra.run.smithi104:> sudo modprobe fuse 2024-03-28T14:08:48.162 DEBUG:teuthology.orchestra.run.smithi104:> cat /proc/self/mounts | awk '{print $2}' 2024-03-28T14:08:48.209 INFO:teuthology.orchestra.run.smithi104.stdout:/sys 2024-03-28T14:08:48.209 INFO:teuthology.orchestra.run.smithi104.stdout:/proc 2024-03-28T14:08:48.209 INFO:teuthology.orchestra.run.smithi104.stdout:/dev 2024-03-28T14:08:48.209 INFO:teuthology.orchestra.run.smithi104.stdout:/dev/pts 2024-03-28T14:08:48.210 INFO:teuthology.orchestra.run.smithi104.stdout:/run 2024-03-28T14:08:48.210 INFO:teuthology.orchestra.run.smithi104.stdout:/ 2024-03-28T14:08:48.210 INFO:teuthology.orchestra.run.smithi104.stdout:/sys/kernel/security 2024-03-28T14:08:48.210 INFO:teuthology.orchestra.run.smithi104.stdout:/dev/shm 2024-03-28T14:08:48.210 INFO:teuthology.orchestra.run.smithi104.stdout:/run/lock 2024-03-28T14:08:48.210 INFO:teuthology.orchestra.run.smithi104.stdout:/sys/fs/cgroup 2024-03-28T14:08:48.210 INFO:teuthology.orchestra.run.smithi104.stdout:/sys/fs/pstore 2024-03-28T14:08:48.210 INFO:teuthology.orchestra.run.smithi104.stdout:/sys/fs/bpf 2024-03-28T14:08:48.210 INFO:teuthology.orchestra.run.smithi104.stdout:/proc/sys/fs/binfmt_misc 2024-03-28T14:08:48.210 INFO:teuthology.orchestra.run.smithi104.stdout:/dev/hugepages 2024-03-28T14:08:48.210 INFO:teuthology.orchestra.run.smithi104.stdout:/dev/mqueue 2024-03-28T14:08:48.210 INFO:teuthology.orchestra.run.smithi104.stdout:/sys/kernel/debug 2024-03-28T14:08:48.211 INFO:teuthology.orchestra.run.smithi104.stdout:/sys/kernel/tracing 2024-03-28T14:08:48.211 INFO:teuthology.orchestra.run.smithi104.stdout:/proc/fs/nfsd 2024-03-28T14:08:48.211 INFO:teuthology.orchestra.run.smithi104.stdout:/sys/fs/fuse/connections 2024-03-28T14:08:48.211 INFO:teuthology.orchestra.run.smithi104.stdout:/sys/kernel/config 2024-03-28T14:08:48.211 INFO:teuthology.orchestra.run.smithi104.stdout:/run/credentials/systemd-sysusers.service 2024-03-28T14:08:48.211 INFO:teuthology.orchestra.run.smithi104.stdout:/run/rpc_pipefs 2024-03-28T14:08:48.211 INFO:teuthology.orchestra.run.smithi104.stdout:/run/user/1000 2024-03-28T14:08:48.211 INFO:teuthology.orchestra.run.smithi104.stdout:/var/lib/ceph 2024-03-28T14:08:48.211 INFO:teuthology.orchestra.run.smithi104.stdout:/var/lib/ceph/osd/ceph-0 2024-03-28T14:08:48.211 INFO:teuthology.orchestra.run.smithi104.stdout:/var/lib/ceph/osd/ceph-1 2024-03-28T14:08:48.211 INFO:teuthology.orchestra.run.smithi104.stdout:/var/lib/ceph/osd/ceph-2 2024-03-28T14:08:48.211 INFO:teuthology.orchestra.run.smithi104.stdout:/var/lib/ceph/osd/ceph-3 2024-03-28T14:08:48.211 INFO:teuthology.orchestra.run.smithi104.stdout:/run/netns 2024-03-28T14:08:48.211 INFO:teuthology.orchestra.run.smithi104.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-28T14:08:48.212 INFO:teuthology.orchestra.run.smithi104.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-28T14:08:48.212 INFO:teuthology.orchestra.run.smithi104.stdout:/home/ubuntu/cephtest/mnt.0 2024-03-28T14:08:48.213 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T14:08:48.214 DEBUG:teuthology.orchestra.run.smithi104:> ls /sys/fs/fuse/connections 2024-03-28T14:08:48.260 INFO:teuthology.orchestra.run.smithi104.stdout:45 2024-03-28T14:08:48.260 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [45] 2024-03-28T14:08:48.260 DEBUG:teuthology.orchestra.run.smithi104:> sudo stdin-killer -- python3 -c ' 2024-03-28T14:08:48.261 DEBUG:teuthology.orchestra.run.smithi104:> import glob 2024-03-28T14:08:48.261 DEBUG:teuthology.orchestra.run.smithi104:> import re 2024-03-28T14:08:48.261 DEBUG:teuthology.orchestra.run.smithi104:> import os 2024-03-28T14:08:48.261 DEBUG:teuthology.orchestra.run.smithi104:> import subprocess 2024-03-28T14:08:48.261 DEBUG:teuthology.orchestra.run.smithi104:> 2024-03-28T14:08:48.261 DEBUG:teuthology.orchestra.run.smithi104:> def _find_admin_socket(client_name): 2024-03-28T14:08:48.261 DEBUG:teuthology.orchestra.run.smithi104:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2024-03-28T14:08:48.261 DEBUG:teuthology.orchestra.run.smithi104:> files = glob.glob(asok_path) 2024-03-28T14:08:48.261 DEBUG:teuthology.orchestra.run.smithi104:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2024-03-28T14:08:48.261 DEBUG:teuthology.orchestra.run.smithi104:> 2024-03-28T14:08:48.261 DEBUG:teuthology.orchestra.run.smithi104:> # Given a non-glob path, it better be there 2024-03-28T14:08:48.261 DEBUG:teuthology.orchestra.run.smithi104:> if "*" not in asok_path: 2024-03-28T14:08:48.261 DEBUG:teuthology.orchestra.run.smithi104:> assert(len(files) == 1) 2024-03-28T14:08:48.261 DEBUG:teuthology.orchestra.run.smithi104:> return files[0] 2024-03-28T14:08:48.261 DEBUG:teuthology.orchestra.run.smithi104:> 2024-03-28T14:08:48.262 DEBUG:teuthology.orchestra.run.smithi104:> for f in files: 2024-03-28T14:08:48.262 DEBUG:teuthology.orchestra.run.smithi104:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-03-28T14:08:48.262 DEBUG:teuthology.orchestra.run.smithi104:> if os.path.exists("/proc/{0}".format(pid)): 2024-03-28T14:08:48.262 DEBUG:teuthology.orchestra.run.smithi104:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-03-28T14:08:48.262 DEBUG:teuthology.orchestra.run.smithi104:> contents = proc_f.read() 2024-03-28T14:08:48.262 DEBUG:teuthology.orchestra.run.smithi104:> if mountpoint in contents: 2024-03-28T14:08:48.262 DEBUG:teuthology.orchestra.run.smithi104:> return f 2024-03-28T14:08:48.262 DEBUG:teuthology.orchestra.run.smithi104:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-03-28T14:08:48.262 DEBUG:teuthology.orchestra.run.smithi104:> 2024-03-28T14:08:48.262 DEBUG:teuthology.orchestra.run.smithi104:> print(_find_admin_socket("client.0")) 2024-03-28T14:08:48.262 DEBUG:teuthology.orchestra.run.smithi104:> ' 2024-03-28T14:08:48.381 INFO:teuthology.orchestra.run.smithi104.stdout:/var/run/ceph/ceph-client.0.36837.asok 2024-03-28T14:08:48.385 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:48 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-28T14:08:48.392 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.36837.asok 2024-03-28T14:08:48.393 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T14:08:48.393 DEBUG:teuthology.orchestra.run.smithi104:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.36837.asok status 2024-03-28T14:08:48.570 INFO:teuthology.orchestra.run.smithi104.stdout:{ 2024-03-28T14:08:48.570 INFO:teuthology.orchestra.run.smithi104.stdout: "metadata": { 2024-03-28T14:08:48.570 INFO:teuthology.orchestra.run.smithi104.stdout: "ceph_sha1": "e0e48ba6ee8c8b4c8f38d3b879d6eafbbb1efc15", 2024-03-28T14:08:48.570 INFO:teuthology.orchestra.run.smithi104.stdout: "ceph_version": "ceph version 19.0.0-2499-ge0e48ba6 (e0e48ba6ee8c8b4c8f38d3b879d6eafbbb1efc15) squid (dev)", 2024-03-28T14:08:48.571 INFO:teuthology.orchestra.run.smithi104.stdout: "entity_id": "0", 2024-03-28T14:08:48.571 INFO:teuthology.orchestra.run.smithi104.stdout: "hostname": "smithi104", 2024-03-28T14:08:48.571 INFO:teuthology.orchestra.run.smithi104.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2024-03-28T14:08:48.571 INFO:teuthology.orchestra.run.smithi104.stdout: "pid": "36837", 2024-03-28T14:08:48.571 INFO:teuthology.orchestra.run.smithi104.stdout: "root": "/" 2024-03-28T14:08:48.571 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-03-28T14:08:48.571 INFO:teuthology.orchestra.run.smithi104.stdout: "dentry_count": 0, 2024-03-28T14:08:48.571 INFO:teuthology.orchestra.run.smithi104.stdout: "dentry_pinned_count": 0, 2024-03-28T14:08:48.571 INFO:teuthology.orchestra.run.smithi104.stdout: "id": 5160, 2024-03-28T14:08:48.571 INFO:teuthology.orchestra.run.smithi104.stdout: "inst": { 2024-03-28T14:08:48.571 INFO:teuthology.orchestra.run.smithi104.stdout: "name": { 2024-03-28T14:08:48.571 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "client", 2024-03-28T14:08:48.571 INFO:teuthology.orchestra.run.smithi104.stdout: "num": 5160 2024-03-28T14:08:48.572 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-03-28T14:08:48.572 INFO:teuthology.orchestra.run.smithi104.stdout: "addr": { 2024-03-28T14:08:48.572 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "v1", 2024-03-28T14:08:48.572 INFO:teuthology.orchestra.run.smithi104.stdout: "addr": "192.168.0.1:0", 2024-03-28T14:08:48.572 INFO:teuthology.orchestra.run.smithi104.stdout: "nonce": 2289957627 2024-03-28T14:08:48.572 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-03-28T14:08:48.572 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-03-28T14:08:48.572 INFO:teuthology.orchestra.run.smithi104.stdout: "addr": { 2024-03-28T14:08:48.572 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "v1", 2024-03-28T14:08:48.572 INFO:teuthology.orchestra.run.smithi104.stdout: "addr": "192.168.0.1:0", 2024-03-28T14:08:48.572 INFO:teuthology.orchestra.run.smithi104.stdout: "nonce": 2289957627 2024-03-28T14:08:48.572 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-03-28T14:08:48.572 INFO:teuthology.orchestra.run.smithi104.stdout: "inst_str": "client.5160 192.168.0.1:0/2289957627", 2024-03-28T14:08:48.573 INFO:teuthology.orchestra.run.smithi104.stdout: "addr_str": "192.168.0.1:0/2289957627", 2024-03-28T14:08:48.573 INFO:teuthology.orchestra.run.smithi104.stdout: "inode_count": 1, 2024-03-28T14:08:48.573 INFO:teuthology.orchestra.run.smithi104.stdout: "mds_epoch": 14, 2024-03-28T14:08:48.573 INFO:teuthology.orchestra.run.smithi104.stdout: "osd_epoch": 30, 2024-03-28T14:08:48.573 INFO:teuthology.orchestra.run.smithi104.stdout: "osd_epoch_barrier": 0, 2024-03-28T14:08:48.573 INFO:teuthology.orchestra.run.smithi104.stdout: "blocklisted": false, 2024-03-28T14:08:48.573 INFO:teuthology.orchestra.run.smithi104.stdout: "fs_name": "cephfs" 2024-03-28T14:08:48.573 INFO:teuthology.orchestra.run.smithi104.stdout:} 2024-03-28T14:08:48.584 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T14:08:48.584 DEBUG:teuthology.orchestra.run.smithi104:> stat --file-system '--printf=%T 2024-03-28T14:08:48.584 DEBUG:teuthology.orchestra.run.smithi104:> ' -- /home/ubuntu/cephtest/mnt.0 2024-03-28T14:08:48.591 INFO:teuthology.orchestra.run.smithi104.stdout:fuseblk 2024-03-28T14:08:48.591 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2024-03-28T14:08:48.592 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T14:08:48.592 DEBUG:teuthology.orchestra.run.smithi104:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-03-28T14:08:48.643 DEBUG:teuthology.orchestra.run.smithi104:> mktemp 2024-03-28T14:08:48.696 INFO:teuthology.orchestra.run.smithi104.stdout:/tmp/tmp.WVxg0bVuZ2 2024-03-28T14:08:48.697 DEBUG:teuthology.orchestra.run.smithi104:> set -ex 2024-03-28T14:08:48.697 DEBUG:teuthology.orchestra.run.smithi104:> dd of=/tmp/tmp.WVxg0bVuZ2 2024-03-28T14:08:48.744 DEBUG:teuthology.orchestra.run.smithi104:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'mkdir -p dir1/dir2/dir3') 2024-03-28T14:08:48.829 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:48 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-03-28T14:08:48.844 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:48 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-28T14:08:48.903 DEBUG:teuthology.orchestra.run.smithi104:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'chmod 755 /tmp/test-cephfspmcncaaq') 2024-03-28T14:08:48.945 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:48 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-28T14:08:48.951 INFO:tasks.cephfs.test_cephfs_shell:Running script "/tmp/test-cephfspmcncaaq" 2024-03-28T14:08:48.951 DEBUG:teuthology.orchestra.run.smithi104:> cephfs-shell -b /tmp/test-cephfspmcncaaq 2024-03-28T14:08:49.302 INFO:teuthology.orchestra.run.smithi104.stderr:Unrecognized command 2024-03-28T14:08:49.332 DEBUG:teuthology.orchestra.run:got remote process result: 127 2024-03-28T14:08:49.333 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-03-28T14:08:49.333 DEBUG:teuthology.orchestra.run.smithi104:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-28T14:08:49.389 DEBUG:teuthology.orchestra.run.smithi171:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-28T14:08:49.451 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-03-28T14:08:49.451 INFO:tasks.cephfs.mount:Terminating background process 2024-03-28T14:08:49.451 DEBUG:teuthology.orchestra.run.smithi104:> set -ex 2024-03-28T14:08:49.451 DEBUG:teuthology.orchestra.run.smithi104:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-28T14:08:49.457 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-28T14:08:49.619 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:49.614+0000 7f88e5015640 1 Processor -- start 2024-03-28T14:08:49.619 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:49.614+0000 7f88e5015640 1 -- start start 2024-03-28T14:08:49.621 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:49.618+0000 7f88e5015640 1 --2- >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f88e0072bf0 0x7f88e00b1e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:49.621 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:49.618+0000 7f88e5015640 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f88e0071be0 0x7f88e00b12f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:49.621 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:49.618+0000 7f88e5015640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f88e0072020 con 0x7f88e00b1830 2024-03-28T14:08:49.621 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:49.618+0000 7f88e5015640 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 -- 0x7f88e0073250 con 0x7f88e0071be0 2024-03-28T14:08:49.621 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:49.618+0000 7f88e5015640 1 -- --> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] -- mon_getmap magic: 0 -- 0x7f88e0072fd0 con 0x7f88e0072bf0 2024-03-28T14:08:49.621 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:49.618+0000 7f88d7fff640 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f88e0071be0 0x7f88e00b12f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:49.621 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:49.618+0000 7f88d7fff640 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f88e0071be0 0x7f88e00b12f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:37726/0 (socket says 172.21.15.104:37726) 2024-03-28T14:08:49.621 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:49.618+0000 7f88d7fff640 1 -- 172.21.15.104:0/4128319501 learned_addr learned my addr 172.21.15.104:0/4128319501 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-03-28T14:08:49.621 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:49.618+0000 7f88ded9d640 1 --2- 172.21.15.104:0/4128319501 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f88e0072bf0 0x7f88e00b1e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:49.621 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:49.618+0000 7f88d7fff640 1 -- 172.21.15.104:0/4128319501 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f88e0072bf0 msgr2=0x7f88e00b1e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T14:08:49.621 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:49.618+0000 7f88d7fff640 1 --2- 172.21.15.104:0/4128319501 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f88e0072bf0 0x7f88e00b1e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:49.621 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:49.618+0000 7f88d7fff640 1 -- 172.21.15.104:0/4128319501 >> v1:172.21.15.171:6789/0 conn(0x7f88e00b1830 legacy=0x7f88e0145ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T14:08:49.622 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:49.618+0000 7f88d7fff640 1 -- 172.21.15.104:0/4128319501 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f88e00734a0 con 0x7f88e0071be0 2024-03-28T14:08:49.622 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:49.618+0000 7f88d7fff640 1 --2- 172.21.15.104:0/4128319501 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f88e0071be0 0x7f88e00b12f0 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f88c8009870 tx=0x7f88c801c400 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7163a71845519323 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:08:49.622 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:49.618+0000 7f88de59c640 1 -- 172.21.15.104:0/4128319501 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88c80b0070 con 0x7f88e0071be0 2024-03-28T14:08:49.623 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:49.618+0000 7f88de59c640 1 -- 172.21.15.104:0/4128319501 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f88c80ab600 con 0x7f88e0071be0 2024-03-28T14:08:49.623 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:49.618+0000 7f88de59c640 1 -- 172.21.15.104:0/4128319501 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88c80ab9d0 con 0x7f88e0071be0 2024-03-28T14:08:49.623 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:49.618+0000 7f88e5015640 1 -- 172.21.15.104:0/4128319501 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f88e0071be0 msgr2=0x7f88e00b12f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:49.624 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:49.618+0000 7f88e5015640 1 --2- 172.21.15.104:0/4128319501 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f88e0071be0 0x7f88e00b12f0 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f88c8009870 tx=0x7f88c801c400 comp rx=0 tx=0).stop 2024-03-28T14:08:49.624 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:49.618+0000 7f88e5015640 1 -- 172.21.15.104:0/4128319501 shutdown_connections 2024-03-28T14:08:49.624 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:49.618+0000 7f88e5015640 1 --2- 172.21.15.104:0/4128319501 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f88e0071be0 0x7f88e00b12f0 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:49.624 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:49.618+0000 7f88e5015640 1 --2- 172.21.15.104:0/4128319501 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f88e0072bf0 0x7f88e00b1e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:49.624 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:49.618+0000 7f88e5015640 1 -- 172.21.15.104:0/4128319501 >> 172.21.15.104:0/4128319501 conn(0x7f88e00b7b10 msgr2=0x7f88e00b7f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T14:08:49.624 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:49.618+0000 7f88e5015640 1 -- 172.21.15.104:0/4128319501 shutdown_connections 2024-03-28T14:08:49.624 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:49.618+0000 7f88e5015640 1 -- 172.21.15.104:0/4128319501 wait complete. 2024-03-28T14:08:49.626 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:49.622+0000 7f88e5015640 1 Processor -- start 2024-03-28T14:08:49.626 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:49.622+0000 7f88e5015640 1 -- start start 2024-03-28T14:08:49.626 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:49.622+0000 7f88e5015640 1 --2- >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f88e0071be0 0x7f88e014ef30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:49.626 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:49.622+0000 7f88e5015640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f88e0072bf0 0x7f88e009c280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:49.626 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:49.622+0000 7f88e5015640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f88e00b1830 0x7f88e009c7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:49.626 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:49.622+0000 7f88e5015640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f88e0152830 con 0x7f88e00b1830 2024-03-28T14:08:49.626 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:49.622+0000 7f88e5015640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f88e01526b0 con 0x7f88e0072bf0 2024-03-28T14:08:49.627 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:49.622+0000 7f88e5015640 1 -- --> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] -- mon_getmap magic: 0 -- 0x7f88e01529b0 con 0x7f88e0071be0 2024-03-28T14:08:49.627 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:49.622+0000 7f88df59e640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f88e00b1830 0x7f88e009c7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:49.627 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:49.622+0000 7f88df59e640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f88e00b1830 0x7f88e009c7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:37736/0 (socket says 172.21.15.104:37736) 2024-03-28T14:08:49.627 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:49.622+0000 7f88df59e640 1 -- 172.21.15.104:0/260040043 learned_addr learned my addr 172.21.15.104:0/260040043 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-03-28T14:08:49.627 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:49.622+0000 7f88df59e640 1 -- 172.21.15.104:0/260040043 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f88e0071be0 msgr2=0x7f88e014ef30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:49.627 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:49.622+0000 7f88df59e640 1 --2- 172.21.15.104:0/260040043 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f88e0071be0 0x7f88e014ef30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:49.627 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:49.622+0000 7f88df59e640 1 -- 172.21.15.104:0/260040043 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f88e0072bf0 msgr2=0x7f88e009c280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:49.627 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:49.622+0000 7f88df59e640 1 --2- 172.21.15.104:0/260040043 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f88e0072bf0 0x7f88e009c280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:49.627 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:49.622+0000 7f88df59e640 1 -- 172.21.15.104:0/260040043 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f88e014fc10 con 0x7f88e00b1830 2024-03-28T14:08:49.627 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:49.622+0000 7f88df59e640 1 --2- 172.21.15.104:0/260040043 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f88e00b1830 0x7f88e009c7c0 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f88d800be60 tx=0x7f88d800bf50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:08:49.627 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:49.622+0000 7f88d77fe640 1 -- 172.21.15.104:0/260040043 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88d8019070 con 0x7f88e00b1830 2024-03-28T14:08:49.627 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:49.622+0000 7f88d77fe640 1 -- 172.21.15.104:0/260040043 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f88d8014440 con 0x7f88e00b1830 2024-03-28T14:08:49.627 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:49.622+0000 7f88e5015640 1 -- 172.21.15.104:0/260040043 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f88e006e0f0 con 0x7f88e00b1830 2024-03-28T14:08:49.628 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:49.622+0000 7f88d77fe640 1 -- 172.21.15.104:0/260040043 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88d8014870 con 0x7f88e00b1830 2024-03-28T14:08:49.628 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:49.622+0000 7f88e5015640 1 -- 172.21.15.104:0/260040043 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f88e006ce60 con 0x7f88e00b1830 2024-03-28T14:08:49.628 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:49.622+0000 7f88e5015640 1 -- 172.21.15.104:0/260040043 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f88e00687c0 con 0x7f88e00b1830 2024-03-28T14:08:49.629 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:49.622+0000 7f88d77fe640 1 -- 172.21.15.104:0/260040043 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) ==== 87803+0+0 (secure 0 0 0) 0x7f88d801c070 con 0x7f88e00b1830 2024-03-28T14:08:49.629 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:49.622+0000 7f88d77fe640 1 --2- 172.21.15.104:0/260040043 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f88c0067fc0 0x7f88c006a480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:49.629 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:49.626+0000 7f88d77fe640 1 -- 172.21.15.104:0/260040043 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 5709+0+0 (secure 0 0 0) 0x7f88d80864b0 con 0x7f88e00b1830 2024-03-28T14:08:49.629 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:49.626+0000 7f88ded9d640 1 --2- 172.21.15.104:0/260040043 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f88c0067fc0 0x7f88c006a480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:49.629 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:49.626+0000 7f88ded9d640 1 --2- 172.21.15.104:0/260040043 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f88c0067fc0 0x7f88c006a480 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f88d00098f0 tx=0x7f88d0008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:08:49.637 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:49.634+0000 7f88d77fe640 1 -- 172.21.15.104:0/260040043 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179117 (secure 0 0 0) 0x7f88d8056c50 con 0x7f88e00b1830 2024-03-28T14:08:49.799 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:49.794+0000 7f88e5015640 1 -- 172.21.15.104:0/260040043 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f88e00b96b0 con 0x7f88e00b1830 2024-03-28T14:08:49.801 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:49.798+0000 7f88d77fe640 1 -- 172.21.15.104:0/260040043 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v30) ==== 74+0+16693 (secure 0 0 0) 0x7f88d8056550 con 0x7f88e00b1830 2024-03-28T14:08:49.802 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-03-28T14:08:49.802 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":30,"fsid":"428a90f2-3351-435f-8fcf-f7e66c78a13e","created":"2024-03-28T14:07:34.862458+0000","modified":"2024-03-28T14:08:42.477796+0000","last_up_change":"2024-03-28T14:07:46.028122+0000","last_in_change":"2024-03-28T14:07:37.157486+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-28T14:07:47.622988+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_metadata","create_time":"2024-03-28T14:08:38.687068+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data","create_time":"2024-03-28T14:08:39.805358+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"30","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"1cb70540-4e42-402f-a446-aaec4ca83007","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6800","nonce":3745690918},{"type":"v1","addr":"172.21.15.104:6801","nonce":3745690918}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":3745690918},{"type":"v1","addr":"172.21.15.104:6803","nonce":3745690918}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":3745690918},{"type":"v1","addr":"172.21.15.104:6807","nonce":3745690918}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":3745690918},{"type":"v1","addr":"172.21.15.104:6805","nonce":3745690918}]},"public_addr":"172.21.15.104:6801/3745690918","cluster_addr":"172.21.15.104:6803/3745690918","heartbeat_back_addr":"172.21.15.104:6807/3745690918","heartbeat_front_addr":"172.21.15.104:6805/3745690918","state":["exists","up"]},{"osd":1,"uuid":"3ab7afdd-b980-418c-81f2-004e16de3dca","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":19438674},{"type":"v1","addr":"172.21.15.104:6810","nonce":19438674}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":19438674},{"type":"v1","addr":"172.21.15.104:6814","nonce":19438674}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":19438674},{"type":"v1","addr":"172.21.15.104:6823","nonce":19438674}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":19438674},{"type":"v1","addr":"172.21.15.104:6819","nonce":19438674}]},"public_addr":"172.21.15.104:6810/19438674","cluster_addr":"172.21.15.104:6814/19438674","heartbeat_back_addr":"172.21.15.104:6823/19438674","heartbeat_front_addr":"172.21.15.104:6819/19438674","state":["exists","up"]},{"osd":2,"uuid":"3e2651c8-ecae-4a9e-a053-27240a79eac2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":2352012626},{"type":"v1","addr":"172.21.15.104:6825","nonce":2352012626}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6826","nonce":2352012626},{"type":"v1","addr":"172.21.15.104:6827","nonce":2352012626}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":2352012626},{"type":"v1","addr":"172.21.15.104:6831","nonce":2352012626}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6828","nonce":2352012626},{"type":"v1","addr":"172.21.15.104:6829","nonce":2352012626}]},"public_addr":"172.21.15.104:6825/2352012626","cluster_addr":"172.21.15.104:6827/2352012626","heartbeat_back_addr":"172.21.15.104:6831/2352012626","heartbeat_front_addr":"172.21.15.104:6829/2352012626","state":["exists","up"]},{"osd":3,"uuid":"925c7e0a-38ef-47cd-b288-8bec8456fb17","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6809","nonce":3568347381},{"type":"v1","addr":"172.21.15.104:6811","nonce":3568347381}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6813","nonce":3568347381},{"type":"v1","addr":"172.21.15.104:6815","nonce":3568347381}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":3568347381},{"type":"v1","addr":"172.21.15.104:6822","nonce":3568347381}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6817","nonce":3568347381},{"type":"v1","addr":"172.21.15.104:6818","nonce":3568347381}]},"public_addr":"172.21.15.104:6811/3568347381","cluster_addr":"172.21.15.104:6815/3568347381","heartbeat_back_addr":"172.21.15.104:6822/3568347381","heartbeat_front_addr":"172.21.15.104:6818/3568347381","state":["exists","up"]},{"osd":4,"uuid":"4a8fac86-e595-4a0e-96e3-57edb3cc3f57","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6800","nonce":2107001077},{"type":"v1","addr":"172.21.15.171:6801","nonce":2107001077}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6802","nonce":2107001077},{"type":"v1","addr":"172.21.15.171:6803","nonce":2107001077}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6806","nonce":2107001077},{"type":"v1","addr":"172.21.15.171:6807","nonce":2107001077}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6804","nonce":2107001077},{"type":"v1","addr":"172.21.15.171:6805","nonce":2107001077}]},"public_addr":"172.21.15.171:6801/2107001077","cluster_addr":"172.21.15.171:6803/2107001077","heartbeat_back_addr":"172.21.15.171:6807/2107001077","heartbeat_front_addr":"172.21.15.171:6805/2107001077","state":["exists","up"]},{"osd":5,"uuid":"e48d50b0-beae-40ae-b15a-9a84dbfe88ab","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6808","nonce":2934826925},{"type":"v1","addr":"172.21.15.171:6810","nonce":2934826925}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6812","nonce":2934826925},{"type":"v1","addr":"172.21.15.171:6814","nonce":2934826925}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6819","nonce":2934826925},{"type":"v1","addr":"172.21.15.171:6821","nonce":2934826925}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6816","nonce":2934826925},{"type":"v1","addr":"172.21.15.171:6817","nonce":2934826925}]},"public_addr":"172.21.15.171:6810/2934826925","cluster_addr":"172.21.15.171:6814/2934826925","heartbeat_back_addr":"172.21.15.171:6821/2934826925","heartbeat_front_addr":"172.21.15.171:6817/2934826925","state":["exists","up"]},{"osd":6,"uuid":"5e7c3292-9936-4f35-b267-aed9d4e49323","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6824","nonce":436742919},{"type":"v1","addr":"172.21.15.171:6825","nonce":436742919}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6826","nonce":436742919},{"type":"v1","addr":"172.21.15.171:6827","nonce":436742919}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6830","nonce":436742919},{"type":"v1","addr":"172.21.15.171:6831","nonce":436742919}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6828","nonce":436742919},{"type":"v1","addr":"172.21.15.171:6829","nonce":436742919}]},"public_addr":"172.21.15.171:6825/436742919","cluster_addr":"172.21.15.171:6827/436742919","heartbeat_back_addr":"172.21.15.171:6831/436742919","heartbeat_front_addr":"172.21.15.171:6829/436742919","state":["exists","up"]},{"osd":7,"uuid":"339b8414-d7f0-4fb0-a446-efaf635cb7ff","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6809","nonce":2055631175},{"type":"v1","addr":"172.21.15.171:6811","nonce":2055631175}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6813","nonce":2055631175},{"type":"v1","addr":"172.21.15.171:6815","nonce":2055631175}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6822","nonce":2055631175},{"type":"v1","addr":"172.21.15.171:6823","nonce":2055631175}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6818","nonce":2055631175},{"type":"v1","addr":"172.21.15.171:6820","nonce":2055631175}]},"public_addr":"172.21.15.171:6811/2055631175","cluster_addr":"172.21.15.171:6815/2055631175","heartbeat_back_addr":"172.21.15.171:6823/2055631175","heartbeat_front_addr":"172.21.15.171:6820/2055631175","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T14:07:42.668881+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T14:07:42.708397+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T14:07:42.634191+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T14:07:42.733662+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T14:07:42.651541+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T14:07:42.783181+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T14:07:42.766310+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T14:07:42.670797+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-28T14:08:49.804 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:49.798+0000 7f88e5015640 1 -- 172.21.15.104:0/260040043 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f88c0067fc0 msgr2=0x7f88c006a480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:49.804 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:49.798+0000 7f88e5015640 1 --2- 172.21.15.104:0/260040043 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f88c0067fc0 0x7f88c006a480 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f88d00098f0 tx=0x7f88d0008040 comp rx=0 tx=0).stop 2024-03-28T14:08:49.804 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:49.798+0000 7f88e5015640 1 -- 172.21.15.104:0/260040043 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f88e00b1830 msgr2=0x7f88e009c7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:49.804 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:49.798+0000 7f88e5015640 1 --2- 172.21.15.104:0/260040043 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f88e00b1830 0x7f88e009c7c0 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f88d800be60 tx=0x7f88d800bf50 comp rx=0 tx=0).stop 2024-03-28T14:08:49.804 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:49.798+0000 7f88e5015640 1 -- 172.21.15.104:0/260040043 shutdown_connections 2024-03-28T14:08:49.805 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:49.798+0000 7f88e5015640 1 --2- 172.21.15.104:0/260040043 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f88c0067fc0 0x7f88c006a480 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:49.805 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:49.798+0000 7f88e5015640 1 --2- 172.21.15.104:0/260040043 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f88e00b1830 0x7f88e009c7c0 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:49.805 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:49.802+0000 7f88e5015640 1 --2- 172.21.15.104:0/260040043 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f88e0072bf0 0x7f88e009c280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:49.805 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:49.802+0000 7f88e5015640 1 --2- 172.21.15.104:0/260040043 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f88e0071be0 0x7f88e014ef30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:49.805 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:49.802+0000 7f88e5015640 1 -- 172.21.15.104:0/260040043 >> 172.21.15.104:0/260040043 conn(0x7f88e00b7b10 msgr2=0x7f88e00ba900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T14:08:49.805 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:49.802+0000 7f88e5015640 1 -- 172.21.15.104:0/260040043 shutdown_connections 2024-03-28T14:08:49.805 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:49.802+0000 7f88e5015640 1 -- 172.21.15.104:0/260040043 wait complete. 2024-03-28T14:08:49.821 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-28T14:07:47.622988+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-28T14:08:38.687068+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '29', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 5, 'pool_name': 'cephfs_data', 'create_time': '2024-03-28T14:08:39.805358+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '30', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-28T14:08:49.821 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-28T14:08:49.949 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:49.946+0000 7fc478335640 1 Processor -- start 2024-03-28T14:08:49.949 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:49.946+0000 7fc478335640 1 -- start start 2024-03-28T14:08:49.949 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:49.946+0000 7fc478335640 1 --2- >> v2:172.21.15.171:3300/0 conn(0x7fc4700ba900 0x7fc4700b6060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:49.950 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:49.946+0000 7fc478335640 1 --2- >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fc4700afe80 0x7fc47006b5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:49.950 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:49.946+0000 7fc478335640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc4700bace0 con 0x7fc47006bb10 2024-03-28T14:08:49.950 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:49.946+0000 7fc478335640 1 -- --> v2:172.21.15.171:3300/0 -- mon_getmap magic: 0 -- 0x7fc4700af710 con 0x7fc4700ba900 2024-03-28T14:08:49.950 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:49.946+0000 7fc478335640 1 -- --> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] -- mon_getmap magic: 0 -- 0x7fc4700af9f0 con 0x7fc4700afe80 2024-03-28T14:08:49.950 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:49.946+0000 7fc4758d0640 1 --2- >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fc4700afe80 0x7fc47006b5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:49.950 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:49.946+0000 7fc4760d1640 1 --2- >> v2:172.21.15.171:3300/0 conn(0x7fc4700ba900 0x7fc4700b6060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:49.950 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:49.946+0000 7fc4758d0640 1 --2- >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fc4700afe80 0x7fc47006b5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.171:3301/0 says I am v2:172.21.15.104:35996/0 (socket says 172.21.15.104:35996) 2024-03-28T14:08:49.950 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:49.946+0000 7fc4758d0640 1 -- 172.21.15.104:0/2480333990 learned_addr learned my addr 172.21.15.104:0/2480333990 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-03-28T14:08:49.951 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:49.946+0000 7fc4758d0640 1 -- 172.21.15.104:0/2480333990 >> v2:172.21.15.171:3300/0 conn(0x7fc4700ba900 msgr2=0x7fc4700b6060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T14:08:49.951 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:49.946+0000 7fc4758d0640 1 --2- 172.21.15.104:0/2480333990 >> v2:172.21.15.171:3300/0 conn(0x7fc4700ba900 0x7fc4700b6060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:49.951 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:49.946+0000 7fc4758d0640 1 -- 172.21.15.104:0/2480333990 >> v1:172.21.15.171:6789/0 conn(0x7fc47006bb10 legacy=0x7fc47014b1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:49.951 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:49.946+0000 7fc4758d0640 1 -- 172.21.15.104:0/2480333990 --> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc4700b52a0 con 0x7fc4700afe80 2024-03-28T14:08:49.952 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:49.950+0000 7fc4758d0640 1 --2- 172.21.15.104:0/2480333990 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fc4700afe80 0x7fc47006b5d0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fc460009870 tx=0x7fc46001c410 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c0515de23a931f16 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:08:49.953 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:49.950+0000 7fc4750cf640 1 -- 172.21.15.104:0/2480333990 <== mon.2 v2:172.21.15.171:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4600ab610 con 0x7fc4700afe80 2024-03-28T14:08:49.953 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:49.950+0000 7fc4750cf640 1 -- 172.21.15.104:0/2480333990 <== mon.2 v2:172.21.15.171:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc4600b0070 con 0x7fc4700afe80 2024-03-28T14:08:49.953 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:49.950+0000 7fc4750cf640 1 -- 172.21.15.104:0/2480333990 <== mon.2 v2:172.21.15.171:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4600ab9d0 con 0x7fc4700afe80 2024-03-28T14:08:49.954 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:49.950+0000 7fc478335640 1 -- 172.21.15.104:0/2480333990 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fc4700afe80 msgr2=0x7fc47006b5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:49.954 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:49.950+0000 7fc478335640 1 --2- 172.21.15.104:0/2480333990 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fc4700afe80 0x7fc47006b5d0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fc460009870 tx=0x7fc46001c410 comp rx=0 tx=0).stop 2024-03-28T14:08:49.954 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:49.950+0000 7fc478335640 1 -- 172.21.15.104:0/2480333990 shutdown_connections 2024-03-28T14:08:49.954 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:49.950+0000 7fc478335640 1 --2- 172.21.15.104:0/2480333990 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fc4700afe80 0x7fc47006b5d0 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:49.954 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:49.950+0000 7fc478335640 1 --2- 172.21.15.104:0/2480333990 >> v2:172.21.15.171:3300/0 conn(0x7fc4700ba900 0x7fc4700b6060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:49.954 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:49.950+0000 7fc478335640 1 -- 172.21.15.104:0/2480333990 >> 172.21.15.104:0/2480333990 conn(0x7fc47006cd40 msgr2=0x7fc4700b9f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T14:08:49.954 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:49.950+0000 7fc478335640 1 -- 172.21.15.104:0/2480333990 shutdown_connections 2024-03-28T14:08:49.954 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:49.950+0000 7fc478335640 1 -- 172.21.15.104:0/2480333990 wait complete. 2024-03-28T14:08:49.956 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:49.950+0000 7fc478335640 1 Processor -- start 2024-03-28T14:08:49.957 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:49.950+0000 7fc478335640 1 -- start start 2024-03-28T14:08:49.957 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:49.950+0000 7fc478335640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fc47006bb10 0x7fc47014a660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:49.957 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:49.950+0000 7fc478335640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc4700afe80 0x7fc4701dfa60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:49.957 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:49.950+0000 7fc478335640 1 --2- >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fc4700ba900 0x7fc47013d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:49.957 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:49.950+0000 7fc478335640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7fc47006c330 con 0x7fc4700afe80 2024-03-28T14:08:49.957 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:49.950+0000 7fc478335640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7fc4700af390 con 0x7fc47006bb10 2024-03-28T14:08:49.957 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:49.950+0000 7fc478335640 1 -- --> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] -- mon_getmap magic: 0 -- 0x7fc47006e8b0 con 0x7fc4700ba900 2024-03-28T14:08:49.957 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:49.950+0000 7fc4758d0640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc4700afe80 0x7fc4701dfa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:49.957 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:49.950+0000 7fc4758d0640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc4700afe80 0x7fc4701dfa60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:37752/0 (socket says 172.21.15.104:37752) 2024-03-28T14:08:49.957 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:49.950+0000 7fc4758d0640 1 -- 172.21.15.104:0/855614091 learned_addr learned my addr 172.21.15.104:0/855614091 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-03-28T14:08:49.957 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:49.950+0000 7fc4768d2640 1 --2- 172.21.15.104:0/855614091 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fc4700ba900 0x7fc47013d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:49.957 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:49.950+0000 7fc4760d1640 1 --2- 172.21.15.104:0/855614091 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fc47006bb10 0x7fc47014a660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:49.957 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:49.950+0000 7fc4758d0640 1 -- 172.21.15.104:0/855614091 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fc4700ba900 msgr2=0x7fc47013d0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:49.958 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:49.950+0000 7fc4758d0640 1 --2- 172.21.15.104:0/855614091 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fc4700ba900 0x7fc47013d0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:49.958 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:49.950+0000 7fc4758d0640 1 -- 172.21.15.104:0/855614091 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fc47006bb10 msgr2=0x7fc47014a660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:49.958 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:49.950+0000 7fc4758d0640 1 --2- 172.21.15.104:0/855614091 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fc47006bb10 0x7fc47014a660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:49.958 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:49.950+0000 7fc4758d0640 1 -- 172.21.15.104:0/855614091 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc4700b2d70 con 0x7fc4700afe80 2024-03-28T14:08:49.958 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:49.950+0000 7fc4760d1640 1 --2- 172.21.15.104:0/855614091 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fc47006bb10 0x7fc47014a660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T14:08:49.958 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:49.950+0000 7fc4758d0640 1 --2- 172.21.15.104:0/855614091 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc4700afe80 0x7fc4701dfa60 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7fc46001aac0 tx=0x7fc46001ac50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:08:49.958 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:49.950+0000 7fc466ffd640 1 -- 172.21.15.104:0/855614091 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4600be5f0 con 0x7fc4700afe80 2024-03-28T14:08:49.958 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:49.950+0000 7fc466ffd640 1 -- 172.21.15.104:0/855614091 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc4600be7b0 con 0x7fc4700afe80 2024-03-28T14:08:49.958 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:49.950+0000 7fc478335640 1 -- 172.21.15.104:0/855614091 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc470145cb0 con 0x7fc4700afe80 2024-03-28T14:08:49.958 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:49.950+0000 7fc466ffd640 1 -- 172.21.15.104:0/855614091 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4600beba0 con 0x7fc4700afe80 2024-03-28T14:08:49.958 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:49.950+0000 7fc478335640 1 -- 172.21.15.104:0/855614091 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc4701e3500 con 0x7fc4700afe80 2024-03-28T14:08:49.959 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:49.954+0000 7fc478335640 1 -- 172.21.15.104:0/855614091 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc4700687c0 con 0x7fc4700afe80 2024-03-28T14:08:49.959 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:49.954+0000 7fc466ffd640 1 -- 172.21.15.104:0/855614091 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) ==== 87803+0+0 (secure 0 0 0) 0x7fc4600b43f0 con 0x7fc4700afe80 2024-03-28T14:08:49.959 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:49.954+0000 7fc466ffd640 1 --2- 172.21.15.104:0/855614091 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7fc454067fc0 0x7fc45406a480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:49.960 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:49.954+0000 7fc466ffd640 1 -- 172.21.15.104:0/855614091 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 5709+0+0 (secure 0 0 0) 0x7fc460121500 con 0x7fc4700afe80 2024-03-28T14:08:49.960 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:49.954+0000 7fc4760d1640 1 --2- 172.21.15.104:0/855614091 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7fc454067fc0 0x7fc45406a480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:49.960 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:49.954+0000 7fc4760d1640 1 --2- 172.21.15.104:0/855614091 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7fc454067fc0 0x7fc45406a480 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fc46c009a40 tx=0x7fc46c008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:08:49.965 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:49.962+0000 7fc466ffd640 1 -- 172.21.15.104:0/855614091 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179117 (secure 0 0 0) 0x7fc4600a8050 con 0x7fc4700afe80 2024-03-28T14:08:50.125 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:50.122+0000 7fc478335640 1 -- 172.21.15.104:0/855614091 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fc4701e2fd0 con 0x7fc4700afe80 2024-03-28T14:08:50.126 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:50.122+0000 7fc466ffd640 1 -- 172.21.15.104:0/855614091 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v30) ==== 93+0+10 (secure 0 0 0) 0x7fc460016150 con 0x7fc4700afe80 2024-03-28T14:08:50.126 INFO:teuthology.orchestra.run.smithi104.stdout:pg_num: 1 2024-03-28T14:08:50.127 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:50.122+0000 7fc478335640 1 -- 172.21.15.104:0/855614091 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7fc454067fc0 msgr2=0x7fc45406a480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:50.128 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:50.122+0000 7fc478335640 1 --2- 172.21.15.104:0/855614091 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7fc454067fc0 0x7fc45406a480 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fc46c009a40 tx=0x7fc46c008040 comp rx=0 tx=0).stop 2024-03-28T14:08:50.128 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:50.122+0000 7fc478335640 1 -- 172.21.15.104:0/855614091 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc4700afe80 msgr2=0x7fc4701dfa60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:50.128 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:50.122+0000 7fc478335640 1 --2- 172.21.15.104:0/855614091 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc4700afe80 0x7fc4701dfa60 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7fc46001aac0 tx=0x7fc46001ac50 comp rx=0 tx=0).stop 2024-03-28T14:08:50.128 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:50.122+0000 7fc478335640 1 -- 172.21.15.104:0/855614091 shutdown_connections 2024-03-28T14:08:50.128 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:50.122+0000 7fc478335640 1 --2- 172.21.15.104:0/855614091 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fc4700ba900 0x7fc47013d0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:50.128 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:50.122+0000 7fc478335640 1 --2- 172.21.15.104:0/855614091 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7fc454067fc0 0x7fc45406a480 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:50.128 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:50.122+0000 7fc478335640 1 --2- 172.21.15.104:0/855614091 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc4700afe80 0x7fc4701dfa60 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:50.128 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:50.122+0000 7fc478335640 1 --2- 172.21.15.104:0/855614091 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fc47006bb10 0x7fc47014a660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:50.128 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:50.122+0000 7fc478335640 1 -- 172.21.15.104:0/855614091 >> 172.21.15.104:0/855614091 conn(0x7fc47006cd40 msgr2=0x7fc470067330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T14:08:50.128 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:50.122+0000 7fc478335640 1 -- 172.21.15.104:0/855614091 shutdown_connections 2024-03-28T14:08:50.128 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:50.122+0000 7fc478335640 1 -- 172.21.15.104:0/855614091 wait complete. 2024-03-28T14:08:50.143 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-28T14:08:50.257 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:50.254+0000 7f85dcf51640 1 Processor -- start 2024-03-28T14:08:50.257 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:50.254+0000 7f85dcf51640 1 -- start start 2024-03-28T14:08:50.260 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:50.254+0000 7f85dcf51640 1 --2- >> v2:172.21.15.171:3300/0 conn(0x7f85d8144cf0 0x7f85d80b0010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:50.260 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:50.254+0000 7f85dcf51640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f85d80b6780 con 0x7f85d80af9b0 2024-03-28T14:08:50.260 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:50.254+0000 7f85dcf51640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f85d806dde0 con 0x7f85d80b63a0 2024-03-28T14:08:50.261 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:50.254+0000 7f85dcf51640 1 -- --> v2:172.21.15.171:3300/0 -- mon_getmap magic: 0 -- 0x7f85d80af610 con 0x7f85d8144cf0 2024-03-28T14:08:50.261 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:50.254+0000 7f85cffff640 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f85d80af9b0 0x7f85d81445e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:54730/0 (socket says 172.21.15.104:54730) 2024-03-28T14:08:50.261 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:50.254+0000 7f85cffff640 1 -- 172.21.15.104:0/3404001066 learned_addr learned my addr 172.21.15.104:0/3404001066 (peer_addr_for_me v1:172.21.15.104:0/0) 2024-03-28T14:08:50.261 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:50.254+0000 7f85d6d9d640 1 --2- 172.21.15.104:0/3404001066 >> v2:172.21.15.171:3300/0 conn(0x7f85d8144cf0 0x7f85d80b0010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:50.261 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:50.254+0000 7f85d5d9b640 1 -- 172.21.15.104:0/3404001066 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1947891418 0 0) 0x7f85d80b6780 con 0x7f85d80af9b0 2024-03-28T14:08:50.261 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:50.254+0000 7f85d5d9b640 1 -- 172.21.15.104:0/3404001066 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f85c4005740 con 0x7f85d80af9b0 2024-03-28T14:08:50.261 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:50.254+0000 7f85d5d9b640 1 -- 172.21.15.104:0/3404001066 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 433925708 0 0) 0x7f85d806dde0 con 0x7f85d80b63a0 2024-03-28T14:08:50.261 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:50.254+0000 7f85d5d9b640 1 -- 172.21.15.104:0/3404001066 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f85d80b6780 con 0x7f85d80b63a0 2024-03-28T14:08:50.261 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:50.254+0000 7f85d5d9b640 1 -- 172.21.15.104:0/3404001066 <== mon.0 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3344132015 0 0) 0x7f85c4005740 con 0x7f85d80af9b0 2024-03-28T14:08:50.261 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:50.254+0000 7f85d5d9b640 1 -- 172.21.15.104:0/3404001066 >> v2:172.21.15.171:3300/0 conn(0x7f85d8144cf0 msgr2=0x7f85d80b0010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T14:08:50.261 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:50.254+0000 7f85d5d9b640 1 --2- 172.21.15.104:0/3404001066 >> v2:172.21.15.171:3300/0 conn(0x7f85d8144cf0 0x7f85d80b0010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:50.261 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:50.254+0000 7f85d5d9b640 1 -- 172.21.15.104:0/3404001066 >> v1:172.21.15.171:6789/0 conn(0x7f85d80b63a0 legacy=0x7f85d8141670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:50.261 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:50.254+0000 7f85d5d9b640 1 -- 172.21.15.104:0/3404001066 --> v1:172.21.15.104:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f85d806e0c0 con 0x7f85d80af9b0 2024-03-28T14:08:50.262 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:50.254+0000 7f85d5d9b640 1 -- 172.21.15.104:0/3404001066 <== mon.0 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3745580849 0 0) 0x7f85c0001680 con 0x7f85d80af9b0 2024-03-28T14:08:50.262 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:50.254+0000 7f85d5d9b640 1 -- 172.21.15.104:0/3404001066 >> v1:172.21.15.104:6789/0 conn(0x7f85d80af9b0 legacy=0x7f85d81445e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:50.262 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:50.254+0000 7f85d5d9b640 1 --2- 172.21.15.104:0/3404001066 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f85c4005d60 0x7f85c4006160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:50.262 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:50.254+0000 7f85d5d9b640 1 --2- 172.21.15.104:0/3404001066 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f85c4006a20 0x7f85c4006ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:50.262 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:50.254+0000 7f85d5d9b640 1 --2- 172.21.15.104:0/3404001066 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f85c40073e0 0x7f85c400dbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:50.262 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:50.254+0000 7f85d5d9b640 1 -- 172.21.15.104:0/3404001066 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f85c400e2e0 con 0x7f85c40073e0 2024-03-28T14:08:50.262 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:50.254+0000 7f85d5d9b640 1 -- 172.21.15.104:0/3404001066 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f85c400e490 con 0x7f85c4006a20 2024-03-28T14:08:50.262 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:50.254+0000 7f85d5d9b640 1 -- 172.21.15.104:0/3404001066 --> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] -- mon_getmap magic: 0 -- 0x7f85c400e690 con 0x7f85c4005d60 2024-03-28T14:08:50.262 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:50.254+0000 7f85d6d9d640 1 --2- 172.21.15.104:0/3404001066 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f85c40073e0 0x7f85c400dbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:50.262 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:50.254+0000 7f85cffff640 1 --2- 172.21.15.104:0/3404001066 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f85c4006a20 0x7f85c4006ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:50.262 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:50.254+0000 7f85d659c640 1 --2- 172.21.15.104:0/3404001066 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f85c4005d60 0x7f85c4006160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:50.262 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:50.254+0000 7f85d6d9d640 1 -- 172.21.15.104:0/3404001066 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f85c4005d60 msgr2=0x7f85c4006160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T14:08:50.262 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:50.254+0000 7f85d6d9d640 1 --2- 172.21.15.104:0/3404001066 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f85c4005d60 0x7f85c4006160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:50.263 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:50.254+0000 7f85d6d9d640 1 -- 172.21.15.104:0/3404001066 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f85c4006a20 msgr2=0x7f85c4006ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T14:08:50.263 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:50.254+0000 7f85d6d9d640 1 --2- 172.21.15.104:0/3404001066 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f85c4006a20 0x7f85c4006ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:50.263 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:50.254+0000 7f85d6d9d640 1 -- 172.21.15.104:0/3404001066 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f85c4005740 con 0x7f85c40073e0 2024-03-28T14:08:50.263 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:50.254+0000 7f85cffff640 1 --2- 172.21.15.104:0/3404001066 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f85c4006a20 0x7f85c4006ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T14:08:50.266 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:50.258+0000 7f85d6d9d640 1 -- 172.21.15.104:0/3404001066 reap_dead start 2024-03-28T14:08:50.266 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:50.258+0000 7f85d6d9d640 1 --2- 172.21.15.104:0/3404001066 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f85c40073e0 0x7f85c400dbe0 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f85d0009fc0 tx=0x7f85d001ccc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c0d0494651b75680 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:08:50.266 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:50.258+0000 7f85d5d9b640 1 -- 172.21.15.104:0/3404001066 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85d0009c10 con 0x7f85c40073e0 2024-03-28T14:08:50.266 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:50.258+0000 7f85d5d9b640 1 -- 172.21.15.104:0/3404001066 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f85d00a5cd0 con 0x7f85c40073e0 2024-03-28T14:08:50.266 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:50.258+0000 7f85d5d9b640 1 -- 172.21.15.104:0/3404001066 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85d0003be0 con 0x7f85c40073e0 2024-03-28T14:08:50.266 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:50.258+0000 7f85dcf51640 1 -- 172.21.15.104:0/3404001066 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f85c40073e0 msgr2=0x7f85c400dbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:50.267 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:50.258+0000 7f85dcf51640 1 --2- 172.21.15.104:0/3404001066 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f85c40073e0 0x7f85c400dbe0 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f85d0009fc0 tx=0x7f85d001ccc0 comp rx=0 tx=0).stop 2024-03-28T14:08:50.267 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:50.258+0000 7f85dcf51640 1 -- 172.21.15.104:0/3404001066 shutdown_connections 2024-03-28T14:08:50.267 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:50.258+0000 7f85dcf51640 1 --2- 172.21.15.104:0/3404001066 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f85c40073e0 0x7f85c400dbe0 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:50.267 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:50.258+0000 7f85dcf51640 1 -- 172.21.15.104:0/3404001066 >> 172.21.15.104:0/3404001066 conn(0x7f85d80ba400 msgr2=0x7f85d80ba810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T14:08:50.267 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:50.258+0000 7f85dcf51640 1 -- 172.21.15.104:0/3404001066 shutdown_connections 2024-03-28T14:08:50.267 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:50.258+0000 7f85dcf51640 1 -- 172.21.15.104:0/3404001066 wait complete. 2024-03-28T14:08:50.267 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:50.262+0000 7f85dcf51640 1 Processor -- start 2024-03-28T14:08:50.267 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:50.262+0000 7f85dcf51640 1 -- start start 2024-03-28T14:08:50.267 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:50.262+0000 7f85dcf51640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f85c40073e0 0x7f85d809f8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:50.267 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:50.262+0000 7f85dcf51640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f85d81cbfd0 0x7f85d81ca5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:50.267 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:50.262+0000 7f85dcf51640 1 --2- >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f85d81cc450 0x7f85d81cab10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:50.267 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:50.262+0000 7f85dcf51640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f85d80b03a0 con 0x7f85c40073e0 2024-03-28T14:08:50.267 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:50.262+0000 7f85dcf51640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f85d80b0220 con 0x7f85d81cbfd0 2024-03-28T14:08:50.267 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:50.262+0000 7f85dcf51640 1 -- --> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] -- mon_getmap magic: 0 -- 0x7f85d81445e0 con 0x7f85d81cc450 2024-03-28T14:08:50.268 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:50.262+0000 7f85d659c640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f85c40073e0 0x7f85d809f8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:50.268 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:50.262+0000 7f85d659c640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f85c40073e0 0x7f85d809f8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:37764/0 (socket says 172.21.15.104:37764) 2024-03-28T14:08:50.268 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:50.262+0000 7f85d659c640 1 -- 172.21.15.104:0/3540539236 learned_addr learned my addr 172.21.15.104:0/3540539236 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-03-28T14:08:50.268 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:50.262+0000 7f85cffff640 1 --2- 172.21.15.104:0/3540539236 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f85d81cbfd0 0x7f85d81ca5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:50.268 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:50.262+0000 7f85d6d9d640 1 --2- 172.21.15.104:0/3540539236 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f85d81cc450 0x7f85d81cab10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:50.268 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:50.262+0000 7f85d659c640 1 -- 172.21.15.104:0/3540539236 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f85d81cc450 msgr2=0x7f85d81cab10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:50.268 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:50.262+0000 7f85d659c640 1 --2- 172.21.15.104:0/3540539236 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f85d81cc450 0x7f85d81cab10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:50.268 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:50.262+0000 7f85d659c640 1 -- 172.21.15.104:0/3540539236 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f85d81cbfd0 msgr2=0x7f85d81ca5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:50.268 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:50.262+0000 7f85d659c640 1 --2- 172.21.15.104:0/3540539236 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f85d81cbfd0 0x7f85d81ca5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:50.268 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:50.262+0000 7f85d659c640 1 -- 172.21.15.104:0/3540539236 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f85d813f260 con 0x7f85c40073e0 2024-03-28T14:08:50.268 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:50.262+0000 7f85cffff640 1 --2- 172.21.15.104:0/3540539236 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f85d81cbfd0 0x7f85d81ca5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T14:08:50.268 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:50.262+0000 7f85d659c640 1 --2- 172.21.15.104:0/3540539236 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f85c40073e0 0x7f85d809f8e0 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f85c80077d0 tx=0x7f85c800f540 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:08:50.268 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:50.262+0000 7f85ceffd640 1 -- 172.21.15.104:0/3540539236 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85c8010040 con 0x7f85c40073e0 2024-03-28T14:08:50.269 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:50.262+0000 7f85ceffd640 1 -- 172.21.15.104:0/3540539236 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f85c801a070 con 0x7f85c40073e0 2024-03-28T14:08:50.269 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:50.262+0000 7f85dcf51640 1 -- 172.21.15.104:0/3540539236 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f85d813d0c0 con 0x7f85c40073e0 2024-03-28T14:08:50.269 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:50.262+0000 7f85ceffd640 1 -- 172.21.15.104:0/3540539236 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85c800fe60 con 0x7f85c40073e0 2024-03-28T14:08:50.270 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:50.262+0000 7f85dcf51640 1 -- 172.21.15.104:0/3540539236 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f85d813ea60 con 0x7f85c40073e0 2024-03-28T14:08:50.271 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:50.262+0000 7f85dcf51640 1 -- 172.21.15.104:0/3540539236 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f85d80687c0 con 0x7f85c40073e0 2024-03-28T14:08:50.271 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:50.262+0000 7f85ceffd640 1 -- 172.21.15.104:0/3540539236 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) ==== 87803+0+0 (secure 0 0 0) 0x7f85c8015830 con 0x7f85c40073e0 2024-03-28T14:08:50.271 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:50.262+0000 7f85ceffd640 1 --2- 172.21.15.104:0/3540539236 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f85b8067fc0 0x7f85b806a480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:50.271 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:50.262+0000 7f85ceffd640 1 -- 172.21.15.104:0/3540539236 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 5709+0+0 (secure 0 0 0) 0x7f85c8089d00 con 0x7f85c40073e0 2024-03-28T14:08:50.271 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:50.262+0000 7f85cffff640 1 --2- 172.21.15.104:0/3540539236 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f85b8067fc0 0x7f85b806a480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:50.271 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:50.266+0000 7f85cffff640 1 --2- 172.21.15.104:0/3540539236 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f85b8067fc0 0x7f85b806a480 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f85c0001210 tx=0x7f85c0009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:08:50.272 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:50.266+0000 7f85ceffd640 1 -- 172.21.15.104:0/3540539236 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179117 (secure 0 0 0) 0x7f85c805a4b0 con 0x7f85c40073e0 2024-03-28T14:08:50.425 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:50.422+0000 7f85dcf51640 1 -- 172.21.15.104:0/3540539236 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f85d813e130 con 0x7f85c40073e0 2024-03-28T14:08:50.425 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:50.422+0000 7f85ceffd640 1 -- 172.21.15.104:0/3540539236 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v30) ==== 104+0+11 (secure 0 0 0) 0x7f85c8059db0 con 0x7f85c40073e0 2024-03-28T14:08:50.426 INFO:teuthology.orchestra.run.smithi104.stdout:pg_num: 32 2024-03-28T14:08:50.428 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:50.422+0000 7f85dcf51640 1 -- 172.21.15.104:0/3540539236 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f85b8067fc0 msgr2=0x7f85b806a480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:50.428 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:50.422+0000 7f85dcf51640 1 --2- 172.21.15.104:0/3540539236 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f85b8067fc0 0x7f85b806a480 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f85c0001210 tx=0x7f85c0009040 comp rx=0 tx=0).stop 2024-03-28T14:08:50.428 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:50.422+0000 7f85dcf51640 1 -- 172.21.15.104:0/3540539236 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f85c40073e0 msgr2=0x7f85d809f8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:50.428 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:50.422+0000 7f85dcf51640 1 --2- 172.21.15.104:0/3540539236 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f85c40073e0 0x7f85d809f8e0 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f85c80077d0 tx=0x7f85c800f540 comp rx=0 tx=0).stop 2024-03-28T14:08:50.428 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:50.422+0000 7f85dcf51640 1 -- 172.21.15.104:0/3540539236 shutdown_connections 2024-03-28T14:08:50.428 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:50.422+0000 7f85dcf51640 1 --2- 172.21.15.104:0/3540539236 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f85d81cc450 0x7f85d81cab10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:50.428 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:50.422+0000 7f85dcf51640 1 --2- 172.21.15.104:0/3540539236 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f85d81cbfd0 0x7f85d81ca5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:50.428 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:50.422+0000 7f85dcf51640 1 --2- 172.21.15.104:0/3540539236 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f85b8067fc0 0x7f85b806a480 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:50.428 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:50.422+0000 7f85dcf51640 1 --2- 172.21.15.104:0/3540539236 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f85c40073e0 0x7f85d809f8e0 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:50.428 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:50.422+0000 7f85dcf51640 1 -- 172.21.15.104:0/3540539236 >> 172.21.15.104:0/3540539236 conn(0x7f85d80ba400 msgr2=0x7f85d80b4900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T14:08:50.429 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:50.422+0000 7f85dcf51640 1 -- 172.21.15.104:0/3540539236 shutdown_connections 2024-03-28T14:08:50.429 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:50.422+0000 7f85dcf51640 1 -- 172.21.15.104:0/3540539236 wait complete. 2024-03-28T14:08:50.444 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-28T14:08:50.572 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:50.570+0000 7fca53be4640 1 Processor -- start 2024-03-28T14:08:50.572 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:50.570+0000 7fca53be4640 1 -- start start 2024-03-28T14:08:50.574 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:50.570+0000 7fca53be4640 1 --2- >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fca4c02c460 0x7fca4c0b54e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:50.574 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:50.570+0000 7fca53be4640 1 --2- >> v2:172.21.15.171:3300/0 conn(0x7fca4c09e8e0 0x7fca4c0b5a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:50.574 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:50.570+0000 7fca53be4640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fca4c02c840 con 0x7fca4c1480d0 2024-03-28T14:08:50.574 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:50.570+0000 7fca53be4640 1 -- --> v2:172.21.15.171:3300/0 -- mon_getmap magic: 0 -- 0x7fca4c147c40 con 0x7fca4c09e8e0 2024-03-28T14:08:50.574 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:50.570+0000 7fca53be4640 1 -- --> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] -- mon_getmap magic: 0 -- 0x7fca4c0aec70 con 0x7fca4c02c460 2024-03-28T14:08:50.574 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:50.570+0000 7fca51980640 1 --2- >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fca4c02c460 0x7fca4c0b54e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:50.574 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:50.570+0000 7fca5117f640 1 --2- >> v2:172.21.15.171:3300/0 conn(0x7fca4c09e8e0 0x7fca4c0b5a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:50.574 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:50.570+0000 7fca51980640 1 --2- >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fca4c02c460 0x7fca4c0b54e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.171:3301/0 says I am v2:172.21.15.104:36030/0 (socket says 172.21.15.104:36030) 2024-03-28T14:08:50.574 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:50.570+0000 7fca51980640 1 -- 172.21.15.104:0/3790281925 learned_addr learned my addr 172.21.15.104:0/3790281925 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-03-28T14:08:50.575 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:50.570+0000 7fca51980640 1 -- 172.21.15.104:0/3790281925 >> v2:172.21.15.171:3300/0 conn(0x7fca4c09e8e0 msgr2=0x7fca4c0b5a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T14:08:50.575 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:50.570+0000 7fca51980640 1 --2- 172.21.15.104:0/3790281925 >> v2:172.21.15.171:3300/0 conn(0x7fca4c09e8e0 0x7fca4c0b5a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:50.575 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:50.570+0000 7fca51980640 1 -- 172.21.15.104:0/3790281925 >> v1:172.21.15.171:6789/0 conn(0x7fca4c1480d0 legacy=0x7fca4c144640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:50.575 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:50.570+0000 7fca51980640 1 -- 172.21.15.104:0/3790281925 --> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fca4c0aef50 con 0x7fca4c02c460 2024-03-28T14:08:50.576 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:50.574+0000 7fca51980640 1 --2- 172.21.15.104:0/3790281925 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fca4c02c460 0x7fca4c0b54e0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fca44009e80 tx=0x7fca4401ca10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bcd145e62b45f700 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:08:50.581 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:50.574+0000 7fca5097e640 1 -- 172.21.15.104:0/3790281925 <== mon.2 v2:172.21.15.171:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca440a56e0 con 0x7fca4c02c460 2024-03-28T14:08:50.581 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:50.574+0000 7fca5097e640 1 -- 172.21.15.104:0/3790281925 <== mon.2 v2:172.21.15.171:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fca440a5880 con 0x7fca4c02c460 2024-03-28T14:08:50.581 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:50.574+0000 7fca5097e640 1 -- 172.21.15.104:0/3790281925 <== mon.2 v2:172.21.15.171:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca440a5c50 con 0x7fca4c02c460 2024-03-28T14:08:50.581 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:50.574+0000 7fca53be4640 1 -- 172.21.15.104:0/3790281925 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fca4c02c460 msgr2=0x7fca4c0b54e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:50.581 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:50.574+0000 7fca53be4640 1 --2- 172.21.15.104:0/3790281925 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fca4c02c460 0x7fca4c0b54e0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fca44009e80 tx=0x7fca4401ca10 comp rx=0 tx=0).stop 2024-03-28T14:08:50.581 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:50.574+0000 7fca53be4640 1 -- 172.21.15.104:0/3790281925 shutdown_connections 2024-03-28T14:08:50.581 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:50.574+0000 7fca53be4640 1 --2- 172.21.15.104:0/3790281925 >> v2:172.21.15.171:3300/0 conn(0x7fca4c09e8e0 0x7fca4c0b5a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:50.582 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:50.574+0000 7fca53be4640 1 --2- 172.21.15.104:0/3790281925 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fca4c02c460 0x7fca4c0b54e0 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:50.582 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:50.574+0000 7fca53be4640 1 -- 172.21.15.104:0/3790281925 >> 172.21.15.104:0/3790281925 conn(0x7fca4c0b95f0 msgr2=0x7fca4c0b99d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T14:08:50.582 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:50.574+0000 7fca53be4640 1 -- 172.21.15.104:0/3790281925 shutdown_connections 2024-03-28T14:08:50.582 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:50.574+0000 7fca53be4640 1 -- 172.21.15.104:0/3790281925 wait complete. 2024-03-28T14:08:50.582 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:50.574+0000 7fca53be4640 1 Processor -- start 2024-03-28T14:08:50.582 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:50.574+0000 7fca53be4640 1 -- start start 2024-03-28T14:08:50.582 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:50.574+0000 7fca53be4640 1 --2- >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fca4c02c460 0x7fca4c072120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:50.582 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:50.574+0000 7fca53be4640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fca4c09e8e0 0x7fca4c0b3100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:50.582 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:50.574+0000 7fca53be4640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fca4c1480d0 0x7fca4c0b3640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:50.583 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:50.574+0000 7fca53be4640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7fca4c06d000 con 0x7fca4c1480d0 2024-03-28T14:08:50.583 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:50.574+0000 7fca53be4640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7fca4c0b0830 con 0x7fca4c09e8e0 2024-03-28T14:08:50.583 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:50.574+0000 7fca53be4640 1 -- --> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] -- mon_getmap magic: 0 -- 0x7fca4c06d180 con 0x7fca4c02c460 2024-03-28T14:08:50.583 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:50.574+0000 7fca52181640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fca4c1480d0 0x7fca4c0b3640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:50.583 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:50.574+0000 7fca52181640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fca4c1480d0 0x7fca4c0b3640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:37774/0 (socket says 172.21.15.104:37774) 2024-03-28T14:08:50.583 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:50.574+0000 7fca52181640 1 -- 172.21.15.104:0/1847691734 learned_addr learned my addr 172.21.15.104:0/1847691734 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-03-28T14:08:50.583 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:50.574+0000 7fca51980640 1 --2- 172.21.15.104:0/1847691734 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fca4c02c460 0x7fca4c072120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:50.583 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:50.574+0000 7fca5117f640 1 --2- 172.21.15.104:0/1847691734 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fca4c09e8e0 0x7fca4c0b3100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:50.583 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:50.574+0000 7fca52181640 1 -- 172.21.15.104:0/1847691734 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fca4c02c460 msgr2=0x7fca4c072120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:50.583 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:50.574+0000 7fca52181640 1 --2- 172.21.15.104:0/1847691734 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fca4c02c460 0x7fca4c072120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:50.583 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:50.574+0000 7fca52181640 1 -- 172.21.15.104:0/1847691734 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fca4c09e8e0 msgr2=0x7fca4c0b3100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:50.583 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:50.574+0000 7fca52181640 1 --2- 172.21.15.104:0/1847691734 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fca4c09e8e0 0x7fca4c0b3100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:50.583 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:50.574+0000 7fca52181640 1 -- 172.21.15.104:0/1847691734 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fca4c0b09b0 con 0x7fca4c1480d0 2024-03-28T14:08:50.583 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:50.574+0000 7fca51980640 1 --2- 172.21.15.104:0/1847691734 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fca4c02c460 0x7fca4c072120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T14:08:50.584 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:50.574+0000 7fca52181640 1 --2- 172.21.15.104:0/1847691734 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fca4c1480d0 0x7fca4c0b3640 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7fca4800c370 tx=0x7fca4800c830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:08:50.584 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:50.574+0000 7fca427fc640 1 -- 172.21.15.104:0/1847691734 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca48005220 con 0x7fca4c1480d0 2024-03-28T14:08:50.584 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:50.574+0000 7fca427fc640 1 -- 172.21.15.104:0/1847691734 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fca480053c0 con 0x7fca4c1480d0 2024-03-28T14:08:50.584 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:50.574+0000 7fca53be4640 1 -- 172.21.15.104:0/1847691734 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fca4c071540 con 0x7fca4c1480d0 2024-03-28T14:08:50.584 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:50.574+0000 7fca427fc640 1 -- 172.21.15.104:0/1847691734 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca48007270 con 0x7fca4c1480d0 2024-03-28T14:08:50.584 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:50.578+0000 7fca53be4640 1 -- 172.21.15.104:0/1847691734 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fca4c073ab0 con 0x7fca4c1480d0 2024-03-28T14:08:50.584 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:50.578+0000 7fca53be4640 1 -- 172.21.15.104:0/1847691734 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fca4c0687c0 con 0x7fca4c1480d0 2024-03-28T14:08:50.585 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:50.578+0000 7fca427fc640 1 -- 172.21.15.104:0/1847691734 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) ==== 87803+0+0 (secure 0 0 0) 0x7fca48019020 con 0x7fca4c1480d0 2024-03-28T14:08:50.585 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:50.578+0000 7fca427fc640 1 --2- 172.21.15.104:0/1847691734 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7fca30067ff0 0x7fca3006a4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:50.585 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:50.578+0000 7fca427fc640 1 -- 172.21.15.104:0/1847691734 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 5709+0+0 (secure 0 0 0) 0x7fca48085840 con 0x7fca4c1480d0 2024-03-28T14:08:50.585 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:50.578+0000 7fca51980640 1 --2- 172.21.15.104:0/1847691734 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7fca30067ff0 0x7fca3006a4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:50.585 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:50.578+0000 7fca51980640 1 --2- 172.21.15.104:0/1847691734 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7fca30067ff0 0x7fca3006a4b0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fca440ae000 tx=0x7fca440a7070 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:08:50.586 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:50.582+0000 7fca427fc640 1 -- 172.21.15.104:0/1847691734 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179117 (secure 0 0 0) 0x7fca48055ff0 con 0x7fca4c1480d0 2024-03-28T14:08:50.768 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:50.766+0000 7fca53be4640 1 -- 172.21.15.104:0/1847691734 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fca4c09e150 con 0x7fca4c1480d0 2024-03-28T14:08:50.768 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:50.766+0000 7fca427fc640 1 -- 172.21.15.104:0/1847691734 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v30) ==== 100+0+11 (secure 0 0 0) 0x7fca480558f0 con 0x7fca4c1480d0 2024-03-28T14:08:50.769 INFO:teuthology.orchestra.run.smithi104.stdout:pg_num: 64 2024-03-28T14:08:50.771 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:50.766+0000 7fca53be4640 1 -- 172.21.15.104:0/1847691734 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7fca30067ff0 msgr2=0x7fca3006a4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:50.771 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:50.766+0000 7fca53be4640 1 --2- 172.21.15.104:0/1847691734 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7fca30067ff0 0x7fca3006a4b0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fca440ae000 tx=0x7fca440a7070 comp rx=0 tx=0).stop 2024-03-28T14:08:50.771 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:50.766+0000 7fca53be4640 1 -- 172.21.15.104:0/1847691734 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fca4c1480d0 msgr2=0x7fca4c0b3640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:50.771 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:50.766+0000 7fca53be4640 1 --2- 172.21.15.104:0/1847691734 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fca4c1480d0 0x7fca4c0b3640 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7fca4800c370 tx=0x7fca4800c830 comp rx=0 tx=0).stop 2024-03-28T14:08:50.771 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:50.766+0000 7fca53be4640 1 -- 172.21.15.104:0/1847691734 shutdown_connections 2024-03-28T14:08:50.771 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:50.766+0000 7fca53be4640 1 --2- 172.21.15.104:0/1847691734 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7fca30067ff0 0x7fca3006a4b0 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:50.772 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:50.766+0000 7fca53be4640 1 --2- 172.21.15.104:0/1847691734 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fca4c1480d0 0x7fca4c0b3640 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:50.772 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:50.766+0000 7fca53be4640 1 --2- 172.21.15.104:0/1847691734 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fca4c09e8e0 0x7fca4c0b3100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:50.772 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:50.766+0000 7fca53be4640 1 --2- 172.21.15.104:0/1847691734 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fca4c02c460 0x7fca4c072120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:50.772 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:50.766+0000 7fca53be4640 1 -- 172.21.15.104:0/1847691734 >> 172.21.15.104:0/1847691734 conn(0x7fca4c0b95f0 msgr2=0x7fca4c0b5fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T14:08:50.772 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:50.766+0000 7fca53be4640 1 -- 172.21.15.104:0/1847691734 shutdown_connections 2024-03-28T14:08:50.772 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:50.766+0000 7fca53be4640 1 -- 172.21.15.104:0/1847691734 wait complete. 2024-03-28T14:08:50.787 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-28T14:08:50.898 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:50.894+0000 7fbfcd6c7640 1 Processor -- start 2024-03-28T14:08:50.898 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:50.894+0000 7fbfcd6c7640 1 -- start start 2024-03-28T14:08:50.898 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:50.894+0000 7fbfcd6c7640 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fbfc806c800 0x7fbfc80b6a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:50.898 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:50.894+0000 7fbfcd6c7640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbfc8148810 con 0x7fbfc8072f80 2024-03-28T14:08:50.898 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:50.894+0000 7fbfcd6c7640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbfc8147d30 con 0x7fbfc80b6f60 2024-03-28T14:08:50.899 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:50.894+0000 7fbfcd6c7640 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 -- 0x7fbfc8148380 con 0x7fbfc806c800 2024-03-28T14:08:50.899 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:50.894+0000 7fbfbffff640 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fbfc806c800 0x7fbfc80b6a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:50.899 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:50.894+0000 7fbfbffff640 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fbfc806c800 0x7fbfc80b6a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:37784/0 (socket says 172.21.15.104:37784) 2024-03-28T14:08:50.899 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:50.894+0000 7fbfbffff640 1 -- 172.21.15.104:0/2613469564 learned_addr learned my addr 172.21.15.104:0/2613469564 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-03-28T14:08:50.899 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:50.894+0000 7fbfc6ffd640 1 --1- 172.21.15.104:0/2613469564 >> v1:172.21.15.104:6789/0 conn(0x7fbfc8072f80 0x7fbfc806d3a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:54736/0 (socket says 172.21.15.104:54736) 2024-03-28T14:08:50.899 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:50.894+0000 7fbfbffff640 1 -- 172.21.15.104:0/2613469564 >> v1:172.21.15.171:6789/0 conn(0x7fbfc80b6f60 legacy=0x7fbfc8152180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T14:08:50.899 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:50.894+0000 7fbfbffff640 1 -- 172.21.15.104:0/2613469564 >> v1:172.21.15.104:6789/0 conn(0x7fbfc8072f80 legacy=0x7fbfc806d3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:50.899 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:50.894+0000 7fbfbffff640 1 -- 172.21.15.104:0/2613469564 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbfc8148010 con 0x7fbfc806c800 2024-03-28T14:08:50.899 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:50.894+0000 7fbfc67fc640 1 -- 172.21.15.104:0/2613469564 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1050401111 0 0) 0x7fbfc8148810 con 0x7fbfc8072f80 2024-03-28T14:08:50.901 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:50.898+0000 7fbfbffff640 1 --2- 172.21.15.104:0/2613469564 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbfc806c800 0x7fbfc80b6a20 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7fbfb0009870 tx=0x7fbfb001c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=91d161a4cb8009d0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:08:50.901 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:50.898+0000 7fbfc67fc640 1 -- 172.21.15.104:0/2613469564 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbfb00b1070 con 0x7fbfc806c800 2024-03-28T14:08:50.901 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:50.898+0000 7fbfc67fc640 1 -- 172.21.15.104:0/2613469564 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbfb00ac600 con 0x7fbfc806c800 2024-03-28T14:08:50.901 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:50.898+0000 7fbfc67fc640 1 -- 172.21.15.104:0/2613469564 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbfb00ac9d0 con 0x7fbfc806c800 2024-03-28T14:08:50.901 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:50.898+0000 7fbfcd6c7640 1 -- 172.21.15.104:0/2613469564 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbfc806c800 msgr2=0x7fbfc80b6a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:50.902 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:50.898+0000 7fbfcd6c7640 1 --2- 172.21.15.104:0/2613469564 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbfc806c800 0x7fbfc80b6a20 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7fbfb0009870 tx=0x7fbfb001c490 comp rx=0 tx=0).stop 2024-03-28T14:08:50.902 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:50.898+0000 7fbfcd6c7640 1 -- 172.21.15.104:0/2613469564 shutdown_connections 2024-03-28T14:08:50.902 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:50.898+0000 7fbfcd6c7640 1 --2- 172.21.15.104:0/2613469564 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbfc806c800 0x7fbfc80b6a20 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:50.902 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:50.898+0000 7fbfcd6c7640 1 -- 172.21.15.104:0/2613469564 >> 172.21.15.104:0/2613469564 conn(0x7fbfc80b7b10 msgr2=0x7fbfc80b7f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T14:08:50.902 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:50.898+0000 7fbfcd6c7640 1 -- 172.21.15.104:0/2613469564 shutdown_connections 2024-03-28T14:08:50.902 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:50.898+0000 7fbfcd6c7640 1 -- 172.21.15.104:0/2613469564 wait complete. 2024-03-28T14:08:50.904 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:50.898+0000 7fbfcd6c7640 1 Processor -- start 2024-03-28T14:08:50.904 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:50.898+0000 7fbfcd6c7640 1 -- start start 2024-03-28T14:08:50.904 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:50.898+0000 7fbfcd6c7640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbfc806c800 0x7fbfc813d720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:50.904 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:50.898+0000 7fbfcd6c7640 1 --2- >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fbfc8072f80 0x7fbfc8149320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:50.904 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:50.898+0000 7fbfcd6c7640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fbfc80b6f60 0x7fbfc81383c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:50.904 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:50.898+0000 7fbfcd6c7640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7fbfc81566e0 con 0x7fbfc806c800 2024-03-28T14:08:50.904 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:50.898+0000 7fbfcd6c7640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7fbfc8147af0 con 0x7fbfc80b6f60 2024-03-28T14:08:50.904 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:50.898+0000 7fbfcd6c7640 1 -- --> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] -- mon_getmap magic: 0 -- 0x7fbfc8156860 con 0x7fbfc8072f80 2024-03-28T14:08:50.905 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:50.898+0000 7fbfc6ffd640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbfc806c800 0x7fbfc813d720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:50.905 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:50.898+0000 7fbfc6ffd640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbfc806c800 0x7fbfc813d720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:37786/0 (socket says 172.21.15.104:37786) 2024-03-28T14:08:50.905 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:50.898+0000 7fbfc6ffd640 1 -- 172.21.15.104:0/3048355362 learned_addr learned my addr 172.21.15.104:0/3048355362 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-03-28T14:08:50.905 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:50.898+0000 7fbfc6ffd640 1 -- 172.21.15.104:0/3048355362 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fbfc8072f80 msgr2=0x7fbfc8149320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:50.905 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:50.898+0000 7fbfc6ffd640 1 --2- 172.21.15.104:0/3048355362 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fbfc8072f80 0x7fbfc8149320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:50.905 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:50.898+0000 7fbfc6ffd640 1 -- 172.21.15.104:0/3048355362 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fbfc80b6f60 msgr2=0x7fbfc81383c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:50.905 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:50.898+0000 7fbfc6ffd640 1 --2- 172.21.15.104:0/3048355362 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fbfc80b6f60 0x7fbfc81383c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:50.905 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:50.898+0000 7fbfc6ffd640 1 -- 172.21.15.104:0/3048355362 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbfc8149860 con 0x7fbfc806c800 2024-03-28T14:08:50.905 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:50.898+0000 7fbfc6ffd640 1 --2- 172.21.15.104:0/3048355362 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbfc806c800 0x7fbfc813d720 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7fbfb80011f0 tx=0x7fbfb800c9b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:08:50.905 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:50.898+0000 7fbfbf7fe640 1 -- 172.21.15.104:0/3048355362 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbfb8018070 con 0x7fbfc806c800 2024-03-28T14:08:50.905 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:50.898+0000 7fbfbf7fe640 1 -- 172.21.15.104:0/3048355362 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbfb80074e0 con 0x7fbfc806c800 2024-03-28T14:08:50.905 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:50.898+0000 7fbfcd6c7640 1 -- 172.21.15.104:0/3048355362 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbfc8149b10 con 0x7fbfc806c800 2024-03-28T14:08:50.905 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:50.898+0000 7fbfbf7fe640 1 -- 172.21.15.104:0/3048355362 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbfb8004a10 con 0x7fbfc806c800 2024-03-28T14:08:50.906 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:50.898+0000 7fbfcd6c7640 1 -- 172.21.15.104:0/3048355362 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbfc8148a40 con 0x7fbfc806c800 2024-03-28T14:08:50.906 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:50.902+0000 7fbfcd6c7640 1 -- 172.21.15.104:0/3048355362 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbfc80687c0 con 0x7fbfc806c800 2024-03-28T14:08:50.907 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:50.902+0000 7fbfbf7fe640 1 -- 172.21.15.104:0/3048355362 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) ==== 87803+0+0 (secure 0 0 0) 0x7fbfb801b070 con 0x7fbfc806c800 2024-03-28T14:08:50.907 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:50.902+0000 7fbfbf7fe640 1 --2- 172.21.15.104:0/3048355362 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7fbfa8067ff0 0x7fbfa806a4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:50.907 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:50.902+0000 7fbfbf7fe640 1 -- 172.21.15.104:0/3048355362 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 5709+0+0 (secure 0 0 0) 0x7fbfb8085750 con 0x7fbfc806c800 2024-03-28T14:08:50.907 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:50.902+0000 7fbfbffff640 1 --2- 172.21.15.104:0/3048355362 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7fbfa8067ff0 0x7fbfa806a4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:50.907 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:50.902+0000 7fbfbffff640 1 --2- 172.21.15.104:0/3048355362 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7fbfa8067ff0 0x7fbfa806a4b0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fbfb00ae000 tx=0x7fbfb0008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:08:50.910 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:50.906+0000 7fbfbf7fe640 1 -- 172.21.15.104:0/3048355362 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179117 (secure 0 0 0) 0x7fbfb8055f90 con 0x7fbfc806c800 2024-03-28T14:08:51.071 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:51.066+0000 7fbfcd6c7640 1 -- 172.21.15.104:0/3048355362 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7fbfc81476c0 con 0x7fbfc806c800 2024-03-28T14:08:51.071 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:51.066+0000 7fbfbf7fe640 1 -- 172.21.15.104:0/3048355362 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v30) ==== 80+0+0 (secure 0 0 0) 0x7fbfb8055890 con 0x7fbfc806c800 2024-03-28T14:08:51.072 INFO:teuthology.orchestra.run.smithi104.stderr:listed 0 entries 2024-03-28T14:08:51.073 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:51.070+0000 7fbfcd6c7640 1 -- 172.21.15.104:0/3048355362 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7fbfa8067ff0 msgr2=0x7fbfa806a4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:51.073 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:51.070+0000 7fbfcd6c7640 1 --2- 172.21.15.104:0/3048355362 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7fbfa8067ff0 0x7fbfa806a4b0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fbfb00ae000 tx=0x7fbfb0008040 comp rx=0 tx=0).stop 2024-03-28T14:08:51.073 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:51.070+0000 7fbfcd6c7640 1 -- 172.21.15.104:0/3048355362 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbfc806c800 msgr2=0x7fbfc813d720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:51.073 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:51.070+0000 7fbfcd6c7640 1 --2- 172.21.15.104:0/3048355362 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbfc806c800 0x7fbfc813d720 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7fbfb80011f0 tx=0x7fbfb800c9b0 comp rx=0 tx=0).stop 2024-03-28T14:08:51.073 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:51.070+0000 7fbfcd6c7640 1 -- 172.21.15.104:0/3048355362 shutdown_connections 2024-03-28T14:08:51.074 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:51.070+0000 7fbfcd6c7640 1 --2- 172.21.15.104:0/3048355362 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fbfc80b6f60 0x7fbfc81383c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:51.074 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:51.070+0000 7fbfcd6c7640 1 --2- 172.21.15.104:0/3048355362 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fbfc8072f80 0x7fbfc8149320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:51.074 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:51.070+0000 7fbfcd6c7640 1 --2- 172.21.15.104:0/3048355362 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7fbfa8067ff0 0x7fbfa806a4b0 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:51.074 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:51.070+0000 7fbfcd6c7640 1 --2- 172.21.15.104:0/3048355362 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbfc806c800 0x7fbfc813d720 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:51.074 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:51.070+0000 7fbfcd6c7640 1 -- 172.21.15.104:0/3048355362 >> 172.21.15.104:0/3048355362 conn(0x7fbfc80b7b10 msgr2=0x7fbfc806d2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T14:08:51.074 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:51.070+0000 7fbfcd6c7640 1 -- 172.21.15.104:0/3048355362 shutdown_connections 2024-03-28T14:08:51.074 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:51.070+0000 7fbfcd6c7640 1 -- 172.21.15.104:0/3048355362 wait complete. 2024-03-28T14:08:51.088 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi104.front.sepia.ceph.com... 2024-03-28T14:08:51.088 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T14:08:51.089 DEBUG:teuthology.orchestra.run.smithi104:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2024-03-28T14:08:51.097 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi104.stderr:ceph-fuse[36837]: fuse finished with error 0 and tester_r 0 2024-03-28T14:08:51.098 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi104.front.sepia.ceph.com 2024-03-28T14:08:51.098 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T14:08:51.098 DEBUG:teuthology.orchestra.run.smithi104:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-03-28T14:08:51.231 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T14:08:51.364 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:51.362+0000 7f8dc87c6640 1 Processor -- start 2024-03-28T14:08:51.365 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:51.362+0000 7f8dc87c6640 1 -- start start 2024-03-28T14:08:51.367 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:51.362+0000 7f8dc87c6640 1 --2- >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f8dc009e8e0 0x7f8dc01480d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:51.367 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:51.362+0000 7f8dc87c6640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8dc0148aa0 con 0x7f8dc0148610 2024-03-28T14:08:51.368 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:51.362+0000 7f8dc87c6640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8dc002c840 con 0x7f8dc002c460 2024-03-28T14:08:51.368 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:51.362+0000 7f8dc87c6640 1 -- --> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] -- mon_getmap magic: 0 -- 0x7f8dc0147c40 con 0x7f8dc009e8e0 2024-03-28T14:08:51.368 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:51.362+0000 7f8dc6d63640 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f8dc0148610 0x7f8dc014ef20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:54748/0 (socket says 172.21.15.104:54748) 2024-03-28T14:08:51.368 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:51.362+0000 7f8dc6d63640 1 -- 172.21.15.104:0/3711243274 learned_addr learned my addr 172.21.15.104:0/3711243274 (peer_addr_for_me v1:172.21.15.104:0/0) 2024-03-28T14:08:51.368 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:51.362+0000 7f8dc5d61640 1 --2- 172.21.15.104:0/3711243274 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f8dc009e8e0 0x7f8dc01480d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:51.368 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:51.362+0000 7f8dc5560640 1 -- 172.21.15.104:0/3711243274 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3655270583 0 0) 0x7f8dc0148aa0 con 0x7f8dc0148610 2024-03-28T14:08:51.368 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:51.362+0000 7f8dc5560640 1 -- 172.21.15.104:0/3711243274 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8da8003530 con 0x7f8dc0148610 2024-03-28T14:08:51.368 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:51.362+0000 7f8dc5560640 1 -- 172.21.15.104:0/3711243274 <== mon.0 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3938714223 0 0) 0x7f8da8003530 con 0x7f8dc0148610 2024-03-28T14:08:51.368 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:51.362+0000 7f8dc5560640 1 -- 172.21.15.104:0/3711243274 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f8dc009e8e0 msgr2=0x7f8dc01480d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T14:08:51.368 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:51.362+0000 7f8dc5560640 1 --2- 172.21.15.104:0/3711243274 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f8dc009e8e0 0x7f8dc01480d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:51.368 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:51.362+0000 7f8dc5560640 1 -- 172.21.15.104:0/3711243274 >> v1:172.21.15.171:6789/0 conn(0x7f8dc002c460 legacy=0x7f8dc00b54e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:51.368 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:51.362+0000 7f8dc5560640 1 -- 172.21.15.104:0/3711243274 --> v1:172.21.15.104:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8dc00aec70 con 0x7f8dc0148610 2024-03-28T14:08:51.368 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:51.362+0000 7f8dc5560640 1 -- 172.21.15.104:0/3711243274 <== mon.0 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3745580849 0 0) 0x7f8db0001680 con 0x7f8dc0148610 2024-03-28T14:08:51.369 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:51.362+0000 7f8dc5560640 1 -- 172.21.15.104:0/3711243274 >> v1:172.21.15.104:6789/0 conn(0x7f8dc0148610 legacy=0x7f8dc014ef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:51.369 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:51.362+0000 7f8dc5560640 1 --2- 172.21.15.104:0/3711243274 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f8da80039f0 0x7f8da8003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:51.369 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:51.362+0000 7f8dc5560640 1 --2- 172.21.15.104:0/3711243274 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f8dc009e8e0 0x7f8da8004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:51.369 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:51.362+0000 7f8dc5560640 1 --2- 172.21.15.104:0/3711243274 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8da8004d50 0x7f8da80051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:51.369 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:51.362+0000 7f8dc5560640 1 -- 172.21.15.104:0/3711243274 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f8dc0147c40 con 0x7f8da8004d50 2024-03-28T14:08:51.369 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:51.362+0000 7f8dc5560640 1 -- 172.21.15.104:0/3711243274 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f8da800bae0 con 0x7f8da80039f0 2024-03-28T14:08:51.369 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:51.362+0000 7f8dc5560640 1 -- 172.21.15.104:0/3711243274 --> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] -- mon_getmap magic: 0 -- 0x7f8da800bce0 con 0x7f8dc009e8e0 2024-03-28T14:08:51.369 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:51.362+0000 7f8dc6d63640 1 --2- 172.21.15.104:0/3711243274 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8da8004d50 0x7f8da80051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:51.369 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:51.362+0000 7f8dc6d63640 1 -- 172.21.15.104:0/3711243274 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f8dc009e8e0 msgr2=0x7f8da8004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T14:08:51.369 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:51.362+0000 7f8dc6d63640 1 --2- 172.21.15.104:0/3711243274 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f8dc009e8e0 0x7f8da8004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:51.369 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:51.362+0000 7f8dc6d63640 1 -- 172.21.15.104:0/3711243274 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f8da80039f0 msgr2=0x7f8da8003e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T14:08:51.369 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:51.362+0000 7f8dc6d63640 1 --2- 172.21.15.104:0/3711243274 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f8da80039f0 0x7f8da8003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:51.369 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:51.362+0000 7f8dc6d63640 1 -- 172.21.15.104:0/3711243274 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8da8003530 con 0x7f8da8004d50 2024-03-28T14:08:51.371 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:51.366+0000 7f8dc6d63640 1 --2- 172.21.15.104:0/3711243274 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8da8004d50 0x7f8da80051d0 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f8db000c0e0 tx=0x7f8db001cb60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8b851bb0f1a7eed2 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:08:51.371 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:51.366+0000 7f8dc5560640 1 -- 172.21.15.104:0/3711243274 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8db0007c30 con 0x7f8da8004d50 2024-03-28T14:08:51.371 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:51.366+0000 7f8dc5560640 1 -- 172.21.15.104:0/3711243274 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8db00a9020 con 0x7f8da8004d50 2024-03-28T14:08:51.371 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:51.366+0000 7f8dc5560640 1 -- 172.21.15.104:0/3711243274 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8db0005b00 con 0x7f8da8004d50 2024-03-28T14:08:51.371 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:51.366+0000 7f8dc87c6640 1 -- 172.21.15.104:0/3711243274 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8da8004d50 msgr2=0x7f8da80051d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:51.371 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:51.366+0000 7f8dc87c6640 1 --2- 172.21.15.104:0/3711243274 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8da8004d50 0x7f8da80051d0 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f8db000c0e0 tx=0x7f8db001cb60 comp rx=0 tx=0).stop 2024-03-28T14:08:51.371 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:51.366+0000 7f8dc6d63640 1 -- 172.21.15.104:0/3711243274 reap_dead start 2024-03-28T14:08:51.371 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:51.366+0000 7f8dc87c6640 1 -- 172.21.15.104:0/3711243274 shutdown_connections 2024-03-28T14:08:51.372 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:51.366+0000 7f8dc87c6640 1 -- 172.21.15.104:0/3711243274 >> 172.21.15.104:0/3711243274 conn(0x7f8dc00b95f0 msgr2=0x7f8dc00b99d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T14:08:51.372 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:51.366+0000 7f8dc87c6640 1 -- 172.21.15.104:0/3711243274 shutdown_connections 2024-03-28T14:08:51.372 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:51.366+0000 7f8dc87c6640 1 -- 172.21.15.104:0/3711243274 wait complete. 2024-03-28T14:08:51.372 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:51.366+0000 7f8dc87c6640 1 Processor -- start 2024-03-28T14:08:51.372 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:51.366+0000 7f8dc87c6640 1 -- start start 2024-03-28T14:08:51.374 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:51.366+0000 7f8dc87c6640 1 --2- >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f8dc00b6740 0x7f8dc01d9f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:51.375 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:51.366+0000 7f8dc87c6640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8dc01450f0 0x7f8dc01da490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:51.376 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:51.366+0000 7f8dc87c6640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f8dc0072d20 0x7f8dc00731d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:51.376 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:51.366+0000 7f8dc87c6640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f8dc00b5bf0 con 0x7f8dc01450f0 2024-03-28T14:08:51.376 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:51.366+0000 7f8dc87c6640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f8dc006e470 con 0x7f8dc0072d20 2024-03-28T14:08:51.376 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:51.366+0000 7f8dc87c6640 1 -- --> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] -- mon_getmap magic: 0 -- 0x7f8dc00b5d70 con 0x7f8dc00b6740 2024-03-28T14:08:51.376 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:51.366+0000 7f8dc5d61640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8dc01450f0 0x7f8dc01da490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:51.376 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:51.366+0000 7f8dc5d61640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8dc01450f0 0x7f8dc01da490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:37816/0 (socket says 172.21.15.104:37816) 2024-03-28T14:08:51.376 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:51.366+0000 7f8dc5d61640 1 -- 172.21.15.104:0/1562629612 learned_addr learned my addr 172.21.15.104:0/1562629612 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-03-28T14:08:51.376 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:51.370+0000 7f8dc6562640 1 --2- 172.21.15.104:0/1562629612 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f8dc00b6740 0x7f8dc01d9f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:51.376 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:51.370+0000 7f8dc5d61640 1 -- 172.21.15.104:0/1562629612 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f8dc00b6740 msgr2=0x7f8dc01d9f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:51.376 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:51.370+0000 7f8dc6d63640 1 --2- 172.21.15.104:0/1562629612 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f8dc0072d20 0x7f8dc00731d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:51.376 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:51.370+0000 7f8dc5d61640 1 --2- 172.21.15.104:0/1562629612 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f8dc00b6740 0x7f8dc01d9f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:51.376 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:51.370+0000 7f8dc5d61640 1 -- 172.21.15.104:0/1562629612 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f8dc0072d20 msgr2=0x7f8dc00731d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:51.376 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:51.370+0000 7f8dc5d61640 1 --2- 172.21.15.104:0/1562629612 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f8dc0072d20 0x7f8dc00731d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:51.377 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:51.370+0000 7f8dc5d61640 1 -- 172.21.15.104:0/1562629612 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8dc013eb20 con 0x7f8dc01450f0 2024-03-28T14:08:51.377 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:51.370+0000 7f8dc6562640 1 --2- 172.21.15.104:0/1562629612 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f8dc00b6740 0x7f8dc01d9f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T14:08:51.377 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:51.370+0000 7f8dc5d61640 1 --2- 172.21.15.104:0/1562629612 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8dc01450f0 0x7f8dc01da490 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f8db40109d0 tx=0x7f8db4010e90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:08:51.377 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:51.370+0000 7f8dbeffd640 1 -- 172.21.15.104:0/1562629612 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8db401e070 con 0x7f8dc01450f0 2024-03-28T14:08:51.377 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:51.370+0000 7f8dbeffd640 1 -- 172.21.15.104:0/1562629612 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8db400edf0 con 0x7f8dc01450f0 2024-03-28T14:08:51.377 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:51.370+0000 7f8dc87c6640 1 -- 172.21.15.104:0/1562629612 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8dc00b6b20 con 0x7f8dc01450f0 2024-03-28T14:08:51.377 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:51.370+0000 7f8dbeffd640 1 -- 172.21.15.104:0/1562629612 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8db4011670 con 0x7f8dc01450f0 2024-03-28T14:08:51.377 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:51.370+0000 7f8dc87c6640 1 -- 172.21.15.104:0/1562629612 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8dc0144c60 con 0x7f8dc01450f0 2024-03-28T14:08:51.377 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:51.370+0000 7f8dc87c6640 1 -- 172.21.15.104:0/1562629612 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8dc00687c0 con 0x7f8dc01450f0 2024-03-28T14:08:51.377 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:51.370+0000 7f8dbeffd640 1 -- 172.21.15.104:0/1562629612 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) ==== 87803+0+0 (secure 0 0 0) 0x7f8db4021070 con 0x7f8dc01450f0 2024-03-28T14:08:51.377 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:51.370+0000 7f8dbeffd640 1 --2- 172.21.15.104:0/1562629612 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f8dac067fc0 0x7f8dac06a480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:51.377 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:51.370+0000 7f8dbeffd640 1 -- 172.21.15.104:0/1562629612 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 5709+0+0 (secure 0 0 0) 0x7f8db4088600 con 0x7f8dc01450f0 2024-03-28T14:08:51.377 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:51.370+0000 7f8dc6562640 1 --2- 172.21.15.104:0/1562629612 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f8dac067fc0 0x7f8dac06a480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:51.378 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:51.370+0000 7f8dc6562640 1 --2- 172.21.15.104:0/1562629612 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f8dac067fc0 0x7f8dac06a480 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f8dc00b7530 tx=0x7f8db8009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:08:51.379 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:51.374+0000 7f8dbeffd640 1 -- 172.21.15.104:0/1562629612 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179117 (secure 0 0 0) 0x7f8db4058e40 con 0x7f8dc01450f0 2024-03-28T14:08:51.561 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:51.558+0000 7f8dc87c6640 1 -- 172.21.15.104:0/1562629612 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f8dc009e150 con 0x7f8dc01450f0 2024-03-28T14:08:51.562 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:51.558+0000 7f8dbeffd640 1 -- 172.21.15.104:0/1562629612 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 14 v14) ==== 94+0+3537 (secure 0 0 0) 0x7f8db4058740 con 0x7f8dc01450f0 2024-03-28T14:08:51.562 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-03-28T14:08:51.562 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":14,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4550,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.171:6835/3418793776","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6834","nonce":3418793776},{"type":"v1","addr":"172.21.15.171:6835","nonce":3418793776}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11}],"filesystems":[{"mdsmap":{"epoch":14,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T14:08:40.854108+0000","modified":"2024-03-28T14:08:42.859657+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4959},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4959":{"gid":4959,"name":"a","rank":0,"incarnation":12,"state":"up:active","state_seq":3,"addr":"172.21.15.104:6833/1094990707","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":1094990707},{"type":"v1","addr":"172.21.15.104:6833","nonce":1094990707}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4959,"qdb_cluster":[4959]},"id":2}]} 2024-03-28T14:08:51.563 INFO:teuthology.orchestra.run.smithi104.stderr:dumped fsmap epoch 14 2024-03-28T14:08:51.564 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:51.562+0000 7f8dc87c6640 1 -- 172.21.15.104:0/1562629612 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f8dac067fc0 msgr2=0x7f8dac06a480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:51.564 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:51.562+0000 7f8dc87c6640 1 --2- 172.21.15.104:0/1562629612 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f8dac067fc0 0x7f8dac06a480 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f8dc00b7530 tx=0x7f8db8009040 comp rx=0 tx=0).stop 2024-03-28T14:08:51.565 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:51.562+0000 7f8dc87c6640 1 -- 172.21.15.104:0/1562629612 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8dc01450f0 msgr2=0x7f8dc01da490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:51.565 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:51.562+0000 7f8dc87c6640 1 --2- 172.21.15.104:0/1562629612 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8dc01450f0 0x7f8dc01da490 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f8db40109d0 tx=0x7f8db4010e90 comp rx=0 tx=0).stop 2024-03-28T14:08:51.565 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:51.562+0000 7f8dc87c6640 1 -- 172.21.15.104:0/1562629612 shutdown_connections 2024-03-28T14:08:51.565 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:51.562+0000 7f8dc87c6640 1 --2- 172.21.15.104:0/1562629612 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f8dc0072d20 0x7f8dc00731d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:51.565 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:51.562+0000 7f8dc87c6640 1 --2- 172.21.15.104:0/1562629612 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f8dac067fc0 0x7f8dac06a480 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:51.565 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:51.562+0000 7f8dc87c6640 1 --2- 172.21.15.104:0/1562629612 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8dc01450f0 0x7f8dc01da490 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:51.565 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:51.562+0000 7f8dc87c6640 1 --2- 172.21.15.104:0/1562629612 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f8dc00b6740 0x7f8dc01d9f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:51.565 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:51.562+0000 7f8dc87c6640 1 -- 172.21.15.104:0/1562629612 >> 172.21.15.104:0/1562629612 conn(0x7f8dc00b95f0 msgr2=0x7f8dc0067330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T14:08:51.565 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:51.562+0000 7f8dc87c6640 1 -- 172.21.15.104:0/1562629612 shutdown_connections 2024-03-28T14:08:51.565 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:51.562+0000 7f8dc87c6640 1 -- 172.21.15.104:0/1562629612 wait complete. 2024-03-28T14:08:51.582 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-28T14:08:51.688 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:51.682+0000 7fa48b2f6640 1 Processor -- start 2024-03-28T14:08:51.688 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:51.682+0000 7fa48b2f6640 1 -- start start 2024-03-28T14:08:51.690 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:51.686+0000 7fa48b2f6640 1 --2- >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fa4840af020 0x7fa4840b0ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:51.690 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:51.686+0000 7fa48b2f6640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa48406bc10 con 0x7fa48406b830 2024-03-28T14:08:51.690 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:51.686+0000 7fa48b2f6640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa4840b51e0 con 0x7fa4840b58b0 2024-03-28T14:08:51.690 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:51.686+0000 7fa48b2f6640 1 -- --> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] -- mon_getmap magic: 0 -- 0x7fa4840b1220 con 0x7fa4840af020 2024-03-28T14:08:51.690 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:51.686+0000 7fa489092640 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7fa48406b830 0x7fa484072d10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:54760/0 (socket says 172.21.15.104:54760) 2024-03-28T14:08:51.691 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:51.686+0000 7fa489092640 1 -- 172.21.15.104:0/2646941838 learned_addr learned my addr 172.21.15.104:0/2646941838 (peer_addr_for_me v1:172.21.15.104:0/0) 2024-03-28T14:08:51.691 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:51.686+0000 7fa47bfff640 1 -- 172.21.15.104:0/2646941838 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3585839565 0 0) 0x7fa48406bc10 con 0x7fa48406b830 2024-03-28T14:08:51.691 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:51.686+0000 7fa47bfff640 1 -- 172.21.15.104:0/2646941838 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa470003530 con 0x7fa48406b830 2024-03-28T14:08:51.691 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:51.686+0000 7fa488891640 1 --2- 172.21.15.104:0/2646941838 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fa4840af020 0x7fa4840b0ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:51.691 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:51.686+0000 7fa47bfff640 1 -- 172.21.15.104:0/2646941838 <== mon.0 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 851696310 0 0) 0x7fa470003530 con 0x7fa48406b830 2024-03-28T14:08:51.691 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:51.686+0000 7fa47bfff640 1 -- 172.21.15.104:0/2646941838 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fa4840af020 msgr2=0x7fa4840b0ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T14:08:51.691 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:51.686+0000 7fa47bfff640 1 --2- 172.21.15.104:0/2646941838 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fa4840af020 0x7fa4840b0ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:51.691 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:51.686+0000 7fa47bfff640 1 -- 172.21.15.104:0/2646941838 >> v1:172.21.15.171:6789/0 conn(0x7fa4840b58b0 legacy=0x7fa4841520e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:51.691 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:51.686+0000 7fa47bfff640 1 -- 172.21.15.104:0/2646941838 --> v1:172.21.15.104:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa4840b54c0 con 0x7fa48406b830 2024-03-28T14:08:51.691 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:51.686+0000 7fa47bfff640 1 -- 172.21.15.104:0/2646941838 <== mon.0 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3745580849 0 0) 0x7fa474001680 con 0x7fa48406b830 2024-03-28T14:08:51.691 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:51.686+0000 7fa488891640 1 --2- 172.21.15.104:0/2646941838 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fa4840af020 0x7fa4840b0ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T14:08:51.691 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:51.686+0000 7fa47bfff640 1 -- 172.21.15.104:0/2646941838 >> v1:172.21.15.104:6789/0 conn(0x7fa48406b830 legacy=0x7fa484072d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:51.691 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:51.686+0000 7fa47bfff640 1 --2- 172.21.15.104:0/2646941838 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fa4840af020 0x7fa4700039f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:51.691 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:51.686+0000 7fa47bfff640 1 --2- 172.21.15.104:0/2646941838 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa470004030 0x7fa470004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:51.692 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:51.686+0000 7fa47bfff640 1 --2- 172.21.15.104:0/2646941838 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fa470004da0 0x7fa470005220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:51.692 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:51.686+0000 7fa47bfff640 1 -- 172.21.15.104:0/2646941838 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7fa4840b1220 con 0x7fa470004030 2024-03-28T14:08:51.692 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:51.686+0000 7fa47bfff640 1 -- 172.21.15.104:0/2646941838 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7fa47000bcb0 con 0x7fa470004da0 2024-03-28T14:08:51.692 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:51.686+0000 7fa47bfff640 1 -- 172.21.15.104:0/2646941838 --> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] -- mon_getmap magic: 0 -- 0x7fa47000beb0 con 0x7fa4840af020 2024-03-28T14:08:51.692 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:51.686+0000 7fa488891640 1 --2- 172.21.15.104:0/2646941838 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa470004030 0x7fa470004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:51.692 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:51.686+0000 7fa489092640 1 --2- 172.21.15.104:0/2646941838 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fa4840af020 0x7fa4700039f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:51.692 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:51.686+0000 7fa488891640 1 -- 172.21.15.104:0/2646941838 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fa4840af020 msgr2=0x7fa4700039f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T14:08:51.692 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:51.686+0000 7fa488891640 1 --2- 172.21.15.104:0/2646941838 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fa4840af020 0x7fa4700039f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:51.692 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:51.686+0000 7fa488891640 1 -- 172.21.15.104:0/2646941838 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fa470004da0 msgr2=0x7fa470005220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T14:08:51.692 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:51.686+0000 7fa488891640 1 --2- 172.21.15.104:0/2646941838 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fa470004da0 0x7fa470005220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:51.692 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:51.686+0000 7fa488891640 1 -- 172.21.15.104:0/2646941838 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa470003530 con 0x7fa470004030 2024-03-28T14:08:51.696 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:51.686+0000 7fa488891640 1 --2- 172.21.15.104:0/2646941838 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa470004030 0x7fa470004810 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7fa4800098d0 tx=0x7fa48001c5d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=62cd2acb8c362eea server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:08:51.696 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:51.686+0000 7fa47bfff640 1 -- 172.21.15.104:0/2646941838 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa48000f040 con 0x7fa470004030 2024-03-28T14:08:51.696 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:51.686+0000 7fa47bfff640 1 -- 172.21.15.104:0/2646941838 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa4800af070 con 0x7fa470004030 2024-03-28T14:08:51.696 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:51.686+0000 7fa47bfff640 1 -- 172.21.15.104:0/2646941838 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa484073630 con 0x7fa470004030 2024-03-28T14:08:51.696 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:51.690+0000 7fa48b2f6640 1 -- 172.21.15.104:0/2646941838 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa470004030 msgr2=0x7fa470004810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:51.696 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:51.690+0000 7fa48b2f6640 1 --2- 172.21.15.104:0/2646941838 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa470004030 0x7fa470004810 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7fa4800098d0 tx=0x7fa48001c5d0 comp rx=0 tx=0).stop 2024-03-28T14:08:51.696 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:51.690+0000 7fa489893640 1 -- 172.21.15.104:0/2646941838 reap_dead start 2024-03-28T14:08:51.696 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:51.690+0000 7fa48b2f6640 1 -- 172.21.15.104:0/2646941838 shutdown_connections 2024-03-28T14:08:51.696 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:51.690+0000 7fa48b2f6640 1 -- 172.21.15.104:0/2646941838 >> 172.21.15.104:0/2646941838 conn(0x7fa4840ba400 msgr2=0x7fa4840ba810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T14:08:51.697 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:51.690+0000 7fa48b2f6640 1 -- 172.21.15.104:0/2646941838 shutdown_connections 2024-03-28T14:08:51.697 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:51.690+0000 7fa48b2f6640 1 -- 172.21.15.104:0/2646941838 wait complete. 2024-03-28T14:08:51.697 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:51.690+0000 7fa48b2f6640 1 Processor -- start 2024-03-28T14:08:51.697 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:51.690+0000 7fa48b2f6640 1 -- start start 2024-03-28T14:08:51.697 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:51.690+0000 7fa48b2f6640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa470004030 0x7fa484151960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:51.697 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:51.690+0000 7fa48b2f6640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fa484070d30 0x7fa4840711b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:51.697 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:51.690+0000 7fa48b2f6640 1 --2- >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fa484140650 0x7fa484140ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:51.697 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:51.690+0000 7fa48b2f6640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7fa4841543a0 con 0x7fa470004030 2024-03-28T14:08:51.697 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:51.690+0000 7fa48b2f6640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7fa484154220 con 0x7fa484070d30 2024-03-28T14:08:51.697 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:51.690+0000 7fa48b2f6640 1 -- --> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] -- mon_getmap magic: 0 -- 0x7fa484154520 con 0x7fa484140650 2024-03-28T14:08:51.697 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:51.690+0000 7fa489092640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa470004030 0x7fa484151960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:51.697 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:51.690+0000 7fa489092640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa470004030 0x7fa484151960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:37830/0 (socket says 172.21.15.104:37830) 2024-03-28T14:08:51.697 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:51.690+0000 7fa489092640 1 -- 172.21.15.104:0/3768895943 learned_addr learned my addr 172.21.15.104:0/3768895943 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-03-28T14:08:51.697 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:51.690+0000 7fa488891640 1 --2- 172.21.15.104:0/3768895943 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fa484070d30 0x7fa4840711b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:51.698 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:51.690+0000 7fa489893640 1 --2- 172.21.15.104:0/3768895943 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fa484140650 0x7fa484140ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:51.698 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:51.690+0000 7fa489092640 1 -- 172.21.15.104:0/3768895943 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fa484140650 msgr2=0x7fa484140ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:51.698 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:51.690+0000 7fa489092640 1 --2- 172.21.15.104:0/3768895943 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fa484140650 0x7fa484140ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:51.698 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:51.690+0000 7fa489092640 1 -- 172.21.15.104:0/3768895943 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fa484070d30 msgr2=0x7fa4840711b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:51.698 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:51.690+0000 7fa489092640 1 --2- 172.21.15.104:0/3768895943 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fa484070d30 0x7fa4840711b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:51.698 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:51.690+0000 7fa489092640 1 -- 172.21.15.104:0/3768895943 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa4840716f0 con 0x7fa470004030 2024-03-28T14:08:51.698 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:51.690+0000 7fa488891640 1 --2- 172.21.15.104:0/3768895943 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fa484070d30 0x7fa4840711b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T14:08:51.698 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:51.690+0000 7fa489893640 1 --2- 172.21.15.104:0/3768895943 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fa484140650 0x7fa484140ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T14:08:51.698 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:51.690+0000 7fa489092640 1 --2- 172.21.15.104:0/3768895943 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa470004030 0x7fa484151960 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7fa47400d770 tx=0x7fa47400dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:08:51.698 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:51.690+0000 7fa479ffb640 1 -- 172.21.15.104:0/3768895943 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa474019070 con 0x7fa470004030 2024-03-28T14:08:51.698 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:51.690+0000 7fa479ffb640 1 -- 172.21.15.104:0/3768895943 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa4740074e0 con 0x7fa470004030 2024-03-28T14:08:51.698 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:51.690+0000 7fa479ffb640 1 -- 172.21.15.104:0/3768895943 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4740049f0 con 0x7fa470004030 2024-03-28T14:08:51.698 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:51.690+0000 7fa48b2f6640 1 -- 172.21.15.104:0/3768895943 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa484145d50 con 0x7fa470004030 2024-03-28T14:08:51.698 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:51.690+0000 7fa48b2f6640 1 -- 172.21.15.104:0/3768895943 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa484152000 con 0x7fa470004030 2024-03-28T14:08:51.699 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:51.694+0000 7fa48b2f6640 1 -- 172.21.15.104:0/3768895943 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa4840687c0 con 0x7fa470004030 2024-03-28T14:08:51.699 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:51.694+0000 7fa479ffb640 1 -- 172.21.15.104:0/3768895943 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) ==== 87803+0+0 (secure 0 0 0) 0x7fa47401c070 con 0x7fa470004030 2024-03-28T14:08:51.699 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:51.694+0000 7fa479ffb640 1 --2- 172.21.15.104:0/3768895943 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7fa468067fc0 0x7fa46806a480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:51.699 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:51.694+0000 7fa479ffb640 1 -- 172.21.15.104:0/3768895943 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 5709+0+0 (secure 0 0 0) 0x7fa474085eb0 con 0x7fa470004030 2024-03-28T14:08:51.700 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:51.694+0000 7fa488891640 1 --2- 172.21.15.104:0/3768895943 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7fa468067fc0 0x7fa46806a480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:51.700 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:51.694+0000 7fa488891640 1 --2- 172.21.15.104:0/3768895943 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7fa468067fc0 0x7fa46806a480 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fa4800ae000 tx=0x7fa4800046d0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:08:51.702 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:51.698+0000 7fa479ffb640 1 -- 172.21.15.104:0/3768895943 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179117 (secure 0 0 0) 0x7fa4840716f0 con 0x7fa470004030 2024-03-28T14:08:51.860 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:51.854+0000 7fa48b2f6640 1 -- 172.21.15.104:0/3768895943 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fa4841492d0 con 0x7fa470004030 2024-03-28T14:08:51.860 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:51.854+0000 7fa479ffb640 1 -- 172.21.15.104:0/3768895943 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v30) ==== 74+0+16693 (secure 0 0 0) 0x7fa4740566f0 con 0x7fa470004030 2024-03-28T14:08:51.861 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-03-28T14:08:51.861 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":30,"fsid":"428a90f2-3351-435f-8fcf-f7e66c78a13e","created":"2024-03-28T14:07:34.862458+0000","modified":"2024-03-28T14:08:42.477796+0000","last_up_change":"2024-03-28T14:07:46.028122+0000","last_in_change":"2024-03-28T14:07:37.157486+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-28T14:07:47.622988+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_metadata","create_time":"2024-03-28T14:08:38.687068+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data","create_time":"2024-03-28T14:08:39.805358+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"30","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"1cb70540-4e42-402f-a446-aaec4ca83007","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6800","nonce":3745690918},{"type":"v1","addr":"172.21.15.104:6801","nonce":3745690918}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":3745690918},{"type":"v1","addr":"172.21.15.104:6803","nonce":3745690918}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":3745690918},{"type":"v1","addr":"172.21.15.104:6807","nonce":3745690918}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":3745690918},{"type":"v1","addr":"172.21.15.104:6805","nonce":3745690918}]},"public_addr":"172.21.15.104:6801/3745690918","cluster_addr":"172.21.15.104:6803/3745690918","heartbeat_back_addr":"172.21.15.104:6807/3745690918","heartbeat_front_addr":"172.21.15.104:6805/3745690918","state":["exists","up"]},{"osd":1,"uuid":"3ab7afdd-b980-418c-81f2-004e16de3dca","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":19438674},{"type":"v1","addr":"172.21.15.104:6810","nonce":19438674}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":19438674},{"type":"v1","addr":"172.21.15.104:6814","nonce":19438674}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":19438674},{"type":"v1","addr":"172.21.15.104:6823","nonce":19438674}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":19438674},{"type":"v1","addr":"172.21.15.104:6819","nonce":19438674}]},"public_addr":"172.21.15.104:6810/19438674","cluster_addr":"172.21.15.104:6814/19438674","heartbeat_back_addr":"172.21.15.104:6823/19438674","heartbeat_front_addr":"172.21.15.104:6819/19438674","state":["exists","up"]},{"osd":2,"uuid":"3e2651c8-ecae-4a9e-a053-27240a79eac2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":2352012626},{"type":"v1","addr":"172.21.15.104:6825","nonce":2352012626}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6826","nonce":2352012626},{"type":"v1","addr":"172.21.15.104:6827","nonce":2352012626}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":2352012626},{"type":"v1","addr":"172.21.15.104:6831","nonce":2352012626}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6828","nonce":2352012626},{"type":"v1","addr":"172.21.15.104:6829","nonce":2352012626}]},"public_addr":"172.21.15.104:6825/2352012626","cluster_addr":"172.21.15.104:6827/2352012626","heartbeat_back_addr":"172.21.15.104:6831/2352012626","heartbeat_front_addr":"172.21.15.104:6829/2352012626","state":["exists","up"]},{"osd":3,"uuid":"925c7e0a-38ef-47cd-b288-8bec8456fb17","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6809","nonce":3568347381},{"type":"v1","addr":"172.21.15.104:6811","nonce":3568347381}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6813","nonce":3568347381},{"type":"v1","addr":"172.21.15.104:6815","nonce":3568347381}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":3568347381},{"type":"v1","addr":"172.21.15.104:6822","nonce":3568347381}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6817","nonce":3568347381},{"type":"v1","addr":"172.21.15.104:6818","nonce":3568347381}]},"public_addr":"172.21.15.104:6811/3568347381","cluster_addr":"172.21.15.104:6815/3568347381","heartbeat_back_addr":"172.21.15.104:6822/3568347381","heartbeat_front_addr":"172.21.15.104:6818/3568347381","state":["exists","up"]},{"osd":4,"uuid":"4a8fac86-e595-4a0e-96e3-57edb3cc3f57","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6800","nonce":2107001077},{"type":"v1","addr":"172.21.15.171:6801","nonce":2107001077}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6802","nonce":2107001077},{"type":"v1","addr":"172.21.15.171:6803","nonce":2107001077}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6806","nonce":2107001077},{"type":"v1","addr":"172.21.15.171:6807","nonce":2107001077}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6804","nonce":2107001077},{"type":"v1","addr":"172.21.15.171:6805","nonce":2107001077}]},"public_addr":"172.21.15.171:6801/2107001077","cluster_addr":"172.21.15.171:6803/2107001077","heartbeat_back_addr":"172.21.15.171:6807/2107001077","heartbeat_front_addr":"172.21.15.171:6805/2107001077","state":["exists","up"]},{"osd":5,"uuid":"e48d50b0-beae-40ae-b15a-9a84dbfe88ab","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6808","nonce":2934826925},{"type":"v1","addr":"172.21.15.171:6810","nonce":2934826925}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6812","nonce":2934826925},{"type":"v1","addr":"172.21.15.171:6814","nonce":2934826925}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6819","nonce":2934826925},{"type":"v1","addr":"172.21.15.171:6821","nonce":2934826925}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6816","nonce":2934826925},{"type":"v1","addr":"172.21.15.171:6817","nonce":2934826925}]},"public_addr":"172.21.15.171:6810/2934826925","cluster_addr":"172.21.15.171:6814/2934826925","heartbeat_back_addr":"172.21.15.171:6821/2934826925","heartbeat_front_addr":"172.21.15.171:6817/2934826925","state":["exists","up"]},{"osd":6,"uuid":"5e7c3292-9936-4f35-b267-aed9d4e49323","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6824","nonce":436742919},{"type":"v1","addr":"172.21.15.171:6825","nonce":436742919}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6826","nonce":436742919},{"type":"v1","addr":"172.21.15.171:6827","nonce":436742919}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6830","nonce":436742919},{"type":"v1","addr":"172.21.15.171:6831","nonce":436742919}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6828","nonce":436742919},{"type":"v1","addr":"172.21.15.171:6829","nonce":436742919}]},"public_addr":"172.21.15.171:6825/436742919","cluster_addr":"172.21.15.171:6827/436742919","heartbeat_back_addr":"172.21.15.171:6831/436742919","heartbeat_front_addr":"172.21.15.171:6829/436742919","state":["exists","up"]},{"osd":7,"uuid":"339b8414-d7f0-4fb0-a446-efaf635cb7ff","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6809","nonce":2055631175},{"type":"v1","addr":"172.21.15.171:6811","nonce":2055631175}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6813","nonce":2055631175},{"type":"v1","addr":"172.21.15.171:6815","nonce":2055631175}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6822","nonce":2055631175},{"type":"v1","addr":"172.21.15.171:6823","nonce":2055631175}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6818","nonce":2055631175},{"type":"v1","addr":"172.21.15.171:6820","nonce":2055631175}]},"public_addr":"172.21.15.171:6811/2055631175","cluster_addr":"172.21.15.171:6815/2055631175","heartbeat_back_addr":"172.21.15.171:6823/2055631175","heartbeat_front_addr":"172.21.15.171:6820/2055631175","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T14:07:42.668881+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T14:07:42.708397+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T14:07:42.634191+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T14:07:42.733662+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T14:07:42.651541+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T14:07:42.783181+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T14:07:42.766310+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T14:07:42.670797+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-28T14:08:51.862 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:51.858+0000 7fa48b2f6640 1 -- 172.21.15.104:0/3768895943 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7fa468067fc0 msgr2=0x7fa46806a480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:51.862 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:51.858+0000 7fa48b2f6640 1 --2- 172.21.15.104:0/3768895943 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7fa468067fc0 0x7fa46806a480 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fa4800ae000 tx=0x7fa4800046d0 comp rx=0 tx=0).stop 2024-03-28T14:08:51.862 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:51.858+0000 7fa48b2f6640 1 -- 172.21.15.104:0/3768895943 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa470004030 msgr2=0x7fa484151960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:51.863 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:51.858+0000 7fa48b2f6640 1 --2- 172.21.15.104:0/3768895943 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa470004030 0x7fa484151960 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7fa47400d770 tx=0x7fa47400dc30 comp rx=0 tx=0).stop 2024-03-28T14:08:51.863 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:51.858+0000 7fa48b2f6640 1 -- 172.21.15.104:0/3768895943 shutdown_connections 2024-03-28T14:08:51.863 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:51.858+0000 7fa48b2f6640 1 --2- 172.21.15.104:0/3768895943 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fa484140650 0x7fa484140ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:51.863 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:51.858+0000 7fa48b2f6640 1 --2- 172.21.15.104:0/3768895943 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fa484070d30 0x7fa4840711b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:51.863 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:51.858+0000 7fa48b2f6640 1 --2- 172.21.15.104:0/3768895943 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7fa468067fc0 0x7fa46806a480 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:51.863 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:51.858+0000 7fa48b2f6640 1 --2- 172.21.15.104:0/3768895943 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa470004030 0x7fa484151960 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:51.863 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:51.858+0000 7fa48b2f6640 1 -- 172.21.15.104:0/3768895943 >> 172.21.15.104:0/3768895943 conn(0x7fa4840ba400 msgr2=0x7fa4840b7200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T14:08:51.863 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:51.858+0000 7fa48b2f6640 1 -- 172.21.15.104:0/3768895943 shutdown_connections 2024-03-28T14:08:51.863 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:51.858+0000 7fa48b2f6640 1 -- 172.21.15.104:0/3768895943 wait complete. 2024-03-28T14:08:51.877 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-28T14:07:47.622988+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-28T14:08:38.687068+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '29', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 5, 'pool_name': 'cephfs_data', 'create_time': '2024-03-28T14:08:39.805358+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '30', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-28T14:08:51.878 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-28T14:08:52.012 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:52.010+0000 7fb9c7904640 1 Processor -- start 2024-03-28T14:08:52.013 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:52.010+0000 7fb9c7904640 1 -- start start 2024-03-28T14:08:52.014 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:52.010+0000 7fb9c7904640 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fb9c013fc30 0x7fb9c0148d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:52.014 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:52.010+0000 7fb9c7904640 1 --2- >> v2:172.21.15.171:3300/0 conn(0x7fb9c006d510 0x7fb9c0149270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:52.014 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:52.010+0000 7fb9c7904640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb9c002c840 con 0x7fb9c002c460 2024-03-28T14:08:52.014 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:52.010+0000 7fb9c7904640 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 -- 0x7fb9c00b7ab0 con 0x7fb9c013fc30 2024-03-28T14:08:52.014 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:52.010+0000 7fb9c7904640 1 -- --> v2:172.21.15.171:3300/0 -- mon_getmap magic: 0 -- 0x7fb9c00b7460 con 0x7fb9c006d510 2024-03-28T14:08:52.014 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:52.010+0000 7fb9c4e9f640 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fb9c013fc30 0x7fb9c0148d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:52.015 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:52.010+0000 7fb9c4e9f640 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fb9c013fc30 0x7fb9c0148d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:37836/0 (socket says 172.21.15.104:37836) 2024-03-28T14:08:52.015 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:52.010+0000 7fb9c4e9f640 1 -- 172.21.15.104:0/3298846799 learned_addr learned my addr 172.21.15.104:0/3298846799 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-03-28T14:08:52.015 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:52.010+0000 7fb9c5ea1640 1 --2- 172.21.15.104:0/3298846799 >> v2:172.21.15.171:3300/0 conn(0x7fb9c006d510 0x7fb9c0149270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:52.015 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:52.010+0000 7fb9c4e9f640 1 -- 172.21.15.104:0/3298846799 >> v2:172.21.15.171:3300/0 conn(0x7fb9c006d510 msgr2=0x7fb9c0149270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T14:08:52.015 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:52.010+0000 7fb9c4e9f640 1 --2- 172.21.15.104:0/3298846799 >> v2:172.21.15.171:3300/0 conn(0x7fb9c006d510 0x7fb9c0149270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:52.015 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:52.010+0000 7fb9c4e9f640 1 -- 172.21.15.104:0/3298846799 >> v1:172.21.15.171:6789/0 conn(0x7fb9c002c460 legacy=0x7fb9c013f520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T14:08:52.015 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:52.010+0000 7fb9c4e9f640 1 -- 172.21.15.104:0/3298846799 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb9c00b7740 con 0x7fb9c013fc30 2024-03-28T14:08:52.017 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:52.014+0000 7fb9c4e9f640 1 --2- 172.21.15.104:0/3298846799 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb9c013fc30 0x7fb9c0148d30 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7fb9c0073de0 tx=0x7fb9b401c3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5f073072a5e04251 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:08:52.017 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:52.014+0000 7fb9b3fff640 1 -- 172.21.15.104:0/3298846799 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9b40b0070 con 0x7fb9c013fc30 2024-03-28T14:08:52.017 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:52.014+0000 7fb9b3fff640 1 -- 172.21.15.104:0/3298846799 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb9b40ab600 con 0x7fb9c013fc30 2024-03-28T14:08:52.017 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:52.014+0000 7fb9b3fff640 1 -- 172.21.15.104:0/3298846799 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9b40ab9d0 con 0x7fb9c013fc30 2024-03-28T14:08:52.018 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:52.014+0000 7fb9c7904640 1 -- 172.21.15.104:0/3298846799 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb9c013fc30 msgr2=0x7fb9c0148d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:52.018 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:52.014+0000 7fb9c7904640 1 --2- 172.21.15.104:0/3298846799 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb9c013fc30 0x7fb9c0148d30 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7fb9c0073de0 tx=0x7fb9b401c3c0 comp rx=0 tx=0).stop 2024-03-28T14:08:52.018 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:52.014+0000 7fb9c7904640 1 -- 172.21.15.104:0/3298846799 shutdown_connections 2024-03-28T14:08:52.018 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:52.014+0000 7fb9c7904640 1 --2- 172.21.15.104:0/3298846799 >> v2:172.21.15.171:3300/0 conn(0x7fb9c006d510 0x7fb9c0149270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:52.018 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:52.014+0000 7fb9c7904640 1 --2- 172.21.15.104:0/3298846799 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb9c013fc30 0x7fb9c0148d30 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:52.019 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:52.014+0000 7fb9c7904640 1 -- 172.21.15.104:0/3298846799 >> 172.21.15.104:0/3298846799 conn(0x7fb9c00b9660 msgr2=0x7fb9c00b9a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T14:08:52.019 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:52.014+0000 7fb9c7904640 1 -- 172.21.15.104:0/3298846799 shutdown_connections 2024-03-28T14:08:52.019 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:52.014+0000 7fb9c7904640 1 -- 172.21.15.104:0/3298846799 wait complete. 2024-03-28T14:08:52.019 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:52.014+0000 7fb9c7904640 1 Processor -- start 2024-03-28T14:08:52.019 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:52.014+0000 7fb9c7904640 1 -- start start 2024-03-28T14:08:52.020 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:52.014+0000 7fb9c7904640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb9c002c460 0x7fb9c009d790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:52.020 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:52.014+0000 7fb9c7904640 1 --2- >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fb9c006d510 0x7fb9c00b32a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:52.020 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:52.018+0000 7fb9c7904640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fb9c013fc30 0x7fb9c01ddd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:52.020 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:52.018+0000 7fb9c7904640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7fb9c0152a00 con 0x7fb9c002c460 2024-03-28T14:08:52.020 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:52.018+0000 7fb9c7904640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7fb9c00b94b0 con 0x7fb9c013fc30 2024-03-28T14:08:52.020 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:52.018+0000 7fb9c7904640 1 -- --> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] -- mon_getmap magic: 0 -- 0x7fb9c0152b80 con 0x7fb9c006d510 2024-03-28T14:08:52.022 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:52.018+0000 7fb9c56a0640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb9c002c460 0x7fb9c009d790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:52.022 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:52.018+0000 7fb9c56a0640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb9c002c460 0x7fb9c009d790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:37844/0 (socket says 172.21.15.104:37844) 2024-03-28T14:08:52.022 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:52.018+0000 7fb9c56a0640 1 -- 172.21.15.104:0/206951257 learned_addr learned my addr 172.21.15.104:0/206951257 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-03-28T14:08:52.022 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:52.018+0000 7fb9c5ea1640 1 --2- 172.21.15.104:0/206951257 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fb9c013fc30 0x7fb9c01ddd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:52.023 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:52.018+0000 7fb9c56a0640 1 -- 172.21.15.104:0/206951257 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fb9c006d510 msgr2=0x7fb9c00b32a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:52.023 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:52.018+0000 7fb9c4e9f640 1 --2- 172.21.15.104:0/206951257 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fb9c006d510 0x7fb9c00b32a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:52.023 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:52.018+0000 7fb9c56a0640 1 --2- 172.21.15.104:0/206951257 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fb9c006d510 0x7fb9c00b32a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:52.023 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:52.018+0000 7fb9c56a0640 1 -- 172.21.15.104:0/206951257 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fb9c013fc30 msgr2=0x7fb9c01ddd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:52.023 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:52.018+0000 7fb9c56a0640 1 --2- 172.21.15.104:0/206951257 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fb9c013fc30 0x7fb9c01ddd80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:52.023 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:52.018+0000 7fb9c56a0640 1 -- 172.21.15.104:0/206951257 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb9c009e5d0 con 0x7fb9c002c460 2024-03-28T14:08:52.023 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:52.018+0000 7fb9c5ea1640 1 --2- 172.21.15.104:0/206951257 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fb9c013fc30 0x7fb9c01ddd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T14:08:52.023 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:52.018+0000 7fb9c56a0640 1 --2- 172.21.15.104:0/206951257 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb9c002c460 0x7fb9c009d790 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7fb9bc00c450 tx=0x7fb9bc00c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:08:52.023 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:52.018+0000 7fb9b1ffb640 1 -- 172.21.15.104:0/206951257 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9bc012830 con 0x7fb9c002c460 2024-03-28T14:08:52.023 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:52.018+0000 7fb9b1ffb640 1 -- 172.21.15.104:0/206951257 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb9bc017070 con 0x7fb9c002c460 2024-03-28T14:08:52.023 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:52.018+0000 7fb9c7904640 1 -- 172.21.15.104:0/206951257 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb9c00996e0 con 0x7fb9c002c460 2024-03-28T14:08:52.023 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:52.018+0000 7fb9b1ffb640 1 -- 172.21.15.104:0/206951257 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9bc012c30 con 0x7fb9c002c460 2024-03-28T14:08:52.023 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:52.018+0000 7fb9c7904640 1 -- 172.21.15.104:0/206951257 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb9c0099330 con 0x7fb9c002c460 2024-03-28T14:08:52.024 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:52.018+0000 7fb9c7904640 1 -- 172.21.15.104:0/206951257 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb9c00b2d80 con 0x7fb9c002c460 2024-03-28T14:08:52.024 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:52.018+0000 7fb9b1ffb640 1 -- 172.21.15.104:0/206951257 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) ==== 87803+0+0 (secure 0 0 0) 0x7fb9bc01c460 con 0x7fb9c002c460 2024-03-28T14:08:52.024 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:52.018+0000 7fb9b1ffb640 1 --2- 172.21.15.104:0/206951257 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7fb9a4067fc0 0x7fb9a406a480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:52.024 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:52.018+0000 7fb9b1ffb640 1 -- 172.21.15.104:0/206951257 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 5709+0+0 (secure 0 0 0) 0x7fb9bc086d10 con 0x7fb9c002c460 2024-03-28T14:08:52.025 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:52.018+0000 7fb9c4e9f640 1 --2- 172.21.15.104:0/206951257 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7fb9a4067fc0 0x7fb9a406a480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:52.025 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:52.018+0000 7fb9c4e9f640 1 --2- 172.21.15.104:0/206951257 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7fb9a4067fc0 0x7fb9a406a480 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fb9b40ad000 tx=0x7fb9b4008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:08:52.031 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:52.026+0000 7fb9b1ffb640 1 -- 172.21.15.104:0/206951257 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179117 (secure 0 0 0) 0x7fb9bc057630 con 0x7fb9c002c460 2024-03-28T14:08:52.191 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:52.186+0000 7fb9c7904640 1 -- 172.21.15.104:0/206951257 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fb9c00687c0 con 0x7fb9c002c460 2024-03-28T14:08:52.192 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:52.190+0000 7fb9b1ffb640 1 -- 172.21.15.104:0/206951257 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v30) ==== 93+0+10 (secure 0 0 0) 0x7fb9bc056f30 con 0x7fb9c002c460 2024-03-28T14:08:52.192 INFO:teuthology.orchestra.run.smithi104.stdout:pg_num: 1 2024-03-28T14:08:52.194 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:52.190+0000 7fb9c7904640 1 -- 172.21.15.104:0/206951257 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7fb9a4067fc0 msgr2=0x7fb9a406a480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:52.194 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:52.190+0000 7fb9c7904640 1 --2- 172.21.15.104:0/206951257 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7fb9a4067fc0 0x7fb9a406a480 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fb9b40ad000 tx=0x7fb9b4008040 comp rx=0 tx=0).stop 2024-03-28T14:08:52.194 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:52.190+0000 7fb9c7904640 1 -- 172.21.15.104:0/206951257 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb9c002c460 msgr2=0x7fb9c009d790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:52.195 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:52.190+0000 7fb9c7904640 1 --2- 172.21.15.104:0/206951257 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb9c002c460 0x7fb9c009d790 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7fb9bc00c450 tx=0x7fb9bc00c910 comp rx=0 tx=0).stop 2024-03-28T14:08:52.195 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:52.190+0000 7fb9c7904640 1 -- 172.21.15.104:0/206951257 shutdown_connections 2024-03-28T14:08:52.195 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:52.190+0000 7fb9c7904640 1 --2- 172.21.15.104:0/206951257 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fb9c013fc30 0x7fb9c01ddd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:52.195 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:52.190+0000 7fb9c7904640 1 --2- 172.21.15.104:0/206951257 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fb9c006d510 0x7fb9c00b32a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:52.195 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:52.190+0000 7fb9c7904640 1 --2- 172.21.15.104:0/206951257 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7fb9a4067fc0 0x7fb9a406a480 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:52.195 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:52.190+0000 7fb9c7904640 1 --2- 172.21.15.104:0/206951257 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb9c002c460 0x7fb9c009d790 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:52.195 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:52.190+0000 7fb9c7904640 1 -- 172.21.15.104:0/206951257 >> 172.21.15.104:0/206951257 conn(0x7fb9c00b9660 msgr2=0x7fb9c013a1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T14:08:52.195 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:52.190+0000 7fb9c7904640 1 -- 172.21.15.104:0/206951257 shutdown_connections 2024-03-28T14:08:52.195 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:52.190+0000 7fb9c7904640 1 -- 172.21.15.104:0/206951257 wait complete. 2024-03-28T14:08:52.211 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-28T14:08:52.340 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:52.338+0000 7f6e8bb57640 1 Processor -- start 2024-03-28T14:08:52.340 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:52.338+0000 7f6e8bb57640 1 -- start start 2024-03-28T14:08:52.342 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:52.338+0000 7f6e8bb57640 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f6e8406de10 0x7f6e8414b6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:52.342 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:52.338+0000 7f6e8bb57640 1 --2- >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f6e840b0b40 0x7f6e841411a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:52.342 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:52.338+0000 7f6e8bb57640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6e840af530 con 0x7f6e841416e0 2024-03-28T14:08:52.342 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:52.338+0000 7f6e8bb57640 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 -- 0x7f6e8414bc30 con 0x7f6e8406de10 2024-03-28T14:08:52.342 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:52.338+0000 7f6e8bb57640 1 -- --> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] -- mon_getmap magic: 0 -- 0x7f6e84148720 con 0x7f6e840b0b40 2024-03-28T14:08:52.342 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:52.338+0000 7f6e898f3640 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f6e8406de10 0x7f6e8414b6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:52.342 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:52.338+0000 7f6e898f3640 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f6e8406de10 0x7f6e8414b6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:37856/0 (socket says 172.21.15.104:37856) 2024-03-28T14:08:52.342 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:52.338+0000 7f6e898f3640 1 -- 172.21.15.104:0/1897869225 learned_addr learned my addr 172.21.15.104:0/1897869225 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-03-28T14:08:52.342 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:52.338+0000 7f6e898f3640 1 -- 172.21.15.104:0/1897869225 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f6e840b0b40 msgr2=0x7f6e841411a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T14:08:52.342 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:52.338+0000 7f6e898f3640 1 --2- 172.21.15.104:0/1897869225 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f6e840b0b40 0x7f6e841411a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:52.342 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:52.338+0000 7f6e898f3640 1 -- 172.21.15.104:0/1897869225 >> v1:172.21.15.171:6789/0 conn(0x7f6e841416e0 legacy=0x7f6e8406b5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T14:08:52.342 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:52.338+0000 7f6e898f3640 1 -- 172.21.15.104:0/1897869225 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6e8406e1f0 con 0x7f6e8406de10 2024-03-28T14:08:52.344 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:52.338+0000 7f6e898f3640 1 --2- 172.21.15.104:0/1897869225 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6e8406de10 0x7f6e8414b6f0 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f6e7c009870 tx=0x7f6e7c01c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bf01072c3cfaa97a server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:08:52.344 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:52.338+0000 7f6e888f1640 1 -- 172.21.15.104:0/1897869225 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e7c0ab5f0 con 0x7f6e8406de10 2024-03-28T14:08:52.344 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:52.338+0000 7f6e888f1640 1 -- 172.21.15.104:0/1897869225 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6e7c0b0070 con 0x7f6e8406de10 2024-03-28T14:08:52.344 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:52.338+0000 7f6e888f1640 1 -- 172.21.15.104:0/1897869225 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e7c0ab9b0 con 0x7f6e8406de10 2024-03-28T14:08:52.345 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:52.342+0000 7f6e8bb57640 1 -- 172.21.15.104:0/1897869225 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6e8406de10 msgr2=0x7f6e8414b6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:52.345 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:52.342+0000 7f6e8bb57640 1 --2- 172.21.15.104:0/1897869225 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6e8406de10 0x7f6e8414b6f0 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f6e7c009870 tx=0x7f6e7c01c520 comp rx=0 tx=0).stop 2024-03-28T14:08:52.345 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:52.342+0000 7f6e8bb57640 1 -- 172.21.15.104:0/1897869225 shutdown_connections 2024-03-28T14:08:52.345 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:52.342+0000 7f6e8bb57640 1 --2- 172.21.15.104:0/1897869225 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f6e840b0b40 0x7f6e841411a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:52.345 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:52.342+0000 7f6e8bb57640 1 --2- 172.21.15.104:0/1897869225 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6e8406de10 0x7f6e8414b6f0 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:52.345 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:52.342+0000 7f6e8bb57640 1 -- 172.21.15.104:0/1897869225 >> 172.21.15.104:0/1897869225 conn(0x7f6e840ba400 msgr2=0x7f6e840ba810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T14:08:52.345 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:52.342+0000 7f6e8bb57640 1 -- 172.21.15.104:0/1897869225 shutdown_connections 2024-03-28T14:08:52.345 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:52.342+0000 7f6e8bb57640 1 -- 172.21.15.104:0/1897869225 wait complete. 2024-03-28T14:08:52.346 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:52.342+0000 7f6e8bb57640 1 Processor -- start 2024-03-28T14:08:52.346 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:52.342+0000 7f6e8bb57640 1 -- start start 2024-03-28T14:08:52.350 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:52.342+0000 7f6e8bb57640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f6e8406de10 0x7f6e841e3d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:52.350 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:52.342+0000 7f6e8bb57640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6e840b0b40 0x7f6e8413d970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:52.350 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:52.342+0000 7f6e8bb57640 1 --2- >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f6e841416e0 0x7f6e841e0940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:52.350 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:52.342+0000 7f6e8bb57640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f6e84155090 con 0x7f6e840b0b40 2024-03-28T14:08:52.350 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:52.342+0000 7f6e8bb57640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f6e840b0fb0 con 0x7f6e8406de10 2024-03-28T14:08:52.350 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:52.342+0000 7f6e8bb57640 1 -- --> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] -- mon_getmap magic: 0 -- 0x7f6e84155210 con 0x7f6e841416e0 2024-03-28T14:08:52.350 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:52.342+0000 7f6e890f2640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6e840b0b40 0x7f6e8413d970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:52.350 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:52.342+0000 7f6e890f2640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6e840b0b40 0x7f6e8413d970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:37862/0 (socket says 172.21.15.104:37862) 2024-03-28T14:08:52.350 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:52.342+0000 7f6e890f2640 1 -- 172.21.15.104:0/1469114592 learned_addr learned my addr 172.21.15.104:0/1469114592 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-03-28T14:08:52.350 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:52.346+0000 7f6e898f3640 1 --2- 172.21.15.104:0/1469114592 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f6e8406de10 0x7f6e841e3d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:52.350 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:52.346+0000 7f6e8a0f4640 1 --2- 172.21.15.104:0/1469114592 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f6e841416e0 0x7f6e841e0940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:52.351 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:52.346+0000 7f6e890f2640 1 -- 172.21.15.104:0/1469114592 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f6e841416e0 msgr2=0x7f6e841e0940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:52.351 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:52.346+0000 7f6e890f2640 1 --2- 172.21.15.104:0/1469114592 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f6e841416e0 0x7f6e841e0940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:52.351 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:52.346+0000 7f6e890f2640 1 -- 172.21.15.104:0/1469114592 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f6e8406de10 msgr2=0x7f6e841e3d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:52.351 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:52.346+0000 7f6e890f2640 1 --2- 172.21.15.104:0/1469114592 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f6e8406de10 0x7f6e841e3d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:52.351 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:52.346+0000 7f6e890f2640 1 -- 172.21.15.104:0/1469114592 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6e84070f60 con 0x7f6e840b0b40 2024-03-28T14:08:52.351 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:52.346+0000 7f6e890f2640 1 --2- 172.21.15.104:0/1469114592 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6e840b0b40 0x7f6e8413d970 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f6e7400b580 tx=0x7f6e7400ba40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:08:52.351 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:52.346+0000 7f6e7a7fc640 1 -- 172.21.15.104:0/1469114592 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e74014070 con 0x7f6e840b0b40 2024-03-28T14:08:52.351 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:52.346+0000 7f6e7a7fc640 1 -- 172.21.15.104:0/1469114592 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6e74010890 con 0x7f6e840b0b40 2024-03-28T14:08:52.351 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:52.346+0000 7f6e8bb57640 1 -- 172.21.15.104:0/1469114592 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6e841e4420 con 0x7f6e840b0b40 2024-03-28T14:08:52.351 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:52.346+0000 7f6e7a7fc640 1 -- 172.21.15.104:0/1469114592 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e74010c80 con 0x7f6e840b0b40 2024-03-28T14:08:52.351 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:52.346+0000 7f6e8bb57640 1 -- 172.21.15.104:0/1469114592 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6e8413aa00 con 0x7f6e840b0b40 2024-03-28T14:08:52.351 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:52.346+0000 7f6e8bb57640 1 -- 172.21.15.104:0/1469114592 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6e840687c0 con 0x7f6e840b0b40 2024-03-28T14:08:52.351 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:52.346+0000 7f6e7a7fc640 1 -- 172.21.15.104:0/1469114592 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) ==== 87803+0+0 (secure 0 0 0) 0x7f6e74017070 con 0x7f6e840b0b40 2024-03-28T14:08:52.352 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:52.346+0000 7f6e7a7fc640 1 --2- 172.21.15.104:0/1469114592 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f6e68067fc0 0x7f6e6806a480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:52.352 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:52.346+0000 7f6e7a7fc640 1 -- 172.21.15.104:0/1469114592 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 5709+0+0 (secure 0 0 0) 0x7f6e74086300 con 0x7f6e840b0b40 2024-03-28T14:08:52.353 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:52.346+0000 7f6e898f3640 1 --2- 172.21.15.104:0/1469114592 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f6e68067fc0 0x7f6e6806a480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:52.353 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:52.346+0000 7f6e898f3640 1 --2- 172.21.15.104:0/1469114592 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f6e68067fc0 0x7f6e6806a480 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f6e7c0af000 tx=0x7f6e7c0a7000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:08:52.354 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:52.350+0000 7f6e7a7fc640 1 -- 172.21.15.104:0/1469114592 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179117 (secure 0 0 0) 0x7f6e74056ac0 con 0x7f6e840b0b40 2024-03-28T14:08:52.637 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:52.634+0000 7f6e8bb57640 1 -- 172.21.15.104:0/1469114592 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f6e840b07e0 con 0x7f6e840b0b40 2024-03-28T14:08:52.637 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:52.634+0000 7f6e7a7fc640 1 -- 172.21.15.104:0/1469114592 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v30) ==== 104+0+11 (secure 0 0 0) 0x7f6e74010e20 con 0x7f6e840b0b40 2024-03-28T14:08:52.638 INFO:teuthology.orchestra.run.smithi104.stdout:pg_num: 32 2024-03-28T14:08:52.640 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:52.638+0000 7f6e8bb57640 1 -- 172.21.15.104:0/1469114592 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f6e68067fc0 msgr2=0x7f6e6806a480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:52.640 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:52.638+0000 7f6e8bb57640 1 --2- 172.21.15.104:0/1469114592 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f6e68067fc0 0x7f6e6806a480 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f6e7c0af000 tx=0x7f6e7c0a7000 comp rx=0 tx=0).stop 2024-03-28T14:08:52.640 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:52.638+0000 7f6e8bb57640 1 -- 172.21.15.104:0/1469114592 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6e840b0b40 msgr2=0x7f6e8413d970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:52.640 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:52.638+0000 7f6e8bb57640 1 --2- 172.21.15.104:0/1469114592 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6e840b0b40 0x7f6e8413d970 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f6e7400b580 tx=0x7f6e7400ba40 comp rx=0 tx=0).stop 2024-03-28T14:08:52.640 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:52.638+0000 7f6e8bb57640 1 -- 172.21.15.104:0/1469114592 shutdown_connections 2024-03-28T14:08:52.641 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:52.638+0000 7f6e8bb57640 1 --2- 172.21.15.104:0/1469114592 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f6e841416e0 0x7f6e841e0940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:52.641 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:52.638+0000 7f6e8bb57640 1 --2- 172.21.15.104:0/1469114592 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f6e68067fc0 0x7f6e6806a480 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:52.641 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:52.638+0000 7f6e8bb57640 1 --2- 172.21.15.104:0/1469114592 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6e840b0b40 0x7f6e8413d970 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:52.641 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:52.638+0000 7f6e8bb57640 1 --2- 172.21.15.104:0/1469114592 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f6e8406de10 0x7f6e841e3d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:52.641 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:52.638+0000 7f6e8bb57640 1 -- 172.21.15.104:0/1469114592 >> 172.21.15.104:0/1469114592 conn(0x7f6e840ba400 msgr2=0x7f6e840b71c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T14:08:52.641 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:52.638+0000 7f6e8bb57640 1 -- 172.21.15.104:0/1469114592 shutdown_connections 2024-03-28T14:08:52.641 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:52.638+0000 7f6e8bb57640 1 -- 172.21.15.104:0/1469114592 wait complete. 2024-03-28T14:08:52.656 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-28T14:08:52.804 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:52.798+0000 7f08c8883640 1 Processor -- start 2024-03-28T14:08:52.804 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:52.802+0000 7f08c8883640 1 -- start start 2024-03-28T14:08:52.806 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:52.802+0000 7f08c8883640 1 --2- >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f08c406c8f0 0x7f08c40b2740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:52.807 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:52.802+0000 7f08c8883640 1 --2- >> v2:172.21.15.171:3300/0 conn(0x7f08c40b3730 0x7f08c4148c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:52.807 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:52.802+0000 7f08c8883640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f08c40b3b30 con 0x7f08c41491b0 2024-03-28T14:08:52.807 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:52.802+0000 7f08c8883640 1 -- --> v2:172.21.15.171:3300/0 -- mon_getmap magic: 0 -- 0x7f08c40b0b30 con 0x7f08c40b3730 2024-03-28T14:08:52.807 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:52.802+0000 7f08c8883640 1 -- --> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] -- mon_getmap magic: 0 -- 0x7f08c40b22b0 con 0x7f08c406c8f0 2024-03-28T14:08:52.807 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:52.802+0000 7f08c2d9d640 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f08c41491b0 0x7f08c4147b80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:54772/0 (socket says 172.21.15.104:54772) 2024-03-28T14:08:52.807 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:52.802+0000 7f08c2d9d640 1 -- 172.21.15.104:0/1063593056 learned_addr learned my addr 172.21.15.104:0/1063593056 (peer_addr_for_me v1:172.21.15.104:0/0) 2024-03-28T14:08:52.807 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:52.802+0000 7f08c159a640 1 -- 172.21.15.104:0/1063593056 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3682838646 0 0) 0x7f08c40b3b30 con 0x7f08c41491b0 2024-03-28T14:08:52.807 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:52.802+0000 7f08c259c640 1 --2- 172.21.15.104:0/1063593056 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f08c406c8f0 0x7f08c40b2740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:52.807 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:52.802+0000 7f08c1d9b640 1 --2- 172.21.15.104:0/1063593056 >> v2:172.21.15.171:3300/0 conn(0x7f08c40b3730 0x7f08c4148c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:52.807 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:52.802+0000 7f08c159a640 1 -- 172.21.15.104:0/1063593056 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f08b0003530 con 0x7f08c41491b0 2024-03-28T14:08:52.807 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:52.802+0000 7f08c159a640 1 -- 172.21.15.104:0/1063593056 <== mon.0 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 547723652 0 0) 0x7f08b0003530 con 0x7f08c41491b0 2024-03-28T14:08:52.807 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:52.802+0000 7f08c159a640 1 -- 172.21.15.104:0/1063593056 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f08c406c8f0 msgr2=0x7f08c40b2740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T14:08:52.807 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:52.802+0000 7f08c159a640 1 --2- 172.21.15.104:0/1063593056 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f08c406c8f0 0x7f08c40b2740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:52.808 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:52.802+0000 7f08c159a640 1 -- 172.21.15.104:0/1063593056 >> v2:172.21.15.171:3300/0 conn(0x7f08c40b3730 msgr2=0x7f08c4148c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T14:08:52.808 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:52.802+0000 7f08c159a640 1 --2- 172.21.15.104:0/1063593056 >> v2:172.21.15.171:3300/0 conn(0x7f08c40b3730 0x7f08c4148c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:52.808 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:52.802+0000 7f08c159a640 1 -- 172.21.15.104:0/1063593056 --> v1:172.21.15.104:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f08c406ccd0 con 0x7f08c41491b0 2024-03-28T14:08:52.808 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:52.802+0000 7f08c159a640 1 -- 172.21.15.104:0/1063593056 <== mon.0 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3745580849 0 0) 0x7f08bc001a00 con 0x7f08c41491b0 2024-03-28T14:08:52.808 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:52.802+0000 7f08c1d9b640 1 --2- 172.21.15.104:0/1063593056 >> v2:172.21.15.171:3300/0 conn(0x7f08c40b3730 0x7f08c4148c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T14:08:52.808 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:52.802+0000 7f08c159a640 1 -- 172.21.15.104:0/1063593056 >> v1:172.21.15.104:6789/0 conn(0x7f08c41491b0 legacy=0x7f08c4147b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:52.808 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:52.802+0000 7f08c159a640 1 --2- 172.21.15.104:0/1063593056 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f08c406c8f0 0x7f08b00039c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:52.808 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:52.802+0000 7f08c159a640 1 --2- 172.21.15.104:0/1063593056 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f08b0003f40 0x7f08b0004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:52.808 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:52.802+0000 7f08c159a640 1 --2- 172.21.15.104:0/1063593056 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f08b0004d50 0x7f08b00051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:52.808 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:52.802+0000 7f08c159a640 1 -- 172.21.15.104:0/1063593056 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f08c40b0b30 con 0x7f08b0003f40 2024-03-28T14:08:52.808 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:52.802+0000 7f08c159a640 1 -- 172.21.15.104:0/1063593056 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f08c40b22b0 con 0x7f08b0004d50 2024-03-28T14:08:52.808 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:52.802+0000 7f08c159a640 1 -- 172.21.15.104:0/1063593056 --> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] -- mon_getmap magic: 0 -- 0x7f08b000bad0 con 0x7f08c406c8f0 2024-03-28T14:08:52.808 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:52.802+0000 7f08c1d9b640 1 --2- 172.21.15.104:0/1063593056 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f08b0003f40 0x7f08b0004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:52.808 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:52.802+0000 7f08c2d9d640 1 --2- 172.21.15.104:0/1063593056 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f08b0004d50 0x7f08b00051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:52.809 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:52.802+0000 7f08c1d9b640 1 -- 172.21.15.104:0/1063593056 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f08c406c8f0 msgr2=0x7f08b00039c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T14:08:52.809 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:52.802+0000 7f08c1d9b640 1 --2- 172.21.15.104:0/1063593056 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f08c406c8f0 0x7f08b00039c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:52.809 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:52.802+0000 7f08c1d9b640 1 -- 172.21.15.104:0/1063593056 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f08b0004d50 msgr2=0x7f08b00051d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T14:08:52.809 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:52.802+0000 7f08c1d9b640 1 --2- 172.21.15.104:0/1063593056 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f08b0004d50 0x7f08b00051d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:52.809 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:52.802+0000 7f08c1d9b640 1 -- 172.21.15.104:0/1063593056 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f08b0003530 con 0x7f08b0003f40 2024-03-28T14:08:52.809 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:52.802+0000 7f08c2d9d640 1 --2- 172.21.15.104:0/1063593056 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f08b0004d50 0x7f08b00051d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T14:08:52.810 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:52.806+0000 7f08c1d9b640 1 --2- 172.21.15.104:0/1063593056 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f08b0003f40 0x7f08b0004810 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f08ac009900 tx=0x7f08ac01c600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7a27c52aa965c4fa server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:08:52.810 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:52.806+0000 7f08c159a640 1 -- 172.21.15.104:0/1063593056 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08ac00f040 con 0x7f08b0003f40 2024-03-28T14:08:52.810 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:52.806+0000 7f08c159a640 1 -- 172.21.15.104:0/1063593056 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f08ac0a9400 con 0x7f08b0003f40 2024-03-28T14:08:52.810 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:52.806+0000 7f08c159a640 1 -- 172.21.15.104:0/1063593056 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08ac0a97b0 con 0x7f08b0003f40 2024-03-28T14:08:52.810 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:52.806+0000 7f08c8883640 1 -- 172.21.15.104:0/1063593056 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f08b0003f40 msgr2=0x7f08b0004810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:52.810 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:52.806+0000 7f08c8883640 1 --2- 172.21.15.104:0/1063593056 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f08b0003f40 0x7f08b0004810 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f08ac009900 tx=0x7f08ac01c600 comp rx=0 tx=0).stop 2024-03-28T14:08:52.811 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:52.806+0000 7f08c2d9d640 1 -- 172.21.15.104:0/1063593056 reap_dead start 2024-03-28T14:08:52.811 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:52.806+0000 7f08c8883640 1 -- 172.21.15.104:0/1063593056 shutdown_connections 2024-03-28T14:08:52.811 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:52.806+0000 7f08c8883640 1 -- 172.21.15.104:0/1063593056 >> 172.21.15.104:0/1063593056 conn(0x7f08c40b8f00 msgr2=0x7f08c40b9310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T14:08:52.811 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:52.806+0000 7f08c8883640 1 -- 172.21.15.104:0/1063593056 shutdown_connections 2024-03-28T14:08:52.811 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:52.806+0000 7f08c8883640 1 -- 172.21.15.104:0/1063593056 wait complete. 2024-03-28T14:08:52.813 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:52.806+0000 7f08c8883640 1 Processor -- start 2024-03-28T14:08:52.813 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:52.806+0000 7f08c8883640 1 -- start start 2024-03-28T14:08:52.814 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:52.806+0000 7f08c8883640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f08b0003f40 0x7f08c409c4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:52.814 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:52.806+0000 7f08c8883640 1 --2- >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f08c4140430 0x7f08c41408b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:52.814 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:52.806+0000 7f08c8883640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f08c41e1e90 0x7f08c41e2310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:52.814 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:52.806+0000 7f08c8883640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f08c40b14e0 con 0x7f08b0003f40 2024-03-28T14:08:52.814 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:52.806+0000 7f08c8883640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f08c41499d0 con 0x7f08c41e1e90 2024-03-28T14:08:52.814 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:52.806+0000 7f08c8883640 1 -- --> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] -- mon_getmap magic: 0 -- 0x7f08c40b1660 con 0x7f08c4140430 2024-03-28T14:08:52.814 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:52.806+0000 7f08c259c640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f08b0003f40 0x7f08c409c4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:52.814 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:52.806+0000 7f08c259c640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f08b0003f40 0x7f08c409c4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:37884/0 (socket says 172.21.15.104:37884) 2024-03-28T14:08:52.814 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:52.806+0000 7f08c259c640 1 -- 172.21.15.104:0/1017999090 learned_addr learned my addr 172.21.15.104:0/1017999090 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-03-28T14:08:52.814 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:52.806+0000 7f08c1d9b640 1 --2- 172.21.15.104:0/1017999090 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f08c4140430 0x7f08c41408b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:52.814 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:52.806+0000 7f08c2d9d640 1 --2- 172.21.15.104:0/1017999090 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f08c41e1e90 0x7f08c41e2310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:52.814 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:52.806+0000 7f08c259c640 1 -- 172.21.15.104:0/1017999090 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f08c4140430 msgr2=0x7f08c41408b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:52.814 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:52.806+0000 7f08c259c640 1 --2- 172.21.15.104:0/1017999090 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f08c4140430 0x7f08c41408b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:52.815 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:52.806+0000 7f08c259c640 1 -- 172.21.15.104:0/1017999090 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f08c41e1e90 msgr2=0x7f08c41e2310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:52.815 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:52.806+0000 7f08c259c640 1 --2- 172.21.15.104:0/1017999090 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f08c41e1e90 0x7f08c41e2310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:52.815 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:52.806+0000 7f08c259c640 1 -- 172.21.15.104:0/1017999090 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f08c41d1c30 con 0x7f08b0003f40 2024-03-28T14:08:52.815 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:52.806+0000 7f08c1d9b640 1 --2- 172.21.15.104:0/1017999090 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f08c4140430 0x7f08c41408b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T14:08:52.815 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:52.810+0000 7f08c259c640 1 --2- 172.21.15.104:0/1017999090 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f08b0003f40 0x7f08c409c4b0 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f08b400e820 tx=0x7f08b400ece0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:08:52.815 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:52.810+0000 7f08baffd640 1 -- 172.21.15.104:0/1017999090 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08b4017070 con 0x7f08b0003f40 2024-03-28T14:08:52.815 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:52.810+0000 7f08baffd640 1 -- 172.21.15.104:0/1017999090 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f08b400c8e0 con 0x7f08b0003f40 2024-03-28T14:08:52.815 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:52.810+0000 7f08c8883640 1 -- 172.21.15.104:0/1017999090 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f08c41449c0 con 0x7f08b0003f40 2024-03-28T14:08:52.815 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:52.810+0000 7f08baffd640 1 -- 172.21.15.104:0/1017999090 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08b400ccd0 con 0x7f08b0003f40 2024-03-28T14:08:52.815 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:52.810+0000 7f08c8883640 1 -- 172.21.15.104:0/1017999090 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f08c409d090 con 0x7f08b0003f40 2024-03-28T14:08:52.815 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:52.810+0000 7f08c8883640 1 -- 172.21.15.104:0/1017999090 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f08c40687c0 con 0x7f08b0003f40 2024-03-28T14:08:52.815 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:52.810+0000 7f08baffd640 1 -- 172.21.15.104:0/1017999090 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) ==== 87803+0+0 (secure 0 0 0) 0x7f08b401a070 con 0x7f08b0003f40 2024-03-28T14:08:52.815 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:52.810+0000 7f08baffd640 1 --2- 172.21.15.104:0/1017999090 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f08a4067ff0 0x7f08a406a4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:52.816 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:52.810+0000 7f08baffd640 1 -- 172.21.15.104:0/1017999090 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 5709+0+0 (secure 0 0 0) 0x7f08b40865a0 con 0x7f08b0003f40 2024-03-28T14:08:52.816 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:52.810+0000 7f08c1d9b640 1 --2- 172.21.15.104:0/1017999090 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f08a4067ff0 0x7f08a406a4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:52.816 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:52.810+0000 7f08c1d9b640 1 --2- 172.21.15.104:0/1017999090 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f08a4067ff0 0x7f08a406a4b0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f08ac0ae000 tx=0x7f08ac0b3040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:08:52.818 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:52.814+0000 7f08baffd640 1 -- 172.21.15.104:0/1017999090 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179117 (secure 0 0 0) 0x7f08b4056dc0 con 0x7f08b0003f40 2024-03-28T14:08:52.968 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:52.966+0000 7f08c8883640 1 -- 172.21.15.104:0/1017999090 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f08c4141990 con 0x7f08b0003f40 2024-03-28T14:08:52.968 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:52.966+0000 7f08baffd640 1 -- 172.21.15.104:0/1017999090 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v30) ==== 100+0+11 (secure 0 0 0) 0x7f08b40566c0 con 0x7f08b0003f40 2024-03-28T14:08:52.969 INFO:teuthology.orchestra.run.smithi104.stdout:pg_num: 64 2024-03-28T14:08:52.971 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:52.966+0000 7f08c8883640 1 -- 172.21.15.104:0/1017999090 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f08a4067ff0 msgr2=0x7f08a406a4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:52.972 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:52.966+0000 7f08c8883640 1 --2- 172.21.15.104:0/1017999090 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f08a4067ff0 0x7f08a406a4b0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f08ac0ae000 tx=0x7f08ac0b3040 comp rx=0 tx=0).stop 2024-03-28T14:08:52.972 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:52.966+0000 7f08c8883640 1 -- 172.21.15.104:0/1017999090 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f08b0003f40 msgr2=0x7f08c409c4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:52.972 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:52.966+0000 7f08c8883640 1 --2- 172.21.15.104:0/1017999090 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f08b0003f40 0x7f08c409c4b0 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f08b400e820 tx=0x7f08b400ece0 comp rx=0 tx=0).stop 2024-03-28T14:08:52.972 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:52.966+0000 7f08c8883640 1 -- 172.21.15.104:0/1017999090 shutdown_connections 2024-03-28T14:08:52.972 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:52.966+0000 7f08c8883640 1 --2- 172.21.15.104:0/1017999090 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f08c41e1e90 0x7f08c41e2310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:52.972 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:52.966+0000 7f08c8883640 1 --2- 172.21.15.104:0/1017999090 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f08c4140430 0x7f08c41408b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:52.972 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:52.966+0000 7f08c8883640 1 --2- 172.21.15.104:0/1017999090 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f08a4067ff0 0x7f08a406a4b0 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:52.972 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:52.966+0000 7f08c8883640 1 --2- 172.21.15.104:0/1017999090 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f08b0003f40 0x7f08c409c4b0 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:52.972 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:52.966+0000 7f08c8883640 1 -- 172.21.15.104:0/1017999090 >> 172.21.15.104:0/1017999090 conn(0x7f08c40b8f00 msgr2=0x7f08c40b4bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T14:08:52.972 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:52.966+0000 7f08c8883640 1 -- 172.21.15.104:0/1017999090 shutdown_connections 2024-03-28T14:08:52.973 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:52.966+0000 7f08c8883640 1 -- 172.21.15.104:0/1017999090 wait complete. 2024-03-28T14:08:52.987 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T14:08:53.104 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:53.098+0000 7fc0e2651640 1 Processor -- start 2024-03-28T14:08:53.104 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:53.102+0000 7fc0e2651640 1 -- start start 2024-03-28T14:08:53.106 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:53.102+0000 7fc0e2651640 1 --2- >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fc0dc02c460 0x7fc0dc0b3ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:53.106 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:53.102+0000 7fc0e2651640 1 --2- >> v2:172.21.15.171:3300/0 conn(0x7fc0dc13fc50 0x7fc0dc1456c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:53.106 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:53.102+0000 7fc0e2651640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc0dc02c840 con 0x7fc0dc0b4530 2024-03-28T14:08:53.106 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:53.102+0000 7fc0e2651640 1 -- --> v2:172.21.15.171:3300/0 -- mon_getmap magic: 0 -- 0x7fc0dc0b4f20 con 0x7fc0dc13fc50 2024-03-28T14:08:53.107 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:53.102+0000 7fc0e2651640 1 -- --> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] -- mon_getmap magic: 0 -- 0x7fc0dc0b5200 con 0x7fc0dc02c460 2024-03-28T14:08:53.107 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:53.102+0000 7fc0db7fe640 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7fc0dc0b4530 0x7fc0dc13f540 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:54774/0 (socket says 172.21.15.104:54774) 2024-03-28T14:08:53.107 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:53.102+0000 7fc0db7fe640 1 -- 172.21.15.104:0/1156972953 learned_addr learned my addr 172.21.15.104:0/1156972953 (peer_addr_for_me v1:172.21.15.104:0/0) 2024-03-28T14:08:53.107 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:53.102+0000 7fc0e0bee640 1 --2- 172.21.15.104:0/1156972953 >> v2:172.21.15.171:3300/0 conn(0x7fc0dc13fc50 0x7fc0dc1456c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:53.107 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:53.102+0000 7fc0dbfff640 1 --2- 172.21.15.104:0/1156972953 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fc0dc02c460 0x7fc0dc0b3ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:53.107 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:53.102+0000 7fc0daffd640 1 -- 172.21.15.104:0/1156972953 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1949654867 0 0) 0x7fc0dc02c840 con 0x7fc0dc0b4530 2024-03-28T14:08:53.107 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:53.102+0000 7fc0daffd640 1 -- 172.21.15.104:0/1156972953 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc0c4003530 con 0x7fc0dc0b4530 2024-03-28T14:08:53.107 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:53.102+0000 7fc0daffd640 1 -- 172.21.15.104:0/1156972953 <== mon.0 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1710033845 0 0) 0x7fc0c4003530 con 0x7fc0dc0b4530 2024-03-28T14:08:53.107 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:53.102+0000 7fc0daffd640 1 -- 172.21.15.104:0/1156972953 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fc0dc02c460 msgr2=0x7fc0dc0b3ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T14:08:53.107 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:53.102+0000 7fc0daffd640 1 --2- 172.21.15.104:0/1156972953 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fc0dc02c460 0x7fc0dc0b3ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:53.107 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:53.102+0000 7fc0daffd640 1 -- 172.21.15.104:0/1156972953 >> v2:172.21.15.171:3300/0 conn(0x7fc0dc13fc50 msgr2=0x7fc0dc1456c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T14:08:53.107 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:53.102+0000 7fc0daffd640 1 --2- 172.21.15.104:0/1156972953 >> v2:172.21.15.171:3300/0 conn(0x7fc0dc13fc50 0x7fc0dc1456c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:53.107 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:53.102+0000 7fc0daffd640 1 -- 172.21.15.104:0/1156972953 --> v1:172.21.15.104:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc0dc0b54e0 con 0x7fc0dc0b4530 2024-03-28T14:08:53.107 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:53.102+0000 7fc0daffd640 1 -- 172.21.15.104:0/1156972953 <== mon.0 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3745580849 0 0) 0x7fc0d0001680 con 0x7fc0dc0b4530 2024-03-28T14:08:53.108 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:53.102+0000 7fc0daffd640 1 -- 172.21.15.104:0/1156972953 >> v1:172.21.15.104:6789/0 conn(0x7fc0dc0b4530 legacy=0x7fc0dc13f540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:53.108 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:53.102+0000 7fc0daffd640 1 --2- 172.21.15.104:0/1156972953 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fc0c40039c0 0x7fc0c4003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:53.108 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:53.102+0000 7fc0daffd640 1 --2- 172.21.15.104:0/1156972953 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fc0dc02c460 0x7fc0c4004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:53.108 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:53.102+0000 7fc0daffd640 1 --2- 172.21.15.104:0/1156972953 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc0c4004d50 0x7fc0c40051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:53.108 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:53.102+0000 7fc0daffd640 1 -- 172.21.15.104:0/1156972953 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7fc0dc0b4f20 con 0x7fc0c4004d50 2024-03-28T14:08:53.108 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:53.102+0000 7fc0daffd640 1 -- 172.21.15.104:0/1156972953 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7fc0dc0b5200 con 0x7fc0c40039c0 2024-03-28T14:08:53.108 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:53.102+0000 7fc0daffd640 1 -- 172.21.15.104:0/1156972953 --> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] -- mon_getmap magic: 0 -- 0x7fc0c400b9d0 con 0x7fc0dc02c460 2024-03-28T14:08:53.108 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:53.102+0000 7fc0e0bee640 1 --2- 172.21.15.104:0/1156972953 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc0c4004d50 0x7fc0c40051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:53.108 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:53.102+0000 7fc0db7fe640 1 --2- 172.21.15.104:0/1156972953 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fc0dc02c460 0x7fc0c4004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:53.108 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:53.102+0000 7fc0dbfff640 1 --2- 172.21.15.104:0/1156972953 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fc0c40039c0 0x7fc0c4003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:53.108 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:53.102+0000 7fc0e0bee640 1 -- 172.21.15.104:0/1156972953 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fc0dc02c460 msgr2=0x7fc0c4004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T14:08:53.108 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:53.102+0000 7fc0e0bee640 1 --2- 172.21.15.104:0/1156972953 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fc0dc02c460 0x7fc0c4004810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:53.108 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:53.102+0000 7fc0e0bee640 1 -- 172.21.15.104:0/1156972953 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fc0c40039c0 msgr2=0x7fc0c4003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T14:08:53.109 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:53.102+0000 7fc0e0bee640 1 --2- 172.21.15.104:0/1156972953 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fc0c40039c0 0x7fc0c4003de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:53.109 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:53.102+0000 7fc0e0bee640 1 -- 172.21.15.104:0/1156972953 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc0c4003530 con 0x7fc0c4004d50 2024-03-28T14:08:53.110 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:53.106+0000 7fc0e0bee640 1 --2- 172.21.15.104:0/1156972953 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc0c4004d50 0x7fc0c40051d0 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7fc0dc09c460 tx=0x7fc0d401ca50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b12c2c2bd1620ba5 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:08:53.111 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:53.106+0000 7fc0daffd640 1 -- 172.21.15.104:0/1156972953 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc0d40b2070 con 0x7fc0c4004d50 2024-03-28T14:08:53.111 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:53.106+0000 7fc0daffd640 1 -- 172.21.15.104:0/1156972953 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc0d40a5530 con 0x7fc0c4004d50 2024-03-28T14:08:53.111 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:53.106+0000 7fc0daffd640 1 -- 172.21.15.104:0/1156972953 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc0d40096b0 con 0x7fc0c4004d50 2024-03-28T14:08:53.111 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:53.106+0000 7fc0e2651640 1 -- 172.21.15.104:0/1156972953 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc0c4004d50 msgr2=0x7fc0c40051d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:53.111 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:53.106+0000 7fc0e2651640 1 --2- 172.21.15.104:0/1156972953 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc0c4004d50 0x7fc0c40051d0 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7fc0dc09c460 tx=0x7fc0d401ca50 comp rx=0 tx=0).stop 2024-03-28T14:08:53.111 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:53.106+0000 7fc0e0bee640 1 -- 172.21.15.104:0/1156972953 reap_dead start 2024-03-28T14:08:53.111 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:53.106+0000 7fc0e2651640 1 -- 172.21.15.104:0/1156972953 shutdown_connections 2024-03-28T14:08:53.111 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:53.106+0000 7fc0e2651640 1 -- 172.21.15.104:0/1156972953 >> 172.21.15.104:0/1156972953 conn(0x7fc0dc0b95f0 msgr2=0x7fc0dc0b99d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T14:08:53.111 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:53.106+0000 7fc0e2651640 1 -- 172.21.15.104:0/1156972953 shutdown_connections 2024-03-28T14:08:53.111 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:53.106+0000 7fc0e2651640 1 -- 172.21.15.104:0/1156972953 wait complete. 2024-03-28T14:08:53.112 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:53.110+0000 7fc0e2651640 1 Processor -- start 2024-03-28T14:08:53.112 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:53.110+0000 7fc0e2651640 1 -- start start 2024-03-28T14:08:53.114 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:53.110+0000 7fc0e2651640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc0dc0af3a0 0x7fc0dc1e0790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:53.114 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:53.110+0000 7fc0e2651640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fc0dc146820 0x7fc0dc1e0cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:53.114 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:53.110+0000 7fc0e2651640 1 --2- >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fc0dc1e1210 0x7fc0dc0b1a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:53.114 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:53.110+0000 7fc0e2651640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7fc0dc0b4170 con 0x7fc0dc0af3a0 2024-03-28T14:08:53.114 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:53.110+0000 7fc0e2651640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7fc0dc0b3ff0 con 0x7fc0dc146820 2024-03-28T14:08:53.114 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:53.110+0000 7fc0e2651640 1 -- --> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] -- mon_getmap magic: 0 -- 0x7fc0dc0b42f0 con 0x7fc0dc1e1210 2024-03-28T14:08:53.114 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:53.110+0000 7fc0dbfff640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc0dc0af3a0 0x7fc0dc1e0790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:53.114 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:53.110+0000 7fc0dbfff640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc0dc0af3a0 0x7fc0dc1e0790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:37912/0 (socket says 172.21.15.104:37912) 2024-03-28T14:08:53.114 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:53.110+0000 7fc0dbfff640 1 -- 172.21.15.104:0/738797786 learned_addr learned my addr 172.21.15.104:0/738797786 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-03-28T14:08:53.115 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:53.110+0000 7fc0db7fe640 1 --2- 172.21.15.104:0/738797786 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fc0dc146820 0x7fc0dc1e0cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:53.115 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:53.110+0000 7fc0dbfff640 1 -- 172.21.15.104:0/738797786 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fc0dc1e1210 msgr2=0x7fc0dc0b1a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:53.115 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:53.110+0000 7fc0e0bee640 1 --2- 172.21.15.104:0/738797786 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fc0dc1e1210 0x7fc0dc0b1a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:53.115 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:53.110+0000 7fc0dbfff640 1 --2- 172.21.15.104:0/738797786 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fc0dc1e1210 0x7fc0dc0b1a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:53.115 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:53.110+0000 7fc0dbfff640 1 -- 172.21.15.104:0/738797786 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fc0dc146820 msgr2=0x7fc0dc1e0cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:53.115 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:53.110+0000 7fc0dbfff640 1 --2- 172.21.15.104:0/738797786 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fc0dc146820 0x7fc0dc1e0cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:53.115 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:53.110+0000 7fc0dbfff640 1 -- 172.21.15.104:0/738797786 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc0dc0b3880 con 0x7fc0dc0af3a0 2024-03-28T14:08:53.115 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:53.110+0000 7fc0db7fe640 1 --2- 172.21.15.104:0/738797786 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fc0dc146820 0x7fc0dc1e0cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T14:08:53.115 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:53.110+0000 7fc0dbfff640 1 --2- 172.21.15.104:0/738797786 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc0dc0af3a0 0x7fc0dc1e0790 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7fc0cc007f60 tx=0x7fc0cc007f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:08:53.115 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:53.110+0000 7fc0d8ff9640 1 -- 172.21.15.104:0/738797786 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc0cc01a020 con 0x7fc0dc0af3a0 2024-03-28T14:08:53.115 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:53.110+0000 7fc0d8ff9640 1 -- 172.21.15.104:0/738797786 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc0cc016470 con 0x7fc0dc0af3a0 2024-03-28T14:08:53.115 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:53.110+0000 7fc0e2651640 1 -- 172.21.15.104:0/738797786 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc0dc143410 con 0x7fc0dc0af3a0 2024-03-28T14:08:53.115 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:53.110+0000 7fc0d8ff9640 1 -- 172.21.15.104:0/738797786 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc0cc016860 con 0x7fc0dc0af3a0 2024-03-28T14:08:53.115 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:53.110+0000 7fc0e2651640 1 -- 172.21.15.104:0/738797786 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc0dc146f60 con 0x7fc0dc0af3a0 2024-03-28T14:08:53.116 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:53.110+0000 7fc0e2651640 1 -- 172.21.15.104:0/738797786 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc0dc0687c0 con 0x7fc0dc0af3a0 2024-03-28T14:08:53.116 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:53.110+0000 7fc0d8ff9640 1 -- 172.21.15.104:0/738797786 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) ==== 87803+0+0 (secure 0 0 0) 0x7fc0cc01d020 con 0x7fc0dc0af3a0 2024-03-28T14:08:53.116 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:53.110+0000 7fc0d8ff9640 1 --2- 172.21.15.104:0/738797786 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7fc0c8067ff0 0x7fc0c806a4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:53.116 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:53.110+0000 7fc0d8ff9640 1 -- 172.21.15.104:0/738797786 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 5709+0+0 (secure 0 0 0) 0x7fc0cc0872d0 con 0x7fc0dc0af3a0 2024-03-28T14:08:53.117 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:53.110+0000 7fc0db7fe640 1 --2- 172.21.15.104:0/738797786 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7fc0c8067ff0 0x7fc0c806a4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:53.117 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:53.114+0000 7fc0db7fe640 1 --2- 172.21.15.104:0/738797786 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7fc0c8067ff0 0x7fc0c806a4b0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fc0d0000c10 tx=0x7fc0d0001550 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:08:53.119 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:53.114+0000 7fc0d8ff9640 1 -- 172.21.15.104:0/738797786 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179117 (secure 0 0 0) 0x7fc0cc024030 con 0x7fc0dc0af3a0 2024-03-28T14:08:53.308 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:53.306+0000 7fc0e2651640 1 -- 172.21.15.104:0/738797786 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fc0dc0b7c90 con 0x7fc0dc0af3a0 2024-03-28T14:08:53.309 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:53.306+0000 7fc0d8ff9640 1 -- 172.21.15.104:0/738797786 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 14 v14) ==== 94+0+3537 (secure 0 0 0) 0x7fc0cc057b10 con 0x7fc0dc0af3a0 2024-03-28T14:08:53.309 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-03-28T14:08:53.309 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":14,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4550,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.171:6835/3418793776","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6834","nonce":3418793776},{"type":"v1","addr":"172.21.15.171:6835","nonce":3418793776}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11}],"filesystems":[{"mdsmap":{"epoch":14,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T14:08:40.854108+0000","modified":"2024-03-28T14:08:42.859657+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4959},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4959":{"gid":4959,"name":"a","rank":0,"incarnation":12,"state":"up:active","state_seq":3,"addr":"172.21.15.104:6833/1094990707","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":1094990707},{"type":"v1","addr":"172.21.15.104:6833","nonce":1094990707}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4959,"qdb_cluster":[4959]},"id":2}]} 2024-03-28T14:08:53.309 INFO:teuthology.orchestra.run.smithi104.stderr:dumped fsmap epoch 14 2024-03-28T14:08:53.311 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:53.306+0000 7fc0e2651640 1 -- 172.21.15.104:0/738797786 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7fc0c8067ff0 msgr2=0x7fc0c806a4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:53.311 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:53.306+0000 7fc0e2651640 1 --2- 172.21.15.104:0/738797786 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7fc0c8067ff0 0x7fc0c806a4b0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fc0d0000c10 tx=0x7fc0d0001550 comp rx=0 tx=0).stop 2024-03-28T14:08:53.311 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:53.306+0000 7fc0e2651640 1 -- 172.21.15.104:0/738797786 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc0dc0af3a0 msgr2=0x7fc0dc1e0790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:53.311 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:53.306+0000 7fc0e2651640 1 --2- 172.21.15.104:0/738797786 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc0dc0af3a0 0x7fc0dc1e0790 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7fc0cc007f60 tx=0x7fc0cc007f90 comp rx=0 tx=0).stop 2024-03-28T14:08:53.311 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:53.306+0000 7fc0e2651640 1 -- 172.21.15.104:0/738797786 shutdown_connections 2024-03-28T14:08:53.311 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:53.306+0000 7fc0e2651640 1 --2- 172.21.15.104:0/738797786 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fc0dc1e1210 0x7fc0dc0b1a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:53.311 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:53.306+0000 7fc0e2651640 1 --2- 172.21.15.104:0/738797786 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fc0dc146820 0x7fc0dc1e0cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:53.312 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:53.306+0000 7fc0e2651640 1 --2- 172.21.15.104:0/738797786 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7fc0c8067ff0 0x7fc0c806a4b0 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:53.312 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:53.306+0000 7fc0e2651640 1 --2- 172.21.15.104:0/738797786 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc0dc0af3a0 0x7fc0dc1e0790 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:53.312 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:53.306+0000 7fc0e2651640 1 -- 172.21.15.104:0/738797786 >> 172.21.15.104:0/738797786 conn(0x7fc0dc0b95f0 msgr2=0x7fc0dc13cfd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T14:08:53.312 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:53.306+0000 7fc0e2651640 1 -- 172.21.15.104:0/738797786 shutdown_connections 2024-03-28T14:08:53.312 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:53.306+0000 7fc0e2651640 1 -- 172.21.15.104:0/738797786 wait complete. 2024-03-28T14:08:53.332 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-28T14:08:53.443 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:53.438+0000 7fdd3bdca640 1 Processor -- start 2024-03-28T14:08:53.444 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:53.438+0000 7fdd3bdca640 1 -- start start 2024-03-28T14:08:53.445 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:53.442+0000 7fdd3bdca640 1 --2- >> v2:172.21.15.171:3300/0 conn(0x7fdd3406dd10 0x7fdd34072cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:53.445 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:53.442+0000 7fdd3bdca640 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fdd340b5120 0x7fdd34152280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:53.445 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:53.442+0000 7fdd3bdca640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdd34073230 con 0x7fdd340b5600 2024-03-28T14:08:53.445 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:53.442+0000 7fdd3bdca640 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 -- 0x7fdd3406e0f0 con 0x7fdd340b5120 2024-03-28T14:08:53.445 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:53.442+0000 7fdd3bdca640 1 -- --> v2:172.21.15.171:3300/0 -- mon_getmap magic: 0 -- 0x7fdd340b0ee0 con 0x7fdd3406dd10 2024-03-28T14:08:53.445 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:53.442+0000 7fdd3a367640 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fdd340b5120 0x7fdd34152280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:53.446 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:53.442+0000 7fdd3a367640 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fdd340b5120 0x7fdd34152280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:37920/0 (socket says 172.21.15.104:37920) 2024-03-28T14:08:53.446 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:53.442+0000 7fdd3a367640 1 -- 172.21.15.104:0/503495355 learned_addr learned my addr 172.21.15.104:0/503495355 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-03-28T14:08:53.446 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:53.442+0000 7fdd39b66640 1 --2- 172.21.15.104:0/503495355 >> v2:172.21.15.171:3300/0 conn(0x7fdd3406dd10 0x7fdd34072cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:53.446 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:53.442+0000 7fdd3a367640 1 -- 172.21.15.104:0/503495355 >> v2:172.21.15.171:3300/0 conn(0x7fdd3406dd10 msgr2=0x7fdd34072cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T14:08:53.446 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:53.442+0000 7fdd3a367640 1 --2- 172.21.15.104:0/503495355 >> v2:172.21.15.171:3300/0 conn(0x7fdd3406dd10 0x7fdd34072cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:53.446 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:53.442+0000 7fdd3a367640 1 -- 172.21.15.104:0/503495355 >> v1:172.21.15.171:6789/0 conn(0x7fdd340b5600 legacy=0x7fdd341417b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T14:08:53.446 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:53.442+0000 7fdd3a367640 1 -- 172.21.15.104:0/503495355 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdd340b0b70 con 0x7fdd340b5120 2024-03-28T14:08:53.447 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:53.442+0000 7fdd3a367640 1 --2- 172.21.15.104:0/503495355 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fdd340b5120 0x7fdd34152280 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7fdd3406e6e0 tx=0x7fdd2c01c9b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fb0d3862d5b0c945 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:08:53.447 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:53.442+0000 7fdd38b64640 1 -- 172.21.15.104:0/503495355 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd2c0a5670 con 0x7fdd340b5120 2024-03-28T14:08:53.447 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:53.442+0000 7fdd38b64640 1 -- 172.21.15.104:0/503495355 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdd2c0a5810 con 0x7fdd340b5120 2024-03-28T14:08:53.447 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:53.442+0000 7fdd38b64640 1 -- 172.21.15.104:0/503495355 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd2c0a5be0 con 0x7fdd340b5120 2024-03-28T14:08:53.448 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:53.442+0000 7fdd3bdca640 1 -- 172.21.15.104:0/503495355 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fdd340b5120 msgr2=0x7fdd34152280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:53.448 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:53.442+0000 7fdd3bdca640 1 --2- 172.21.15.104:0/503495355 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fdd340b5120 0x7fdd34152280 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7fdd3406e6e0 tx=0x7fdd2c01c9b0 comp rx=0 tx=0).stop 2024-03-28T14:08:53.448 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:53.442+0000 7fdd3bdca640 1 -- 172.21.15.104:0/503495355 shutdown_connections 2024-03-28T14:08:53.449 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:53.442+0000 7fdd3bdca640 1 --2- 172.21.15.104:0/503495355 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fdd340b5120 0x7fdd34152280 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:53.449 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:53.442+0000 7fdd3bdca640 1 --2- 172.21.15.104:0/503495355 >> v2:172.21.15.171:3300/0 conn(0x7fdd3406dd10 0x7fdd34072cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:53.449 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:53.442+0000 7fdd3bdca640 1 -- 172.21.15.104:0/503495355 >> 172.21.15.104:0/503495355 conn(0x7fdd340ba390 msgr2=0x7fdd340ba7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T14:08:53.449 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:53.442+0000 7fdd3bdca640 1 -- 172.21.15.104:0/503495355 shutdown_connections 2024-03-28T14:08:53.449 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:53.442+0000 7fdd3bdca640 1 -- 172.21.15.104:0/503495355 wait complete. 2024-03-28T14:08:53.449 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:53.446+0000 7fdd3bdca640 1 Processor -- start 2024-03-28T14:08:53.449 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:53.446+0000 7fdd3bdca640 1 -- start start 2024-03-28T14:08:53.451 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:53.446+0000 7fdd3bdca640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fdd3406dd10 0x7fdd341e0f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:53.451 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:53.446+0000 7fdd3bdca640 1 --2- >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fdd340b5120 0x7fdd34144180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:53.452 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:53.446+0000 7fdd3bdca640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fdd340b5600 0x7fdd34144cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:53.452 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:53.446+0000 7fdd3bdca640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7fdd34155370 con 0x7fdd340b5600 2024-03-28T14:08:53.452 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:53.446+0000 7fdd3bdca640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7fdd341551f0 con 0x7fdd3406dd10 2024-03-28T14:08:53.452 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:53.446+0000 7fdd3bdca640 1 -- --> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] -- mon_getmap magic: 0 -- 0x7fdd341554f0 con 0x7fdd340b5120 2024-03-28T14:08:53.452 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:53.446+0000 7fdd3a367640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fdd340b5600 0x7fdd34144cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:53.452 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:53.446+0000 7fdd3a367640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fdd340b5600 0x7fdd34144cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:37934/0 (socket says 172.21.15.104:37934) 2024-03-28T14:08:53.452 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:53.446+0000 7fdd3a367640 1 -- 172.21.15.104:0/392715313 learned_addr learned my addr 172.21.15.104:0/392715313 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-03-28T14:08:53.452 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:53.446+0000 7fdd39b66640 1 --2- 172.21.15.104:0/392715313 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fdd3406dd10 0x7fdd341e0f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:53.452 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:53.446+0000 7fdd39365640 1 --2- 172.21.15.104:0/392715313 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fdd340b5120 0x7fdd34144180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:53.453 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:53.446+0000 7fdd3a367640 1 -- 172.21.15.104:0/392715313 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fdd340b5120 msgr2=0x7fdd34144180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:53.453 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:53.446+0000 7fdd3a367640 1 --2- 172.21.15.104:0/392715313 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fdd340b5120 0x7fdd34144180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:53.453 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:53.446+0000 7fdd3a367640 1 -- 172.21.15.104:0/392715313 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fdd3406dd10 msgr2=0x7fdd341e0f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:53.453 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:53.446+0000 7fdd3a367640 1 --2- 172.21.15.104:0/392715313 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fdd3406dd10 0x7fdd341e0f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:53.453 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:53.446+0000 7fdd3a367640 1 -- 172.21.15.104:0/392715313 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdd34142f30 con 0x7fdd340b5600 2024-03-28T14:08:53.453 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:53.446+0000 7fdd3a367640 1 --2- 172.21.15.104:0/392715313 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fdd340b5600 0x7fdd34144cd0 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7fdd2c01a5e0 tx=0x7fdd2c019750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:08:53.453 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:53.446+0000 7fdd2a7fc640 1 -- 172.21.15.104:0/392715313 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd2c0bc020 con 0x7fdd340b5600 2024-03-28T14:08:53.453 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:53.446+0000 7fdd2a7fc640 1 -- 172.21.15.104:0/392715313 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdd2c0b88a0 con 0x7fdd340b5600 2024-03-28T14:08:53.454 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:53.446+0000 7fdd3bdca640 1 -- 172.21.15.104:0/392715313 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdd341e1450 con 0x7fdd340b5600 2024-03-28T14:08:53.454 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:53.446+0000 7fdd2a7fc640 1 -- 172.21.15.104:0/392715313 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd2c009510 con 0x7fdd340b5600 2024-03-28T14:08:53.454 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:53.446+0000 7fdd3bdca640 1 -- 172.21.15.104:0/392715313 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdd341446c0 con 0x7fdd340b5600 2024-03-28T14:08:53.454 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:53.446+0000 7fdd3bdca640 1 -- 172.21.15.104:0/392715313 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdd340687c0 con 0x7fdd340b5600 2024-03-28T14:08:53.454 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:53.446+0000 7fdd2a7fc640 1 -- 172.21.15.104:0/392715313 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) ==== 87803+0+0 (secure 0 0 0) 0x7fdd2c0bf020 con 0x7fdd340b5600 2024-03-28T14:08:53.455 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:53.450+0000 7fdd2a7fc640 1 --2- 172.21.15.104:0/392715313 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7fdd18067ff0 0x7fdd1806a4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:53.455 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:53.450+0000 7fdd2a7fc640 1 -- 172.21.15.104:0/392715313 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 5709+0+0 (secure 0 0 0) 0x7fdd2c121ac0 con 0x7fdd340b5600 2024-03-28T14:08:53.455 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:53.450+0000 7fdd39b66640 1 --2- 172.21.15.104:0/392715313 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7fdd18067ff0 0x7fdd1806a4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:53.455 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:53.450+0000 7fdd39b66640 1 --2- 172.21.15.104:0/392715313 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7fdd18067ff0 0x7fdd1806a4b0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fdd3000b800 tx=0x7fdd30008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:08:53.456 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:53.450+0000 7fdd2a7fc640 1 -- 172.21.15.104:0/392715313 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179117 (secure 0 0 0) 0x7fdd2c0c7020 con 0x7fdd340b5600 2024-03-28T14:08:53.606 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:53.602+0000 7fdd3bdca640 1 -- 172.21.15.104:0/392715313 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fdd340712c0 con 0x7fdd340b5600 2024-03-28T14:08:53.607 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:53.602+0000 7fdd2a7fc640 1 -- 172.21.15.104:0/392715313 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v30) ==== 74+0+16693 (secure 0 0 0) 0x7fdd2c0165a0 con 0x7fdd340b5600 2024-03-28T14:08:53.608 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-03-28T14:08:53.608 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":30,"fsid":"428a90f2-3351-435f-8fcf-f7e66c78a13e","created":"2024-03-28T14:07:34.862458+0000","modified":"2024-03-28T14:08:42.477796+0000","last_up_change":"2024-03-28T14:07:46.028122+0000","last_in_change":"2024-03-28T14:07:37.157486+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-28T14:07:47.622988+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_metadata","create_time":"2024-03-28T14:08:38.687068+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data","create_time":"2024-03-28T14:08:39.805358+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"30","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"1cb70540-4e42-402f-a446-aaec4ca83007","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6800","nonce":3745690918},{"type":"v1","addr":"172.21.15.104:6801","nonce":3745690918}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":3745690918},{"type":"v1","addr":"172.21.15.104:6803","nonce":3745690918}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":3745690918},{"type":"v1","addr":"172.21.15.104:6807","nonce":3745690918}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":3745690918},{"type":"v1","addr":"172.21.15.104:6805","nonce":3745690918}]},"public_addr":"172.21.15.104:6801/3745690918","cluster_addr":"172.21.15.104:6803/3745690918","heartbeat_back_addr":"172.21.15.104:6807/3745690918","heartbeat_front_addr":"172.21.15.104:6805/3745690918","state":["exists","up"]},{"osd":1,"uuid":"3ab7afdd-b980-418c-81f2-004e16de3dca","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":19438674},{"type":"v1","addr":"172.21.15.104:6810","nonce":19438674}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":19438674},{"type":"v1","addr":"172.21.15.104:6814","nonce":19438674}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":19438674},{"type":"v1","addr":"172.21.15.104:6823","nonce":19438674}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":19438674},{"type":"v1","addr":"172.21.15.104:6819","nonce":19438674}]},"public_addr":"172.21.15.104:6810/19438674","cluster_addr":"172.21.15.104:6814/19438674","heartbeat_back_addr":"172.21.15.104:6823/19438674","heartbeat_front_addr":"172.21.15.104:6819/19438674","state":["exists","up"]},{"osd":2,"uuid":"3e2651c8-ecae-4a9e-a053-27240a79eac2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":2352012626},{"type":"v1","addr":"172.21.15.104:6825","nonce":2352012626}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6826","nonce":2352012626},{"type":"v1","addr":"172.21.15.104:6827","nonce":2352012626}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":2352012626},{"type":"v1","addr":"172.21.15.104:6831","nonce":2352012626}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6828","nonce":2352012626},{"type":"v1","addr":"172.21.15.104:6829","nonce":2352012626}]},"public_addr":"172.21.15.104:6825/2352012626","cluster_addr":"172.21.15.104:6827/2352012626","heartbeat_back_addr":"172.21.15.104:6831/2352012626","heartbeat_front_addr":"172.21.15.104:6829/2352012626","state":["exists","up"]},{"osd":3,"uuid":"925c7e0a-38ef-47cd-b288-8bec8456fb17","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6809","nonce":3568347381},{"type":"v1","addr":"172.21.15.104:6811","nonce":3568347381}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6813","nonce":3568347381},{"type":"v1","addr":"172.21.15.104:6815","nonce":3568347381}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":3568347381},{"type":"v1","addr":"172.21.15.104:6822","nonce":3568347381}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6817","nonce":3568347381},{"type":"v1","addr":"172.21.15.104:6818","nonce":3568347381}]},"public_addr":"172.21.15.104:6811/3568347381","cluster_addr":"172.21.15.104:6815/3568347381","heartbeat_back_addr":"172.21.15.104:6822/3568347381","heartbeat_front_addr":"172.21.15.104:6818/3568347381","state":["exists","up"]},{"osd":4,"uuid":"4a8fac86-e595-4a0e-96e3-57edb3cc3f57","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6800","nonce":2107001077},{"type":"v1","addr":"172.21.15.171:6801","nonce":2107001077}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6802","nonce":2107001077},{"type":"v1","addr":"172.21.15.171:6803","nonce":2107001077}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6806","nonce":2107001077},{"type":"v1","addr":"172.21.15.171:6807","nonce":2107001077}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6804","nonce":2107001077},{"type":"v1","addr":"172.21.15.171:6805","nonce":2107001077}]},"public_addr":"172.21.15.171:6801/2107001077","cluster_addr":"172.21.15.171:6803/2107001077","heartbeat_back_addr":"172.21.15.171:6807/2107001077","heartbeat_front_addr":"172.21.15.171:6805/2107001077","state":["exists","up"]},{"osd":5,"uuid":"e48d50b0-beae-40ae-b15a-9a84dbfe88ab","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6808","nonce":2934826925},{"type":"v1","addr":"172.21.15.171:6810","nonce":2934826925}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6812","nonce":2934826925},{"type":"v1","addr":"172.21.15.171:6814","nonce":2934826925}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6819","nonce":2934826925},{"type":"v1","addr":"172.21.15.171:6821","nonce":2934826925}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6816","nonce":2934826925},{"type":"v1","addr":"172.21.15.171:6817","nonce":2934826925}]},"public_addr":"172.21.15.171:6810/2934826925","cluster_addr":"172.21.15.171:6814/2934826925","heartbeat_back_addr":"172.21.15.171:6821/2934826925","heartbeat_front_addr":"172.21.15.171:6817/2934826925","state":["exists","up"]},{"osd":6,"uuid":"5e7c3292-9936-4f35-b267-aed9d4e49323","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6824","nonce":436742919},{"type":"v1","addr":"172.21.15.171:6825","nonce":436742919}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6826","nonce":436742919},{"type":"v1","addr":"172.21.15.171:6827","nonce":436742919}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6830","nonce":436742919},{"type":"v1","addr":"172.21.15.171:6831","nonce":436742919}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6828","nonce":436742919},{"type":"v1","addr":"172.21.15.171:6829","nonce":436742919}]},"public_addr":"172.21.15.171:6825/436742919","cluster_addr":"172.21.15.171:6827/436742919","heartbeat_back_addr":"172.21.15.171:6831/436742919","heartbeat_front_addr":"172.21.15.171:6829/436742919","state":["exists","up"]},{"osd":7,"uuid":"339b8414-d7f0-4fb0-a446-efaf635cb7ff","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6809","nonce":2055631175},{"type":"v1","addr":"172.21.15.171:6811","nonce":2055631175}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6813","nonce":2055631175},{"type":"v1","addr":"172.21.15.171:6815","nonce":2055631175}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6822","nonce":2055631175},{"type":"v1","addr":"172.21.15.171:6823","nonce":2055631175}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6818","nonce":2055631175},{"type":"v1","addr":"172.21.15.171:6820","nonce":2055631175}]},"public_addr":"172.21.15.171:6811/2055631175","cluster_addr":"172.21.15.171:6815/2055631175","heartbeat_back_addr":"172.21.15.171:6823/2055631175","heartbeat_front_addr":"172.21.15.171:6820/2055631175","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T14:07:42.668881+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T14:07:42.708397+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T14:07:42.634191+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T14:07:42.733662+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T14:07:42.651541+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T14:07:42.783181+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T14:07:42.766310+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T14:07:42.670797+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-28T14:08:53.610 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:53.606+0000 7fdd3bdca640 1 -- 172.21.15.104:0/392715313 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7fdd18067ff0 msgr2=0x7fdd1806a4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:53.610 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:53.606+0000 7fdd3bdca640 1 --2- 172.21.15.104:0/392715313 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7fdd18067ff0 0x7fdd1806a4b0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fdd3000b800 tx=0x7fdd30008040 comp rx=0 tx=0).stop 2024-03-28T14:08:53.610 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:53.606+0000 7fdd3bdca640 1 -- 172.21.15.104:0/392715313 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fdd340b5600 msgr2=0x7fdd34144cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:53.610 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:53.606+0000 7fdd3bdca640 1 --2- 172.21.15.104:0/392715313 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fdd340b5600 0x7fdd34144cd0 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7fdd2c01a5e0 tx=0x7fdd2c019750 comp rx=0 tx=0).stop 2024-03-28T14:08:53.610 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:53.606+0000 7fdd3bdca640 1 -- 172.21.15.104:0/392715313 shutdown_connections 2024-03-28T14:08:53.610 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:53.606+0000 7fdd3bdca640 1 --2- 172.21.15.104:0/392715313 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7fdd18067ff0 0x7fdd1806a4b0 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:53.610 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:53.606+0000 7fdd3bdca640 1 --2- 172.21.15.104:0/392715313 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fdd340b5600 0x7fdd34144cd0 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:53.610 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:53.606+0000 7fdd3bdca640 1 --2- 172.21.15.104:0/392715313 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fdd340b5120 0x7fdd34144180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:53.611 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:53.606+0000 7fdd3bdca640 1 --2- 172.21.15.104:0/392715313 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fdd3406dd10 0x7fdd341e0f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:53.611 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:53.606+0000 7fdd3bdca640 1 -- 172.21.15.104:0/392715313 >> 172.21.15.104:0/392715313 conn(0x7fdd340ba390 msgr2=0x7fdd340717c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T14:08:53.611 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:53.606+0000 7fdd3bdca640 1 -- 172.21.15.104:0/392715313 shutdown_connections 2024-03-28T14:08:53.611 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:53.606+0000 7fdd3bdca640 1 -- 172.21.15.104:0/392715313 wait complete. 2024-03-28T14:08:53.625 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-03-28T14:08:53.625 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T14:08:53.736 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:53.734+0000 7f727a5c6640 1 Processor -- start 2024-03-28T14:08:53.736 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:53.734+0000 7f727a5c6640 1 -- start start 2024-03-28T14:08:53.737 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:53.734+0000 7f727a5c6640 1 --2- >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f72740b63a0 0x7f72740af9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:53.737 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:53.734+0000 7f727a5c6640 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f72741445e0 0x7f7274144a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:53.737 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:53.734+0000 7f727a5c6640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7274144f50 con 0x7f72740b0010 2024-03-28T14:08:53.738 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:53.734+0000 7f727a5c6640 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 -- 0x7f72740af610 con 0x7f72741445e0 2024-03-28T14:08:53.738 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:53.734+0000 7f727a5c6640 1 -- --> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] -- mon_getmap magic: 0 -- 0x7f72740b0490 con 0x7f72740b63a0 2024-03-28T14:08:53.738 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:53.734+0000 7f7278b63640 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f72741445e0 0x7f7274144a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:53.738 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:53.734+0000 7f7278b63640 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f72741445e0 0x7f7274144a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:37940/0 (socket says 172.21.15.104:37940) 2024-03-28T14:08:53.738 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:53.734+0000 7f7278b63640 1 -- 172.21.15.104:0/1406560018 learned_addr learned my addr 172.21.15.104:0/1406560018 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-03-28T14:08:53.738 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:53.734+0000 7f7273fff640 1 --2- 172.21.15.104:0/1406560018 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f72740b63a0 0x7f72740af9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:53.738 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:53.734+0000 7f7278b63640 1 -- 172.21.15.104:0/1406560018 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f72740b63a0 msgr2=0x7f72740af9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T14:08:53.738 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:53.734+0000 7f7278b63640 1 --2- 172.21.15.104:0/1406560018 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f72740b63a0 0x7f72740af9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:53.738 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:53.734+0000 7f7278b63640 1 -- 172.21.15.104:0/1406560018 >> v1:172.21.15.171:6789/0 conn(0x7f72740b0010 legacy=0x7f7274141670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:53.739 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:53.734+0000 7f7278b63640 1 -- 172.21.15.104:0/1406560018 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f72740b6780 con 0x7f72741445e0 2024-03-28T14:08:53.740 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:53.734+0000 7f7278b63640 1 --2- 172.21.15.104:0/1406560018 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f72741445e0 0x7f7274144a10 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f726c00d150 tx=0x7f726c01eba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5a9d8249cb34eaf0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:08:53.740 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:53.734+0000 7f7272ffd640 1 -- 172.21.15.104:0/1406560018 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f726c00b2d0 con 0x7f72741445e0 2024-03-28T14:08:53.740 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:53.734+0000 7f7272ffd640 1 -- 172.21.15.104:0/1406560018 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f726c00b470 con 0x7f72741445e0 2024-03-28T14:08:53.740 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:53.734+0000 7f7272ffd640 1 -- 172.21.15.104:0/1406560018 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f726c0a7a80 con 0x7f72741445e0 2024-03-28T14:08:53.740 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:53.738+0000 7f727a5c6640 1 -- 172.21.15.104:0/1406560018 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f72741445e0 msgr2=0x7f7274144a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:53.741 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:53.738+0000 7f727a5c6640 1 --2- 172.21.15.104:0/1406560018 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f72741445e0 0x7f7274144a10 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f726c00d150 tx=0x7f726c01eba0 comp rx=0 tx=0).stop 2024-03-28T14:08:53.741 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:53.738+0000 7f727a5c6640 1 -- 172.21.15.104:0/1406560018 shutdown_connections 2024-03-28T14:08:53.741 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:53.738+0000 7f727a5c6640 1 --2- 172.21.15.104:0/1406560018 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f72741445e0 0x7f7274144a10 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:53.741 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:53.738+0000 7f727a5c6640 1 --2- 172.21.15.104:0/1406560018 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f72740b63a0 0x7f72740af9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:53.741 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:53.738+0000 7f727a5c6640 1 -- 172.21.15.104:0/1406560018 >> 172.21.15.104:0/1406560018 conn(0x7f72740ba400 msgr2=0x7f72740ba810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T14:08:53.741 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:53.738+0000 7f727a5c6640 1 -- 172.21.15.104:0/1406560018 shutdown_connections 2024-03-28T14:08:53.741 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:53.738+0000 7f727a5c6640 1 -- 172.21.15.104:0/1406560018 wait complete. 2024-03-28T14:08:53.744 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:53.738+0000 7f727a5c6640 1 Processor -- start 2024-03-28T14:08:53.744 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:53.738+0000 7f727a5c6640 1 -- start start 2024-03-28T14:08:53.745 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:53.738+0000 7f727a5c6640 1 --2- >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f72740b0010 0x7f727409f8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:53.745 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:53.738+0000 7f727a5c6640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f72740b63a0 0x7f72741e2e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:53.745 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:53.738+0000 7f727a5c6640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f72741445e0 0x7f7274139c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:53.745 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:53.738+0000 7f727a5c6640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f727406bf60 con 0x7f72741445e0 2024-03-28T14:08:53.745 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:53.738+0000 7f727a5c6640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f727406bde0 con 0x7f72740b63a0 2024-03-28T14:08:53.745 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:53.738+0000 7f727a5c6640 1 -- --> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] -- mon_getmap magic: 0 -- 0x7f727406c0e0 con 0x7f72740b0010 2024-03-28T14:08:53.745 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:53.738+0000 7f7278b63640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f72741445e0 0x7f7274139c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:53.745 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:53.738+0000 7f7278b63640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f72741445e0 0x7f7274139c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:37954/0 (socket says 172.21.15.104:37954) 2024-03-28T14:08:53.746 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:53.738+0000 7f7278b63640 1 -- 172.21.15.104:0/2260790159 learned_addr learned my addr 172.21.15.104:0/2260790159 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-03-28T14:08:53.746 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:53.738+0000 7f7273fff640 1 --2- 172.21.15.104:0/2260790159 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f72740b0010 0x7f727409f8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:53.746 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:53.738+0000 7f72737fe640 1 --2- 172.21.15.104:0/2260790159 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f72740b63a0 0x7f72741e2e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:53.746 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:53.738+0000 7f7278b63640 1 -- 172.21.15.104:0/2260790159 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f72740b0010 msgr2=0x7f727409f8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:53.746 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:53.738+0000 7f7278b63640 1 --2- 172.21.15.104:0/2260790159 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f72740b0010 0x7f727409f8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:53.746 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:53.738+0000 7f7278b63640 1 -- 172.21.15.104:0/2260790159 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f72740b63a0 msgr2=0x7f72741e2e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:53.746 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:53.738+0000 7f7278b63640 1 --2- 172.21.15.104:0/2260790159 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f72740b63a0 0x7f72741e2e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:53.746 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:53.738+0000 7f7278b63640 1 -- 172.21.15.104:0/2260790159 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f72741383f0 con 0x7f72741445e0 2024-03-28T14:08:53.746 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:53.738+0000 7f72737fe640 1 --2- 172.21.15.104:0/2260790159 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f72740b63a0 0x7f72741e2e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T14:08:53.747 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:53.738+0000 7f7278b63640 1 --2- 172.21.15.104:0/2260790159 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f72741445e0 0x7f7274139c00 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f726c01c7d0 tx=0x7f726c01b940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:08:53.747 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:53.738+0000 7f7270ff9640 1 -- 172.21.15.104:0/2260790159 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f726c00b720 con 0x7f72741445e0 2024-03-28T14:08:53.747 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:53.738+0000 7f7270ff9640 1 -- 172.21.15.104:0/2260790159 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f726c0be070 con 0x7f72741445e0 2024-03-28T14:08:53.747 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:53.738+0000 7f7270ff9640 1 -- 172.21.15.104:0/2260790159 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f726c009a20 con 0x7f72741445e0 2024-03-28T14:08:53.747 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:53.738+0000 7f727a5c6640 1 -- 172.21.15.104:0/2260790159 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f72741dfc80 con 0x7f72741445e0 2024-03-28T14:08:53.747 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:53.738+0000 7f727a5c6640 1 -- 172.21.15.104:0/2260790159 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f72741e0760 con 0x7f72741445e0 2024-03-28T14:08:53.747 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:53.738+0000 7f727a5c6640 1 -- 172.21.15.104:0/2260790159 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f72740687c0 con 0x7f72741445e0 2024-03-28T14:08:53.747 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:53.742+0000 7f7270ff9640 1 -- 172.21.15.104:0/2260790159 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) ==== 87803+0+0 (secure 0 0 0) 0x7f726c0b6070 con 0x7f72741445e0 2024-03-28T14:08:53.747 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:53.742+0000 7f7270ff9640 1 --2- 172.21.15.104:0/2260790159 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f7260067fc0 0x7f726006a480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:53.747 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:53.742+0000 7f7270ff9640 1 -- 172.21.15.104:0/2260790159 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 5709+0+0 (secure 0 0 0) 0x7f726c1213c0 con 0x7f72741445e0 2024-03-28T14:08:53.747 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:53.742+0000 7f7273fff640 1 --2- 172.21.15.104:0/2260790159 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f7260067fc0 0x7f726006a480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:53.748 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:53.742+0000 7f7273fff640 1 --2- 172.21.15.104:0/2260790159 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f7260067fc0 0x7f726006a480 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f726400b9d0 tx=0x7f7264008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:08:53.748 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:53.746+0000 7f7270ff9640 1 -- 172.21.15.104:0/2260790159 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179117 (secure 0 0 0) 0x7f726c018820 con 0x7f72741445e0 2024-03-28T14:08:53.929 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:53.926+0000 7f727a5c6640 1 -- 172.21.15.104:0/2260790159 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f727402d400 con 0x7f72741445e0 2024-03-28T14:08:53.930 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:53.926+0000 7f7270ff9640 1 -- 172.21.15.104:0/2260790159 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 14 v14) ==== 94+0+3537 (secure 0 0 0) 0x7f726c0aee40 con 0x7f72741445e0 2024-03-28T14:08:53.930 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-03-28T14:08:53.930 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":14,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4550,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.171:6835/3418793776","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6834","nonce":3418793776},{"type":"v1","addr":"172.21.15.171:6835","nonce":3418793776}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11}],"filesystems":[{"mdsmap":{"epoch":14,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T14:08:40.854108+0000","modified":"2024-03-28T14:08:42.859657+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4959},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4959":{"gid":4959,"name":"a","rank":0,"incarnation":12,"state":"up:active","state_seq":3,"addr":"172.21.15.104:6833/1094990707","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":1094990707},{"type":"v1","addr":"172.21.15.104:6833","nonce":1094990707}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4959,"qdb_cluster":[4959]},"id":2}]} 2024-03-28T14:08:53.930 INFO:teuthology.orchestra.run.smithi104.stderr:dumped fsmap epoch 14 2024-03-28T14:08:53.932 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:53.926+0000 7f727a5c6640 1 -- 172.21.15.104:0/2260790159 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f7260067fc0 msgr2=0x7f726006a480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:53.932 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:53.926+0000 7f727a5c6640 1 --2- 172.21.15.104:0/2260790159 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f7260067fc0 0x7f726006a480 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f726400b9d0 tx=0x7f7264008040 comp rx=0 tx=0).stop 2024-03-28T14:08:53.932 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:53.926+0000 7f727a5c6640 1 -- 172.21.15.104:0/2260790159 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f72741445e0 msgr2=0x7f7274139c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:53.932 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:53.926+0000 7f727a5c6640 1 --2- 172.21.15.104:0/2260790159 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f72741445e0 0x7f7274139c00 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f726c01c7d0 tx=0x7f726c01b940 comp rx=0 tx=0).stop 2024-03-28T14:08:53.932 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:53.926+0000 7f727a5c6640 1 -- 172.21.15.104:0/2260790159 shutdown_connections 2024-03-28T14:08:53.932 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:53.926+0000 7f727a5c6640 1 --2- 172.21.15.104:0/2260790159 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f7260067fc0 0x7f726006a480 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:53.933 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:53.926+0000 7f727a5c6640 1 --2- 172.21.15.104:0/2260790159 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f72741445e0 0x7f7274139c00 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:53.933 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:53.930+0000 7f727a5c6640 1 --2- 172.21.15.104:0/2260790159 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f72740b63a0 0x7f72741e2e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:53.933 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:53.930+0000 7f727a5c6640 1 --2- 172.21.15.104:0/2260790159 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f72740b0010 0x7f727409f8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:53.933 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:53.930+0000 7f727a5c6640 1 -- 172.21.15.104:0/2260790159 >> 172.21.15.104:0/2260790159 conn(0x7f72740ba400 msgr2=0x7f7274142450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T14:08:53.933 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:53.930+0000 7f727a5c6640 1 -- 172.21.15.104:0/2260790159 shutdown_connections 2024-03-28T14:08:53.933 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:53.930+0000 7f727a5c6640 1 -- 172.21.15.104:0/2260790159 wait complete. 2024-03-28T14:08:53.947 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T14:08:54.056 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:54.050+0000 7efd3a551640 1 Processor -- start 2024-03-28T14:08:54.056 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:54.054+0000 7efd3a551640 1 -- start start 2024-03-28T14:08:54.058 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:54.054+0000 7efd3a551640 1 --2- >> v2:172.21.15.171:3300/0 conn(0x7efd3414af90 0x7efd340b7a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:54.058 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:54.054+0000 7efd3a551640 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7efd340b4e20 0x7efd34141a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:54.058 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:54.054+0000 7efd3a551640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efd340b6ea0 con 0x7efd340b5470 2024-03-28T14:08:54.058 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:54.054+0000 7efd3a551640 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 -- 0x7efd340b4990 con 0x7efd340b4e20 2024-03-28T14:08:54.058 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:54.054+0000 7efd3a551640 1 -- --> v2:172.21.15.171:3300/0 -- mon_getmap magic: 0 -- 0x7efd340b7180 con 0x7efd3414af90 2024-03-28T14:08:54.058 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:54.054+0000 7efd38aee640 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7efd340b4e20 0x7efd34141a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:54.058 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:54.054+0000 7efd38aee640 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7efd340b4e20 0x7efd34141a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:37962/0 (socket says 172.21.15.104:37962) 2024-03-28T14:08:54.058 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:54.054+0000 7efd38aee640 1 -- 172.21.15.104:0/2371977429 learned_addr learned my addr 172.21.15.104:0/2371977429 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-03-28T14:08:54.058 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:54.054+0000 7efd38aee640 1 -- 172.21.15.104:0/2371977429 >> v2:172.21.15.171:3300/0 conn(0x7efd3414af90 msgr2=0x7efd340b7a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T14:08:54.059 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:54.054+0000 7efd33fff640 1 --2- 172.21.15.104:0/2371977429 >> v2:172.21.15.171:3300/0 conn(0x7efd3414af90 0x7efd340b7a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:54.059 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:54.054+0000 7efd38aee640 1 --2- 172.21.15.104:0/2371977429 >> v2:172.21.15.171:3300/0 conn(0x7efd3414af90 0x7efd340b7a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:54.059 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:54.054+0000 7efd38aee640 1 -- 172.21.15.104:0/2371977429 >> v1:172.21.15.171:6789/0 conn(0x7efd340b5470 legacy=0x7efd34141360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T14:08:54.059 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:54.054+0000 7efd38aee640 1 -- 172.21.15.104:0/2371977429 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efd3414b7e0 con 0x7efd340b4e20 2024-03-28T14:08:54.062 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:54.054+0000 7efd38aee640 1 --2- 172.21.15.104:0/2371977429 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7efd340b4e20 0x7efd34141a70 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7efd340ba740 tx=0x7efd2c01c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f4ec2e10a7db8fa2 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:08:54.063 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:54.054+0000 7efd337fe640 1 -- 172.21.15.104:0/2371977429 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd2c0a5800 con 0x7efd340b4e20 2024-03-28T14:08:54.063 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:54.054+0000 7efd337fe640 1 -- 172.21.15.104:0/2371977429 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efd2c0a59a0 con 0x7efd340b4e20 2024-03-28T14:08:54.063 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:54.054+0000 7efd337fe640 1 -- 172.21.15.104:0/2371977429 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd2c0a5d70 con 0x7efd340b4e20 2024-03-28T14:08:54.063 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:54.054+0000 7efd3a551640 1 -- 172.21.15.104:0/2371977429 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7efd340b4e20 msgr2=0x7efd34141a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:54.063 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:54.054+0000 7efd3a551640 1 --2- 172.21.15.104:0/2371977429 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7efd340b4e20 0x7efd34141a70 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7efd340ba740 tx=0x7efd2c01c920 comp rx=0 tx=0).stop 2024-03-28T14:08:54.063 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:54.054+0000 7efd3a551640 1 -- 172.21.15.104:0/2371977429 shutdown_connections 2024-03-28T14:08:54.063 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:54.054+0000 7efd3a551640 1 --2- 172.21.15.104:0/2371977429 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7efd340b4e20 0x7efd34141a70 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:54.063 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:54.054+0000 7efd3a551640 1 --2- 172.21.15.104:0/2371977429 >> v2:172.21.15.171:3300/0 conn(0x7efd3414af90 0x7efd340b7a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:54.063 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:54.054+0000 7efd3a551640 1 -- 172.21.15.104:0/2371977429 >> 172.21.15.104:0/2371977429 conn(0x7efd340b9b60 msgr2=0x7efd340b9f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T14:08:54.063 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:54.054+0000 7efd3a551640 1 -- 172.21.15.104:0/2371977429 shutdown_connections 2024-03-28T14:08:54.063 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:54.054+0000 7efd3a551640 1 -- 172.21.15.104:0/2371977429 wait complete. 2024-03-28T14:08:54.063 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:54.058+0000 7efd3a551640 1 Processor -- start 2024-03-28T14:08:54.063 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:54.058+0000 7efd3a551640 1 -- start start 2024-03-28T14:08:54.064 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:54.058+0000 7efd3a551640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7efd340b4e20 0x7efd34144aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:54.064 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:54.058+0000 7efd3a551640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7efd340b5470 0x7efd34143550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:54.064 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:54.058+0000 7efd3a551640 1 --2- >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7efd3414af90 0x7efd341429d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:54.064 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:54.058+0000 7efd3a551640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7efd34154c70 con 0x7efd340b4e20 2024-03-28T14:08:54.064 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:54.058+0000 7efd3a551640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7efd34154af0 con 0x7efd340b5470 2024-03-28T14:08:54.064 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:54.058+0000 7efd3a551640 1 -- --> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] -- mon_getmap magic: 0 -- 0x7efd34154df0 con 0x7efd3414af90 2024-03-28T14:08:54.064 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:54.058+0000 7efd33fff640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7efd340b4e20 0x7efd34144aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:54.064 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:54.058+0000 7efd33fff640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7efd340b4e20 0x7efd34144aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:37972/0 (socket says 172.21.15.104:37972) 2024-03-28T14:08:54.064 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:54.058+0000 7efd33fff640 1 -- 172.21.15.104:0/4177413225 learned_addr learned my addr 172.21.15.104:0/4177413225 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-03-28T14:08:54.064 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:54.058+0000 7efd33fff640 1 -- 172.21.15.104:0/4177413225 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7efd3414af90 msgr2=0x7efd341429d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:54.064 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:54.058+0000 7efd38aee640 1 --2- 172.21.15.104:0/4177413225 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7efd3414af90 0x7efd341429d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:54.064 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:54.058+0000 7efd33fff640 1 --2- 172.21.15.104:0/4177413225 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7efd3414af90 0x7efd341429d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:54.065 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:54.058+0000 7efd2bfff640 1 --2- 172.21.15.104:0/4177413225 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7efd340b5470 0x7efd34143550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:54.065 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:54.058+0000 7efd33fff640 1 -- 172.21.15.104:0/4177413225 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7efd340b5470 msgr2=0x7efd34143550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:54.065 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:54.058+0000 7efd33fff640 1 --2- 172.21.15.104:0/4177413225 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7efd340b5470 0x7efd34143550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:54.065 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:54.058+0000 7efd33fff640 1 -- 172.21.15.104:0/4177413225 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efd341420a0 con 0x7efd340b4e20 2024-03-28T14:08:54.065 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:54.058+0000 7efd33fff640 1 --2- 172.21.15.104:0/4177413225 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7efd340b4e20 0x7efd34144aa0 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7efd2400c7f0 tx=0x7efd2400ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:08:54.065 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:54.058+0000 7efd317fa640 1 -- 172.21.15.104:0/4177413225 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd24015070 con 0x7efd340b4e20 2024-03-28T14:08:54.065 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:54.058+0000 7efd317fa640 1 -- 172.21.15.104:0/4177413225 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efd24007820 con 0x7efd340b4e20 2024-03-28T14:08:54.065 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:54.058+0000 7efd3a551640 1 -- 172.21.15.104:0/4177413225 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efd34144040 con 0x7efd340b4e20 2024-03-28T14:08:54.065 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:54.058+0000 7efd317fa640 1 -- 172.21.15.104:0/4177413225 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd24007c10 con 0x7efd340b4e20 2024-03-28T14:08:54.065 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:54.058+0000 7efd3a551640 1 -- 172.21.15.104:0/4177413225 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efd3413e7d0 con 0x7efd340b4e20 2024-03-28T14:08:54.065 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:54.058+0000 7efd3a551640 1 -- 172.21.15.104:0/4177413225 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efd340687c0 con 0x7efd340b4e20 2024-03-28T14:08:54.066 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:54.058+0000 7efd317fa640 1 -- 172.21.15.104:0/4177413225 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) ==== 87803+0+0 (secure 0 0 0) 0x7efd24018070 con 0x7efd340b4e20 2024-03-28T14:08:54.066 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:54.062+0000 7efd317fa640 1 --2- 172.21.15.104:0/4177413225 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7efd14067fc0 0x7efd1406a480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:54.066 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:54.062+0000 7efd317fa640 1 -- 172.21.15.104:0/4177413225 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 5709+0+0 (secure 0 0 0) 0x7efd240864f0 con 0x7efd340b4e20 2024-03-28T14:08:54.066 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:54.062+0000 7efd2bfff640 1 --2- 172.21.15.104:0/4177413225 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7efd14067fc0 0x7efd1406a480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:54.066 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:54.062+0000 7efd2bfff640 1 --2- 172.21.15.104:0/4177413225 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7efd14067fc0 0x7efd1406a480 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7efd3414a9b0 tx=0x7efd1c009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:08:54.068 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:54.066+0000 7efd317fa640 1 -- 172.21.15.104:0/4177413225 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179117 (secure 0 0 0) 0x7efd24056d30 con 0x7efd340b4e20 2024-03-28T14:08:54.247 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:54.242+0000 7efd3a551640 1 -- 172.21.15.104:0/4177413225 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7efd3406bba0 con 0x7efd340b4e20 2024-03-28T14:08:54.247 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:54.242+0000 7efd317fa640 1 -- 172.21.15.104:0/4177413225 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 14 v14) ==== 94+0+3537 (secure 0 0 0) 0x7efd24056630 con 0x7efd340b4e20 2024-03-28T14:08:54.248 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-03-28T14:08:54.248 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":14,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4550,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.171:6835/3418793776","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6834","nonce":3418793776},{"type":"v1","addr":"172.21.15.171:6835","nonce":3418793776}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11}],"filesystems":[{"mdsmap":{"epoch":14,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T14:08:40.854108+0000","modified":"2024-03-28T14:08:42.859657+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4959},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4959":{"gid":4959,"name":"a","rank":0,"incarnation":12,"state":"up:active","state_seq":3,"addr":"172.21.15.104:6833/1094990707","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":1094990707},{"type":"v1","addr":"172.21.15.104:6833","nonce":1094990707}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4959,"qdb_cluster":[4959]},"id":2}]} 2024-03-28T14:08:54.248 INFO:teuthology.orchestra.run.smithi104.stderr:dumped fsmap epoch 14 2024-03-28T14:08:54.250 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:54.246+0000 7efd3a551640 1 -- 172.21.15.104:0/4177413225 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7efd14067fc0 msgr2=0x7efd1406a480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:54.250 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:54.246+0000 7efd3a551640 1 --2- 172.21.15.104:0/4177413225 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7efd14067fc0 0x7efd1406a480 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7efd3414a9b0 tx=0x7efd1c009040 comp rx=0 tx=0).stop 2024-03-28T14:08:54.250 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:54.246+0000 7efd3a551640 1 -- 172.21.15.104:0/4177413225 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7efd340b4e20 msgr2=0x7efd34144aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:54.250 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:54.246+0000 7efd3a551640 1 --2- 172.21.15.104:0/4177413225 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7efd340b4e20 0x7efd34144aa0 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7efd2400c7f0 tx=0x7efd2400ccb0 comp rx=0 tx=0).stop 2024-03-28T14:08:54.250 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:54.246+0000 7efd3a551640 1 -- 172.21.15.104:0/4177413225 shutdown_connections 2024-03-28T14:08:54.250 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:54.246+0000 7efd3a551640 1 --2- 172.21.15.104:0/4177413225 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7efd3414af90 0x7efd341429d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:54.250 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:54.246+0000 7efd3a551640 1 --2- 172.21.15.104:0/4177413225 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7efd340b5470 0x7efd34143550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:54.250 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:54.246+0000 7efd3a551640 1 --2- 172.21.15.104:0/4177413225 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7efd14067fc0 0x7efd1406a480 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:54.251 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:54.246+0000 7efd3a551640 1 --2- 172.21.15.104:0/4177413225 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7efd340b4e20 0x7efd34144aa0 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:54.251 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:54.246+0000 7efd3a551640 1 -- 172.21.15.104:0/4177413225 >> 172.21.15.104:0/4177413225 conn(0x7efd340b9b60 msgr2=0x7efd34137ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T14:08:54.251 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:54.246+0000 7efd3a551640 1 -- 172.21.15.104:0/4177413225 shutdown_connections 2024-03-28T14:08:54.251 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:54.246+0000 7efd3a551640 1 -- 172.21.15.104:0/4177413225 wait complete. 2024-03-28T14:08:54.266 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-28T14:08:54.373 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:54.370+0000 7f0517494640 1 Processor -- start 2024-03-28T14:08:54.373 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:54.370+0000 7f0517494640 1 -- start start 2024-03-28T14:08:54.375 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:54.370+0000 7f0517494640 1 --2- >> v2:172.21.15.171:3300/0 conn(0x7f05100ba970 0x7f05100b5d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:54.375 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:54.370+0000 7f0517494640 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f051006b5d0 0x7f051006ba80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:54.375 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:54.370+0000 7f0517494640 1 --2- >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f051006bfc0 0x7f051014b090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:54.375 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:54.370+0000 7f0517494640 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 -- 0x7f05100b5910 con 0x7f051006b5d0 2024-03-28T14:08:54.375 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:54.370+0000 7f0517494640 1 -- --> v2:172.21.15.171:3300/0 -- mon_getmap magic: 0 -- 0x7f051006c430 con 0x7f05100ba970 2024-03-28T14:08:54.375 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:54.370+0000 7f0517494640 1 -- --> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] -- mon_getmap magic: 0 -- 0x7f051006ea00 con 0x7f051006bfc0 2024-03-28T14:08:54.375 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:54.370+0000 7f0514a2f640 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f051006b5d0 0x7f051006ba80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:54.375 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:54.370+0000 7f0514a2f640 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f051006b5d0 0x7f051006ba80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:37980/0 (socket says 172.21.15.104:37980) 2024-03-28T14:08:54.376 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:54.370+0000 7f0514a2f640 1 -- 172.21.15.104:0/3303613753 learned_addr learned my addr 172.21.15.104:0/3303613753 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-03-28T14:08:54.376 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:54.370+0000 7f0514a2f640 1 -- 172.21.15.104:0/3303613753 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f051006bfc0 msgr2=0x7f051014b090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T14:08:54.376 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:54.370+0000 7f0514a2f640 1 --2- 172.21.15.104:0/3303613753 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f051006bfc0 0x7f051014b090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:54.376 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:54.370+0000 7f0514a2f640 1 -- 172.21.15.104:0/3303613753 >> v2:172.21.15.171:3300/0 conn(0x7f05100ba970 msgr2=0x7f05100b5d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T14:08:54.376 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:54.370+0000 7f0515230640 1 --2- 172.21.15.104:0/3303613753 >> v2:172.21.15.171:3300/0 conn(0x7f05100ba970 0x7f05100b5d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:54.376 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:54.370+0000 7f0514a2f640 1 --2- 172.21.15.104:0/3303613753 >> v2:172.21.15.171:3300/0 conn(0x7f05100ba970 0x7f05100b5d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:54.376 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:54.370+0000 7f0514a2f640 1 -- 172.21.15.104:0/3303613753 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f051006e760 con 0x7f051006b5d0 2024-03-28T14:08:54.378 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:54.374+0000 7f0514a2f640 1 --2- 172.21.15.104:0/3303613753 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f051006b5d0 0x7f051006ba80 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f051006da00 tx=0x7f050001c1d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=359a6a9b0330dc16 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:08:54.378 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:54.374+0000 7f0507fff640 1 -- 172.21.15.104:0/3303613753 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05000b0070 con 0x7f051006b5d0 2024-03-28T14:08:54.378 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:54.374+0000 7f0507fff640 1 -- 172.21.15.104:0/3303613753 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f05000ab600 con 0x7f051006b5d0 2024-03-28T14:08:54.378 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:54.374+0000 7f0507fff640 1 -- 172.21.15.104:0/3303613753 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05000ab9d0 con 0x7f051006b5d0 2024-03-28T14:08:54.379 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:54.374+0000 7f0517494640 1 -- 172.21.15.104:0/3303613753 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f051006b5d0 msgr2=0x7f051006ba80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:54.379 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:54.374+0000 7f0517494640 1 --2- 172.21.15.104:0/3303613753 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f051006b5d0 0x7f051006ba80 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f051006da00 tx=0x7f050001c1d0 comp rx=0 tx=0).stop 2024-03-28T14:08:54.379 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:54.374+0000 7f0517494640 1 -- 172.21.15.104:0/3303613753 shutdown_connections 2024-03-28T14:08:54.379 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:54.374+0000 7f0517494640 1 --2- 172.21.15.104:0/3303613753 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f051006bfc0 0x7f051014b090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:54.380 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:54.374+0000 7f0517494640 1 --2- 172.21.15.104:0/3303613753 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f051006b5d0 0x7f051006ba80 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:54.380 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:54.374+0000 7f0517494640 1 --2- 172.21.15.104:0/3303613753 >> v2:172.21.15.171:3300/0 conn(0x7f05100ba970 0x7f05100b5d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:54.380 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:54.374+0000 7f0517494640 1 -- 172.21.15.104:0/3303613753 >> 172.21.15.104:0/3303613753 conn(0x7f051006cd70 msgr2=0x7f05100ba000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T14:08:54.380 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:54.374+0000 7f0517494640 1 -- 172.21.15.104:0/3303613753 shutdown_connections 2024-03-28T14:08:54.381 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:54.374+0000 7f0517494640 1 -- 172.21.15.104:0/3303613753 wait complete. 2024-03-28T14:08:54.381 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:54.374+0000 7f0517494640 1 Processor -- start 2024-03-28T14:08:54.381 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:54.374+0000 7f0517494640 1 -- start start 2024-03-28T14:08:54.385 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:54.374+0000 7f0517494640 1 --2- >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f051006b5d0 0x7f0510070d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:54.385 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:54.374+0000 7f0517494640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f051006bfc0 0x7f05101e06b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:54.385 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:54.374+0000 7f0517494640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f05100ba970 0x7f05101e0bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:54.385 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:54.374+0000 7f0517494640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f05100b62d0 con 0x7f051006bfc0 2024-03-28T14:08:54.386 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:54.374+0000 7f0517494640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f05100b4b30 con 0x7f05100ba970 2024-03-28T14:08:54.386 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:54.374+0000 7f0517494640 1 -- --> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] -- mon_getmap magic: 0 -- 0x7f05100af3c0 con 0x7f051006b5d0 2024-03-28T14:08:54.386 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:54.374+0000 7f0514a2f640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f051006bfc0 0x7f05101e06b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:54.386 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:54.374+0000 7f0514a2f640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f051006bfc0 0x7f05101e06b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:37982/0 (socket says 172.21.15.104:37982) 2024-03-28T14:08:54.386 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:54.374+0000 7f0514a2f640 1 -- 172.21.15.104:0/4148614945 learned_addr learned my addr 172.21.15.104:0/4148614945 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-03-28T14:08:54.386 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:54.378+0000 7f0515230640 1 --2- 172.21.15.104:0/4148614945 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f051006b5d0 0x7f0510070d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:54.386 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:54.378+0000 7f0514a2f640 1 -- 172.21.15.104:0/4148614945 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f051006b5d0 msgr2=0x7f0510070d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:54.386 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:54.378+0000 7f0514a2f640 1 --2- 172.21.15.104:0/4148614945 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f051006b5d0 0x7f0510070d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:54.386 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:54.378+0000 7f0515a31640 1 --2- 172.21.15.104:0/4148614945 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f05100ba970 0x7f05101e0bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:54.387 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:54.378+0000 7f0514a2f640 1 -- 172.21.15.104:0/4148614945 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f05100ba970 msgr2=0x7f05101e0bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:54.387 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:54.378+0000 7f0514a2f640 1 --2- 172.21.15.104:0/4148614945 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f05100ba970 0x7f05101e0bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:54.387 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:54.378+0000 7f0514a2f640 1 -- 172.21.15.104:0/4148614945 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f051009ec00 con 0x7f051006bfc0 2024-03-28T14:08:54.387 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:54.378+0000 7f0514a2f640 1 --2- 172.21.15.104:0/4148614945 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f051006bfc0 0x7f05101e06b0 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f0500000c10 tx=0x7f0500018f70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:08:54.388 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:54.378+0000 7f0505ffb640 1 -- 172.21.15.104:0/4148614945 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05000b8020 con 0x7f051006bfc0 2024-03-28T14:08:54.388 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:54.378+0000 7f0505ffb640 1 -- 172.21.15.104:0/4148614945 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f05000aa7a0 con 0x7f051006bfc0 2024-03-28T14:08:54.388 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:54.378+0000 7f0505ffb640 1 -- 172.21.15.104:0/4148614945 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05000b0040 con 0x7f051006bfc0 2024-03-28T14:08:54.388 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:54.378+0000 7f0517494640 1 -- 172.21.15.104:0/4148614945 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0510071f00 con 0x7f051006bfc0 2024-03-28T14:08:54.388 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:54.378+0000 7f0517494640 1 -- 172.21.15.104:0/4148614945 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f051013b310 con 0x7f051006bfc0 2024-03-28T14:08:54.388 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:54.378+0000 7f0517494640 1 -- 172.21.15.104:0/4148614945 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f051013a770 con 0x7f051006bfc0 2024-03-28T14:08:54.388 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:54.378+0000 7f0505ffb640 1 -- 172.21.15.104:0/4148614945 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) ==== 87803+0+0 (secure 0 0 0) 0x7f05000bb020 con 0x7f051006bfc0 2024-03-28T14:08:54.389 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:54.378+0000 7f0505ffb640 1 --2- 172.21.15.104:0/4148614945 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f04f4067fc0 0x7f04f406a480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:54.389 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:54.378+0000 7f0505ffb640 1 -- 172.21.15.104:0/4148614945 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 5709+0+0 (secure 0 0 0) 0x7f0500120460 con 0x7f051006bfc0 2024-03-28T14:08:54.389 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:54.378+0000 7f0515230640 1 --2- 172.21.15.104:0/4148614945 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f04f4067fc0 0x7f04f406a480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:54.389 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:54.378+0000 7f0515230640 1 --2- 172.21.15.104:0/4148614945 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f04f4067fc0 0x7f04f406a480 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f0510140e40 tx=0x7f050c008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:08:54.390 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:54.382+0000 7f0505ffb640 1 -- 172.21.15.104:0/4148614945 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179117 (secure 0 0 0) 0x7f05000c2030 con 0x7f051006bfc0 2024-03-28T14:08:54.586 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:54.582+0000 7f0517494640 1 -- 172.21.15.104:0/4148614945 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f05101e0410 con 0x7f051006bfc0 2024-03-28T14:08:54.588 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:54.586+0000 7f0505ffb640 1 -- 172.21.15.104:0/4148614945 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v30) ==== 74+0+16693 (secure 0 0 0) 0x7f05000a4e20 con 0x7f051006bfc0 2024-03-28T14:08:54.589 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-03-28T14:08:54.589 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":30,"fsid":"428a90f2-3351-435f-8fcf-f7e66c78a13e","created":"2024-03-28T14:07:34.862458+0000","modified":"2024-03-28T14:08:42.477796+0000","last_up_change":"2024-03-28T14:07:46.028122+0000","last_in_change":"2024-03-28T14:07:37.157486+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-28T14:07:47.622988+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_metadata","create_time":"2024-03-28T14:08:38.687068+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data","create_time":"2024-03-28T14:08:39.805358+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"30","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"1cb70540-4e42-402f-a446-aaec4ca83007","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6800","nonce":3745690918},{"type":"v1","addr":"172.21.15.104:6801","nonce":3745690918}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":3745690918},{"type":"v1","addr":"172.21.15.104:6803","nonce":3745690918}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":3745690918},{"type":"v1","addr":"172.21.15.104:6807","nonce":3745690918}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":3745690918},{"type":"v1","addr":"172.21.15.104:6805","nonce":3745690918}]},"public_addr":"172.21.15.104:6801/3745690918","cluster_addr":"172.21.15.104:6803/3745690918","heartbeat_back_addr":"172.21.15.104:6807/3745690918","heartbeat_front_addr":"172.21.15.104:6805/3745690918","state":["exists","up"]},{"osd":1,"uuid":"3ab7afdd-b980-418c-81f2-004e16de3dca","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":19438674},{"type":"v1","addr":"172.21.15.104:6810","nonce":19438674}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":19438674},{"type":"v1","addr":"172.21.15.104:6814","nonce":19438674}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":19438674},{"type":"v1","addr":"172.21.15.104:6823","nonce":19438674}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":19438674},{"type":"v1","addr":"172.21.15.104:6819","nonce":19438674}]},"public_addr":"172.21.15.104:6810/19438674","cluster_addr":"172.21.15.104:6814/19438674","heartbeat_back_addr":"172.21.15.104:6823/19438674","heartbeat_front_addr":"172.21.15.104:6819/19438674","state":["exists","up"]},{"osd":2,"uuid":"3e2651c8-ecae-4a9e-a053-27240a79eac2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":2352012626},{"type":"v1","addr":"172.21.15.104:6825","nonce":2352012626}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6826","nonce":2352012626},{"type":"v1","addr":"172.21.15.104:6827","nonce":2352012626}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":2352012626},{"type":"v1","addr":"172.21.15.104:6831","nonce":2352012626}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6828","nonce":2352012626},{"type":"v1","addr":"172.21.15.104:6829","nonce":2352012626}]},"public_addr":"172.21.15.104:6825/2352012626","cluster_addr":"172.21.15.104:6827/2352012626","heartbeat_back_addr":"172.21.15.104:6831/2352012626","heartbeat_front_addr":"172.21.15.104:6829/2352012626","state":["exists","up"]},{"osd":3,"uuid":"925c7e0a-38ef-47cd-b288-8bec8456fb17","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6809","nonce":3568347381},{"type":"v1","addr":"172.21.15.104:6811","nonce":3568347381}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6813","nonce":3568347381},{"type":"v1","addr":"172.21.15.104:6815","nonce":3568347381}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":3568347381},{"type":"v1","addr":"172.21.15.104:6822","nonce":3568347381}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6817","nonce":3568347381},{"type":"v1","addr":"172.21.15.104:6818","nonce":3568347381}]},"public_addr":"172.21.15.104:6811/3568347381","cluster_addr":"172.21.15.104:6815/3568347381","heartbeat_back_addr":"172.21.15.104:6822/3568347381","heartbeat_front_addr":"172.21.15.104:6818/3568347381","state":["exists","up"]},{"osd":4,"uuid":"4a8fac86-e595-4a0e-96e3-57edb3cc3f57","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6800","nonce":2107001077},{"type":"v1","addr":"172.21.15.171:6801","nonce":2107001077}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6802","nonce":2107001077},{"type":"v1","addr":"172.21.15.171:6803","nonce":2107001077}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6806","nonce":2107001077},{"type":"v1","addr":"172.21.15.171:6807","nonce":2107001077}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6804","nonce":2107001077},{"type":"v1","addr":"172.21.15.171:6805","nonce":2107001077}]},"public_addr":"172.21.15.171:6801/2107001077","cluster_addr":"172.21.15.171:6803/2107001077","heartbeat_back_addr":"172.21.15.171:6807/2107001077","heartbeat_front_addr":"172.21.15.171:6805/2107001077","state":["exists","up"]},{"osd":5,"uuid":"e48d50b0-beae-40ae-b15a-9a84dbfe88ab","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6808","nonce":2934826925},{"type":"v1","addr":"172.21.15.171:6810","nonce":2934826925}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6812","nonce":2934826925},{"type":"v1","addr":"172.21.15.171:6814","nonce":2934826925}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6819","nonce":2934826925},{"type":"v1","addr":"172.21.15.171:6821","nonce":2934826925}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6816","nonce":2934826925},{"type":"v1","addr":"172.21.15.171:6817","nonce":2934826925}]},"public_addr":"172.21.15.171:6810/2934826925","cluster_addr":"172.21.15.171:6814/2934826925","heartbeat_back_addr":"172.21.15.171:6821/2934826925","heartbeat_front_addr":"172.21.15.171:6817/2934826925","state":["exists","up"]},{"osd":6,"uuid":"5e7c3292-9936-4f35-b267-aed9d4e49323","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6824","nonce":436742919},{"type":"v1","addr":"172.21.15.171:6825","nonce":436742919}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6826","nonce":436742919},{"type":"v1","addr":"172.21.15.171:6827","nonce":436742919}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6830","nonce":436742919},{"type":"v1","addr":"172.21.15.171:6831","nonce":436742919}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6828","nonce":436742919},{"type":"v1","addr":"172.21.15.171:6829","nonce":436742919}]},"public_addr":"172.21.15.171:6825/436742919","cluster_addr":"172.21.15.171:6827/436742919","heartbeat_back_addr":"172.21.15.171:6831/436742919","heartbeat_front_addr":"172.21.15.171:6829/436742919","state":["exists","up"]},{"osd":7,"uuid":"339b8414-d7f0-4fb0-a446-efaf635cb7ff","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6809","nonce":2055631175},{"type":"v1","addr":"172.21.15.171:6811","nonce":2055631175}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6813","nonce":2055631175},{"type":"v1","addr":"172.21.15.171:6815","nonce":2055631175}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6822","nonce":2055631175},{"type":"v1","addr":"172.21.15.171:6823","nonce":2055631175}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6818","nonce":2055631175},{"type":"v1","addr":"172.21.15.171:6820","nonce":2055631175}]},"public_addr":"172.21.15.171:6811/2055631175","cluster_addr":"172.21.15.171:6815/2055631175","heartbeat_back_addr":"172.21.15.171:6823/2055631175","heartbeat_front_addr":"172.21.15.171:6820/2055631175","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T14:07:42.668881+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T14:07:42.708397+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T14:07:42.634191+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T14:07:42.733662+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T14:07:42.651541+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T14:07:42.783181+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T14:07:42.766310+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T14:07:42.670797+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-28T14:08:54.593 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:54.590+0000 7f0517494640 1 -- 172.21.15.104:0/4148614945 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f04f4067fc0 msgr2=0x7f04f406a480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:54.593 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:54.590+0000 7f0517494640 1 --2- 172.21.15.104:0/4148614945 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f04f4067fc0 0x7f04f406a480 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f0510140e40 tx=0x7f050c008040 comp rx=0 tx=0).stop 2024-03-28T14:08:54.593 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:54.590+0000 7f0517494640 1 -- 172.21.15.104:0/4148614945 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f051006bfc0 msgr2=0x7f05101e06b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:54.593 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:54.590+0000 7f0517494640 1 --2- 172.21.15.104:0/4148614945 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f051006bfc0 0x7f05101e06b0 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f0500000c10 tx=0x7f0500018f70 comp rx=0 tx=0).stop 2024-03-28T14:08:54.593 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:54.590+0000 7f0517494640 1 -- 172.21.15.104:0/4148614945 shutdown_connections 2024-03-28T14:08:54.593 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:54.590+0000 7f0517494640 1 --2- 172.21.15.104:0/4148614945 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f05100ba970 0x7f05101e0bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:54.593 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:54.590+0000 7f0517494640 1 --2- 172.21.15.104:0/4148614945 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f04f4067fc0 0x7f04f406a480 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:54.593 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:54.590+0000 7f0517494640 1 --2- 172.21.15.104:0/4148614945 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f051006bfc0 0x7f05101e06b0 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:54.594 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:54.590+0000 7f0517494640 1 --2- 172.21.15.104:0/4148614945 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f051006b5d0 0x7f0510070d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:54.594 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:54.590+0000 7f0517494640 1 -- 172.21.15.104:0/4148614945 >> 172.21.15.104:0/4148614945 conn(0x7f051006cd70 msgr2=0x7f05100b7250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T14:08:54.594 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:54.590+0000 7f0517494640 1 -- 172.21.15.104:0/4148614945 shutdown_connections 2024-03-28T14:08:54.594 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:54.590+0000 7f0517494640 1 -- 172.21.15.104:0/4148614945 wait complete. 2024-03-28T14:08:54.609 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-03-28T14:08:54.722 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:54.718+0000 7f22bbd0a640 1 Processor -- start 2024-03-28T14:08:54.723 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:54.718+0000 7f22bbd0a640 1 -- start start 2024-03-28T14:08:54.725 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:54.718+0000 7f22bbd0a640 1 --2- >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f22b40b5120 0x7f22b4153290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:54.725 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:54.718+0000 7f22bbd0a640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f22b406e0f0 con 0x7f22b406dd10 2024-03-28T14:08:54.725 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:54.718+0000 7f22bbd0a640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f22b40b0ee0 con 0x7f22b40b5600 2024-03-28T14:08:54.725 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:54.718+0000 7f22bbd0a640 1 -- --> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] -- mon_getmap magic: 0 -- 0x7f22b40b0b70 con 0x7f22b40b5120 2024-03-28T14:08:54.725 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:54.718+0000 7f22b9aa6640 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f22b406dd10 0x7f22b4072cf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:54790/0 (socket says 172.21.15.104:54790) 2024-03-28T14:08:54.726 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:54.718+0000 7f22b9aa6640 1 -- 172.21.15.104:0/2948294820 learned_addr learned my addr 172.21.15.104:0/2948294820 (peer_addr_for_me v1:172.21.15.104:0/0) 2024-03-28T14:08:54.726 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:54.722+0000 7f22ba2a7640 1 --2- 172.21.15.104:0/2948294820 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f22b40b5120 0x7f22b4153290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:54.726 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:54.722+0000 7f22b8aa4640 1 -- 172.21.15.104:0/2948294820 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1729005970 0 0) 0x7f22b406e0f0 con 0x7f22b406dd10 2024-03-28T14:08:54.726 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:54.722+0000 7f22b8aa4640 1 -- 172.21.15.104:0/2948294820 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f22a0003530 con 0x7f22b406dd10 2024-03-28T14:08:54.726 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:54.722+0000 7f22b8aa4640 1 -- 172.21.15.104:0/2948294820 <== mon.0 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3134363245 0 0) 0x7f22a0003530 con 0x7f22b406dd10 2024-03-28T14:08:54.726 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:54.722+0000 7f22b8aa4640 1 -- 172.21.15.104:0/2948294820 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f22b40b5120 msgr2=0x7f22b4153290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T14:08:54.726 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:54.722+0000 7f22b8aa4640 1 --2- 172.21.15.104:0/2948294820 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f22b40b5120 0x7f22b4153290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:54.726 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:54.722+0000 7f22b8aa4640 1 -- 172.21.15.104:0/2948294820 >> v1:172.21.15.171:6789/0 conn(0x7f22b40b5600 legacy=0x7f22b41417b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:54.726 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:54.722+0000 7f22b8aa4640 1 -- 172.21.15.104:0/2948294820 --> v1:172.21.15.104:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f22b40b6130 con 0x7f22b406dd10 2024-03-28T14:08:54.726 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:54.722+0000 7f22b8aa4640 1 -- 172.21.15.104:0/2948294820 <== mon.0 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3745580849 0 0) 0x7f22b0001680 con 0x7f22b406dd10 2024-03-28T14:08:54.726 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:54.722+0000 7f22b8aa4640 1 -- 172.21.15.104:0/2948294820 >> v1:172.21.15.104:6789/0 conn(0x7f22b406dd10 legacy=0x7f22b4072cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:54.726 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:54.722+0000 7f22b8aa4640 1 --2- 172.21.15.104:0/2948294820 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f22b40b5120 0x7f22a00039f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:54.726 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:54.722+0000 7f22b8aa4640 1 --2- 172.21.15.104:0/2948294820 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f22a0003f70 0x7f22a0004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:54.727 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:54.722+0000 7f22b8aa4640 1 --2- 172.21.15.104:0/2948294820 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f22a0004d50 0x7f22a00051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:54.727 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:54.722+0000 7f22b8aa4640 1 -- 172.21.15.104:0/2948294820 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f22b40b0b70 con 0x7f22a0003f70 2024-03-28T14:08:54.727 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:54.722+0000 7f22b8aa4640 1 -- 172.21.15.104:0/2948294820 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f22a000bab0 con 0x7f22a0004d50 2024-03-28T14:08:54.727 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:54.722+0000 7f22b8aa4640 1 -- 172.21.15.104:0/2948294820 --> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] -- mon_getmap magic: 0 -- 0x7f22a000bcb0 con 0x7f22b40b5120 2024-03-28T14:08:54.727 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:54.722+0000 7f22b92a5640 1 --2- 172.21.15.104:0/2948294820 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f22a0003f70 0x7f22a0004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:54.727 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:54.722+0000 7f22ba2a7640 1 --2- 172.21.15.104:0/2948294820 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f22a0004d50 0x7f22a00051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:54.727 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:54.722+0000 7f22b92a5640 1 -- 172.21.15.104:0/2948294820 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f22b40b5120 msgr2=0x7f22a00039f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T14:08:54.727 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:54.722+0000 7f22b92a5640 1 --2- 172.21.15.104:0/2948294820 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f22b40b5120 0x7f22a00039f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:54.727 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:54.722+0000 7f22b92a5640 1 -- 172.21.15.104:0/2948294820 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f22a0004d50 msgr2=0x7f22a00051d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T14:08:54.727 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:54.722+0000 7f22b92a5640 1 --2- 172.21.15.104:0/2948294820 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f22a0004d50 0x7f22a00051d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:54.727 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:54.722+0000 7f22b92a5640 1 -- 172.21.15.104:0/2948294820 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f22a0003530 con 0x7f22a0003f70 2024-03-28T14:08:54.729 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:54.722+0000 7f22b92a5640 1 --2- 172.21.15.104:0/2948294820 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f22a0003f70 0x7f22a0004810 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f22a4001780 tx=0x7f22a401c420 comp rx=0 tx=0).ready entity=mon.0 client_cookie=46cc5a70895b053d server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:08:54.729 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:54.722+0000 7f22b8aa4640 1 -- 172.21.15.104:0/2948294820 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22a40013a0 con 0x7f22a0003f70 2024-03-28T14:08:54.729 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:54.722+0000 7f22b8aa4640 1 -- 172.21.15.104:0/2948294820 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f22a40a9070 con 0x7f22a0003f70 2024-03-28T14:08:54.729 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:54.722+0000 7f22b8aa4640 1 -- 172.21.15.104:0/2948294820 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22a4003600 con 0x7f22a0003f70 2024-03-28T14:08:54.729 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:54.722+0000 7f22bbd0a640 1 -- 172.21.15.104:0/2948294820 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f22a0003f70 msgr2=0x7f22a0004810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:54.729 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:54.722+0000 7f22bbd0a640 1 --2- 172.21.15.104:0/2948294820 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f22a0003f70 0x7f22a0004810 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f22a4001780 tx=0x7f22a401c420 comp rx=0 tx=0).stop 2024-03-28T14:08:54.729 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:54.722+0000 7f22ba2a7640 1 -- 172.21.15.104:0/2948294820 reap_dead start 2024-03-28T14:08:54.730 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:54.722+0000 7f22bbd0a640 1 -- 172.21.15.104:0/2948294820 shutdown_connections 2024-03-28T14:08:54.730 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:54.722+0000 7f22bbd0a640 1 -- 172.21.15.104:0/2948294820 >> 172.21.15.104:0/2948294820 conn(0x7f22b40ba390 msgr2=0x7f22b40ba7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T14:08:54.730 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:54.722+0000 7f22bbd0a640 1 -- 172.21.15.104:0/2948294820 shutdown_connections 2024-03-28T14:08:54.730 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:54.722+0000 7f22bbd0a640 1 -- 172.21.15.104:0/2948294820 wait complete. 2024-03-28T14:08:54.730 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:54.726+0000 7f22bbd0a640 1 Processor -- start 2024-03-28T14:08:54.730 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:54.726+0000 7f22bbd0a640 1 -- start start 2024-03-28T14:08:54.733 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:54.726+0000 7f22bbd0a640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f22a0003f70 0x7f22b41e13d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:54.733 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:54.726+0000 7f22bbd0a640 1 --2- >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f22b4145570 0x7f22b4148da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:54.733 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:54.726+0000 7f22bbd0a640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f22b414b290 0x7f22b41dd300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:54.733 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:54.726+0000 7f22bbd0a640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f22b40af300 con 0x7f22b414b290 2024-03-28T14:08:54.733 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:54.726+0000 7f22bbd0a640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f22b4142150 con 0x7f22a0003f70 2024-03-28T14:08:54.734 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:54.726+0000 7f22bbd0a640 1 -- --> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] -- mon_getmap magic: 0 -- 0x7f22b40af480 con 0x7f22b4145570 2024-03-28T14:08:54.734 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:54.726+0000 7f22ba2a7640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f22b414b290 0x7f22b41dd300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:54.734 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:54.726+0000 7f22ba2a7640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f22b414b290 0x7f22b41dd300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:38000/0 (socket says 172.21.15.104:38000) 2024-03-28T14:08:54.734 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:54.726+0000 7f22ba2a7640 1 -- 172.21.15.104:0/172633955 learned_addr learned my addr 172.21.15.104:0/172633955 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-03-28T14:08:54.734 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:54.726+0000 7f22b9aa6640 1 --2- 172.21.15.104:0/172633955 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f22a0003f70 0x7f22b41e13d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:54.734 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:54.726+0000 7f22b92a5640 1 --2- 172.21.15.104:0/172633955 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f22b4145570 0x7f22b4148da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:54.734 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:54.726+0000 7f22ba2a7640 1 -- 172.21.15.104:0/172633955 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f22b4145570 msgr2=0x7f22b4148da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:54.734 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:54.726+0000 7f22ba2a7640 1 --2- 172.21.15.104:0/172633955 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f22b4145570 0x7f22b4148da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:54.734 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:54.726+0000 7f22ba2a7640 1 -- 172.21.15.104:0/172633955 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f22a0003f70 msgr2=0x7f22b41e13d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:54.734 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:54.726+0000 7f22ba2a7640 1 --2- 172.21.15.104:0/172633955 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f22a0003f70 0x7f22b41e13d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:54.735 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:54.726+0000 7f22ba2a7640 1 -- 172.21.15.104:0/172633955 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f22b414b770 con 0x7f22b414b290 2024-03-28T14:08:54.735 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:54.726+0000 7f22b9aa6640 1 --2- 172.21.15.104:0/172633955 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f22a0003f70 0x7f22b41e13d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T14:08:54.735 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:54.726+0000 7f22b92a5640 1 --2- 172.21.15.104:0/172633955 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f22b4145570 0x7f22b4148da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T14:08:54.735 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:54.726+0000 7f22ba2a7640 1 --2- 172.21.15.104:0/172633955 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f22b414b290 0x7f22b41dd300 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f22b40b0610 tx=0x7f22ac00f4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:08:54.735 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:54.726+0000 7f22aa7fc640 1 -- 172.21.15.104:0/172633955 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22ac0039a0 con 0x7f22b414b290 2024-03-28T14:08:54.735 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:54.726+0000 7f22aa7fc640 1 -- 172.21.15.104:0/172633955 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f22ac003b40 con 0x7f22b414b290 2024-03-28T14:08:54.735 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:54.726+0000 7f22bbd0a640 1 -- 172.21.15.104:0/172633955 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f22b41e08f0 con 0x7f22b414b290 2024-03-28T14:08:54.735 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:54.726+0000 7f22aa7fc640 1 -- 172.21.15.104:0/172633955 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22ac004020 con 0x7f22b414b290 2024-03-28T14:08:54.735 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:54.726+0000 7f22bbd0a640 1 -- 172.21.15.104:0/172633955 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f22b4147d60 con 0x7f22b414b290 2024-03-28T14:08:54.735 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:54.726+0000 7f22bbd0a640 1 -- 172.21.15.104:0/172633955 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f22b40687c0 con 0x7f22b414b290 2024-03-28T14:08:54.735 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:54.730+0000 7f22aa7fc640 1 -- 172.21.15.104:0/172633955 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) ==== 87803+0+0 (secure 0 0 0) 0x7f22ac0041c0 con 0x7f22b414b290 2024-03-28T14:08:54.735 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:54.730+0000 7f22aa7fc640 1 --2- 172.21.15.104:0/172633955 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f2298067fc0 0x7f229806a480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:54.735 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:54.730+0000 7f22aa7fc640 1 -- 172.21.15.104:0/172633955 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 5709+0+0 (secure 0 0 0) 0x7f22ac086af0 con 0x7f22b414b290 2024-03-28T14:08:54.735 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:54.730+0000 7f22b9aa6640 1 --2- 172.21.15.104:0/172633955 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f2298067fc0 0x7f229806a480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:54.736 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:54.730+0000 7f22b9aa6640 1 --2- 172.21.15.104:0/172633955 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f2298067fc0 0x7f229806a480 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f22b0004880 tx=0x7f22b000c040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:08:54.737 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:54.734+0000 7f22aa7fc640 1 -- 172.21.15.104:0/172633955 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179117 (secure 0 0 0) 0x7f22ac012070 con 0x7f22b414b290 2024-03-28T14:08:54.919 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:54.914+0000 7f22bbd0a640 1 -- 172.21.15.104:0/172633955 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7f22b40b81c0 con 0x7f22b414b290 2024-03-28T14:08:54.920 INFO:tasks.ceph.mon.a.smithi104.stderr:2024-03-28T14:08:54.918+0000 7f55f386d640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-28T14:08:54.922 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:54.918+0000 7f22aa7fc640 1 -- 172.21.15.104:0/172633955 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v15) ==== 161+0+0 (secure 0 0 0) 0x7f22ac0572a0 con 0x7f22b414b290 2024-03-28T14:08:54.922 INFO:teuthology.orchestra.run.smithi104.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-03-28T14:08:54.925 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:54.922+0000 7f22bbd0a640 1 -- 172.21.15.104:0/172633955 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f2298067fc0 msgr2=0x7f229806a480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:54.926 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:54.922+0000 7f22bbd0a640 1 --2- 172.21.15.104:0/172633955 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f2298067fc0 0x7f229806a480 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f22b0004880 tx=0x7f22b000c040 comp rx=0 tx=0).stop 2024-03-28T14:08:54.926 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:54.922+0000 7f22bbd0a640 1 -- 172.21.15.104:0/172633955 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f22b414b290 msgr2=0x7f22b41dd300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:54.926 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:54.922+0000 7f22bbd0a640 1 --2- 172.21.15.104:0/172633955 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f22b414b290 0x7f22b41dd300 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f22b40b0610 tx=0x7f22ac00f4a0 comp rx=0 tx=0).stop 2024-03-28T14:08:54.926 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:54.922+0000 7f22bbd0a640 1 -- 172.21.15.104:0/172633955 shutdown_connections 2024-03-28T14:08:54.926 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:54.922+0000 7f22bbd0a640 1 --2- 172.21.15.104:0/172633955 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f2298067fc0 0x7f229806a480 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:54.926 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:54.922+0000 7f22bbd0a640 1 --2- 172.21.15.104:0/172633955 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f22b414b290 0x7f22b41dd300 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:54.926 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:54.922+0000 7f22bbd0a640 1 --2- 172.21.15.104:0/172633955 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f22b4145570 0x7f22b4148da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:54.926 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:54.922+0000 7f22bbd0a640 1 --2- 172.21.15.104:0/172633955 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f22a0003f70 0x7f22b41e13d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:54.926 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:54.922+0000 7f22bbd0a640 1 -- 172.21.15.104:0/172633955 >> 172.21.15.104:0/172633955 conn(0x7f22b40ba390 msgr2=0x7f22b4067330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T14:08:54.926 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:54.922+0000 7f22bbd0a640 1 -- 172.21.15.104:0/172633955 shutdown_connections 2024-03-28T14:08:54.926 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:54.922+0000 7f22bbd0a640 1 -- 172.21.15.104:0/172633955 wait complete. 2024-03-28T14:08:54.928 INFO:tasks.ceph.mds.a.smithi104.stderr: -2634> 2024-03-28T14:08:44.882+0000 7fd6f06a8640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-28T14:08:54.944 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-03-28T14:08:54.964 INFO:tasks.ceph.mds.a.smithi104.stdout:starting mds.a at 2024-03-28T14:08:55.057 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:55.054+0000 7f5d19c8f640 1 Processor -- start 2024-03-28T14:08:55.057 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:55.054+0000 7f5d19c8f640 1 -- start start 2024-03-28T14:08:55.058 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:55.054+0000 7f5d19c8f640 1 --2- >> v2:172.21.15.171:3300/0 conn(0x7f5d140b63a0 0x7f5d140af9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:55.058 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:55.054+0000 7f5d19c8f640 1 --2- >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f5d140b0010 0x7f5d14141670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:55.058 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:55.054+0000 7f5d19c8f640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5d140b6780 con 0x7f5d14141bb0 2024-03-28T14:08:55.058 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:55.054+0000 7f5d19c8f640 1 -- --> v2:172.21.15.171:3300/0 -- mon_getmap magic: 0 -- 0x7f5d140b04c0 con 0x7f5d140b63a0 2024-03-28T14:08:55.058 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:55.054+0000 7f5d19c8f640 1 -- --> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] -- mon_getmap magic: 0 -- 0x7f5d140af610 con 0x7f5d140b0010 2024-03-28T14:08:55.058 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:55.054+0000 7f5d12ffd640 1 --2- >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f5d140b0010 0x7f5d14141670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:55.058 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:55.054+0000 7f5d137fe640 1 --2- >> v2:172.21.15.171:3300/0 conn(0x7f5d140b63a0 0x7f5d140af9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:55.058 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:55.054+0000 7f5d12ffd640 1 --2- >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f5d140b0010 0x7f5d14141670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.171:3301/0 says I am v2:172.21.15.104:36308/0 (socket says 172.21.15.104:36308) 2024-03-28T14:08:55.058 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:55.054+0000 7f5d12ffd640 1 -- 172.21.15.104:0/3675516254 learned_addr learned my addr 172.21.15.104:0/3675516254 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-03-28T14:08:55.059 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:55.054+0000 7f5d12ffd640 1 -- 172.21.15.104:0/3675516254 >> v2:172.21.15.171:3300/0 conn(0x7f5d140b63a0 msgr2=0x7f5d140af9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T14:08:55.059 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:55.054+0000 7f5d12ffd640 1 --2- 172.21.15.104:0/3675516254 >> v2:172.21.15.171:3300/0 conn(0x7f5d140b63a0 0x7f5d140af9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:55.059 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:55.054+0000 7f5d12ffd640 1 -- 172.21.15.104:0/3675516254 >> v1:172.21.15.171:6789/0 conn(0x7f5d14141bb0 legacy=0x7f5d141445e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:55.059 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:55.054+0000 7f5d12ffd640 1 -- 172.21.15.104:0/3675516254 --> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5d1406dde0 con 0x7f5d140b0010 2024-03-28T14:08:55.059 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:55.054+0000 7f5d137fe640 1 --2- 172.21.15.104:0/3675516254 >> v2:172.21.15.171:3300/0 conn(0x7f5d140b63a0 0x7f5d140af9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T14:08:55.060 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:55.058+0000 7f5d12ffd640 1 --2- 172.21.15.104:0/3675516254 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f5d140b0010 0x7f5d14141670 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f5d08009870 tx=0x7f5d0801c4d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2b5f1bfeb4d202da server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:08:55.061 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:55.058+0000 7f5d127fc640 1 -- 172.21.15.104:0/3675516254 <== mon.2 v2:172.21.15.171:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d080b0070 con 0x7f5d140b0010 2024-03-28T14:08:55.061 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:55.058+0000 7f5d127fc640 1 -- 172.21.15.104:0/3675516254 <== mon.2 v2:172.21.15.171:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5d080ab600 con 0x7f5d140b0010 2024-03-28T14:08:55.061 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:55.058+0000 7f5d127fc640 1 -- 172.21.15.104:0/3675516254 <== mon.2 v2:172.21.15.171:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d080ab9d0 con 0x7f5d140b0010 2024-03-28T14:08:55.062 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:55.058+0000 7f5d19c8f640 1 -- 172.21.15.104:0/3675516254 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f5d140b0010 msgr2=0x7f5d14141670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:55.062 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:55.058+0000 7f5d19c8f640 1 --2- 172.21.15.104:0/3675516254 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f5d140b0010 0x7f5d14141670 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f5d08009870 tx=0x7f5d0801c4d0 comp rx=0 tx=0).stop 2024-03-28T14:08:55.062 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:55.058+0000 7f5d19c8f640 1 -- 172.21.15.104:0/3675516254 shutdown_connections 2024-03-28T14:08:55.062 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:55.058+0000 7f5d19c8f640 1 --2- 172.21.15.104:0/3675516254 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f5d140b0010 0x7f5d14141670 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:55.062 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:55.058+0000 7f5d19c8f640 1 --2- 172.21.15.104:0/3675516254 >> v2:172.21.15.171:3300/0 conn(0x7f5d140b63a0 0x7f5d140af9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:55.062 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:55.058+0000 7f5d19c8f640 1 -- 172.21.15.104:0/3675516254 >> 172.21.15.104:0/3675516254 conn(0x7f5d140ba400 msgr2=0x7f5d140ba810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T14:08:55.063 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:55.058+0000 7f5d19c8f640 1 -- 172.21.15.104:0/3675516254 shutdown_connections 2024-03-28T14:08:55.063 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:55.058+0000 7f5d19c8f640 1 -- 172.21.15.104:0/3675516254 wait complete. 2024-03-28T14:08:55.064 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:55.058+0000 7f5d19c8f640 1 Processor -- start 2024-03-28T14:08:55.065 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:55.058+0000 7f5d19c8f640 1 -- start start 2024-03-28T14:08:55.065 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:55.058+0000 7f5d19c8f640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f5d140b0010 0x7f5d1406bd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:55.065 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:55.058+0000 7f5d19c8f640 1 --2- >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f5d140b63a0 0x7f5d140b2b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:55.065 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:55.058+0000 7f5d19c8f640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5d14141bb0 0x7f5d141e3530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:55.065 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:55.058+0000 7f5d19c8f640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f5d14151930 con 0x7f5d14141bb0 2024-03-28T14:08:55.065 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:55.058+0000 7f5d19c8f640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f5d141517b0 con 0x7f5d140b0010 2024-03-28T14:08:55.065 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:55.058+0000 7f5d19c8f640 1 -- --> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] -- mon_getmap magic: 0 -- 0x7f5d14151ab0 con 0x7f5d140b63a0 2024-03-28T14:08:55.065 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:55.062+0000 7f5d13fff640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5d14141bb0 0x7f5d141e3530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:55.065 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:55.062+0000 7f5d13fff640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5d14141bb0 0x7f5d141e3530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:38018/0 (socket says 172.21.15.104:38018) 2024-03-28T14:08:55.065 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:55.062+0000 7f5d13fff640 1 -- 172.21.15.104:0/1818199795 learned_addr learned my addr 172.21.15.104:0/1818199795 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-03-28T14:08:55.065 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:55.062+0000 7f5d137fe640 1 --2- 172.21.15.104:0/1818199795 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f5d140b0010 0x7f5d1406bd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:55.065 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:55.062+0000 7f5d12ffd640 1 --2- 172.21.15.104:0/1818199795 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f5d140b63a0 0x7f5d140b2b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:55.066 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:55.062+0000 7f5d13fff640 1 -- 172.21.15.104:0/1818199795 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f5d140b63a0 msgr2=0x7f5d140b2b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:55.066 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:55.062+0000 7f5d13fff640 1 --2- 172.21.15.104:0/1818199795 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f5d140b63a0 0x7f5d140b2b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:55.066 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:55.062+0000 7f5d13fff640 1 -- 172.21.15.104:0/1818199795 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f5d140b0010 msgr2=0x7f5d1406bd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:55.066 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:55.062+0000 7f5d13fff640 1 --2- 172.21.15.104:0/1818199795 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f5d140b0010 0x7f5d1406bd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:55.066 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:55.062+0000 7f5d13fff640 1 -- 172.21.15.104:0/1818199795 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5d140b1cd0 con 0x7f5d14141bb0 2024-03-28T14:08:55.066 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:55.062+0000 7f5d13fff640 1 --2- 172.21.15.104:0/1818199795 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5d14141bb0 0x7f5d141e3530 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f5d0c00c730 tx=0x7f5d0c00cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:08:55.066 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:55.062+0000 7f5cfbfff640 1 -- 172.21.15.104:0/1818199795 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d0c003b80 con 0x7f5d14141bb0 2024-03-28T14:08:55.066 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:55.062+0000 7f5cfbfff640 1 -- 172.21.15.104:0/1818199795 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5d0c003d20 con 0x7f5d14141bb0 2024-03-28T14:08:55.066 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:55.062+0000 7f5cfbfff640 1 -- 172.21.15.104:0/1818199795 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d0c00aa30 con 0x7f5d14141bb0 2024-03-28T14:08:55.066 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:55.062+0000 7f5d19c8f640 1 -- 172.21.15.104:0/1818199795 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5d140b1fb0 con 0x7f5d14141bb0 2024-03-28T14:08:55.066 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:55.062+0000 7f5d19c8f640 1 -- 172.21.15.104:0/1818199795 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5d140bab40 con 0x7f5d14141bb0 2024-03-28T14:08:55.067 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:55.062+0000 7f5d19c8f640 1 -- 172.21.15.104:0/1818199795 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5d140687c0 con 0x7f5d14141bb0 2024-03-28T14:08:55.068 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:55.062+0000 7f5cfbfff640 1 -- 172.21.15.104:0/1818199795 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) ==== 87803+0+0 (secure 0 0 0) 0x7f5d0c00abd0 con 0x7f5d14141bb0 2024-03-28T14:08:55.068 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:55.062+0000 7f5cfbfff640 1 --2- 172.21.15.104:0/1818199795 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f5d00067ff0 0x7f5d0006a4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:55.068 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:55.062+0000 7f5cfbfff640 1 -- 172.21.15.104:0/1818199795 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(31..31 src has 1..31) ==== 5795+0+0 (secure 0 0 0) 0x7f5d0c0862e0 con 0x7f5d14141bb0 2024-03-28T14:08:55.068 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:55.062+0000 7f5d137fe640 1 --2- 172.21.15.104:0/1818199795 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f5d00067ff0 0x7f5d0006a4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:55.068 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:55.062+0000 7f5d137fe640 1 --2- 172.21.15.104:0/1818199795 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f5d00067ff0 0x7f5d0006a4b0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f5d04009580 tx=0x7f5d04009190 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:08:55.070 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:55.066+0000 7f5cfbfff640 1 -- 172.21.15.104:0/1818199795 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179117 (secure 0 0 0) 0x7f5d0c056360 con 0x7f5d14141bb0 2024-03-28T14:08:55.267 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:55.262+0000 7f5d19c8f640 1 -- 172.21.15.104:0/1818199795 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) -- 0x7f5d1413f8d0 con 0x7f5d14141bb0 2024-03-28T14:08:55.924 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:55.922+0000 7f5cfbfff640 1 -- 172.21.15.104:0/1818199795 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v16) ==== 104+0+0 (secure 0 0 0) 0x7f5d0c05a660 con 0x7f5d14141bb0 2024-03-28T14:08:55.930 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:55.926+0000 7f5d19c8f640 1 -- 172.21.15.104:0/1818199795 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f5d00067ff0 msgr2=0x7f5d0006a4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:55.930 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:55.926+0000 7f5d19c8f640 1 --2- 172.21.15.104:0/1818199795 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f5d00067ff0 0x7f5d0006a4b0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f5d04009580 tx=0x7f5d04009190 comp rx=0 tx=0).stop 2024-03-28T14:08:55.930 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:55.926+0000 7f5d19c8f640 1 -- 172.21.15.104:0/1818199795 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5d14141bb0 msgr2=0x7f5d141e3530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:55.930 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:55.926+0000 7f5d19c8f640 1 --2- 172.21.15.104:0/1818199795 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5d14141bb0 0x7f5d141e3530 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f5d0c00c730 tx=0x7f5d0c00cbf0 comp rx=0 tx=0).stop 2024-03-28T14:08:55.930 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:55.926+0000 7f5d19c8f640 1 -- 172.21.15.104:0/1818199795 shutdown_connections 2024-03-28T14:08:55.930 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:55.926+0000 7f5d19c8f640 1 --2- 172.21.15.104:0/1818199795 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f5d00067ff0 0x7f5d0006a4b0 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:55.930 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:55.926+0000 7f5d19c8f640 1 --2- 172.21.15.104:0/1818199795 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5d14141bb0 0x7f5d141e3530 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:55.931 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:55.926+0000 7f5d19c8f640 1 --2- 172.21.15.104:0/1818199795 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f5d140b63a0 0x7f5d140b2b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:55.931 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:55.926+0000 7f5d19c8f640 1 --2- 172.21.15.104:0/1818199795 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f5d140b0010 0x7f5d1406bd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:55.931 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:55.926+0000 7f5d19c8f640 1 -- 172.21.15.104:0/1818199795 >> 172.21.15.104:0/1818199795 conn(0x7f5d140ba400 msgr2=0x7f5d1413e470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T14:08:55.931 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:55.926+0000 7f5d19c8f640 1 -- 172.21.15.104:0/1818199795 shutdown_connections 2024-03-28T14:08:55.931 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:55.926+0000 7f5d19c8f640 1 -- 172.21.15.104:0/1818199795 wait complete. 2024-03-28T14:08:55.953 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-03-28T14:08:56.117 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:56.114+0000 7f5e0c0bb640 1 Processor -- start 2024-03-28T14:08:56.117 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:56.114+0000 7f5e0c0bb640 1 -- start start 2024-03-28T14:08:56.120 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:56.114+0000 7f5e0c0bb640 1 --2- >> v2:172.21.15.171:3300/0 conn(0x7f5e040b4e00 0x7f5e040b7a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:56.120 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:56.114+0000 7f5e0c0bb640 1 --2- >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f5e0414b840 0x7f5e040b5c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:56.120 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:56.114+0000 7f5e0c0bb640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5e040b51e0 con 0x7f5e040b61a0 2024-03-28T14:08:56.120 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:56.114+0000 7f5e0c0bb640 1 -- --> v2:172.21.15.171:3300/0 -- mon_getmap magic: 0 -- 0x7f5e0414ad60 con 0x7f5e040b4e00 2024-03-28T14:08:56.120 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:56.114+0000 7f5e0c0bb640 1 -- --> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] -- mon_getmap magic: 0 -- 0x7f5e0414b040 con 0x7f5e0414b840 2024-03-28T14:08:56.120 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:56.114+0000 7f5e0a658640 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f5e040b61a0 0x7f5e04141500 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:54808/0 (socket says 172.21.15.104:54808) 2024-03-28T14:08:56.120 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:56.114+0000 7f5e0a658640 1 -- 172.21.15.104:0/1134556405 learned_addr learned my addr 172.21.15.104:0/1134556405 (peer_addr_for_me v1:172.21.15.104:0/0) 2024-03-28T14:08:56.120 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:56.114+0000 7f5e08e55640 1 -- 172.21.15.104:0/1134556405 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 659133561 0 0) 0x7f5e040b51e0 con 0x7f5e040b61a0 2024-03-28T14:08:56.120 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:56.114+0000 7f5e08e55640 1 -- 172.21.15.104:0/1134556405 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5dec003530 con 0x7f5e040b61a0 2024-03-28T14:08:56.120 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:56.114+0000 7f5e09656640 1 --2- 172.21.15.104:0/1134556405 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f5e0414b840 0x7f5e040b5c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:56.120 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:56.114+0000 7f5e09e57640 1 --2- 172.21.15.104:0/1134556405 >> v2:172.21.15.171:3300/0 conn(0x7f5e040b4e00 0x7f5e040b7a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:56.121 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:56.114+0000 7f5e08e55640 1 -- 172.21.15.104:0/1134556405 <== mon.0 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3835716889 0 0) 0x7f5dec003530 con 0x7f5e040b61a0 2024-03-28T14:08:56.121 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:56.114+0000 7f5e08e55640 1 -- 172.21.15.104:0/1134556405 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f5e0414b840 msgr2=0x7f5e040b5c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T14:08:56.121 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:56.114+0000 7f5e08e55640 1 --2- 172.21.15.104:0/1134556405 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f5e0414b840 0x7f5e040b5c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:56.121 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:56.114+0000 7f5e08e55640 1 -- 172.21.15.104:0/1134556405 >> v2:172.21.15.171:3300/0 conn(0x7f5e040b4e00 msgr2=0x7f5e040b7a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T14:08:56.121 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:56.114+0000 7f5e08e55640 1 --2- 172.21.15.104:0/1134556405 >> v2:172.21.15.171:3300/0 conn(0x7f5e040b4e00 0x7f5e040b7a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:56.121 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:56.114+0000 7f5e08e55640 1 -- 172.21.15.104:0/1134556405 --> v1:172.21.15.104:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5e0414b320 con 0x7f5e040b61a0 2024-03-28T14:08:56.121 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:56.114+0000 7f5e08e55640 1 -- 172.21.15.104:0/1134556405 <== mon.0 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3745580849 0 0) 0x7f5df4001680 con 0x7f5e040b61a0 2024-03-28T14:08:56.121 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:56.114+0000 7f5e08e55640 1 -- 172.21.15.104:0/1134556405 >> v1:172.21.15.104:6789/0 conn(0x7f5e040b61a0 legacy=0x7f5e04141500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:56.121 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:56.114+0000 7f5e08e55640 1 --2- 172.21.15.104:0/1134556405 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f5e0414b840 0x7f5dec0039c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:56.121 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:56.114+0000 7f5e08e55640 1 --2- 172.21.15.104:0/1134556405 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5dec003f40 0x7f5dec004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:56.121 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:56.114+0000 7f5e08e55640 1 --2- 172.21.15.104:0/1134556405 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f5dec004d50 0x7f5dec0051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:56.121 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:56.114+0000 7f5e08e55640 1 -- 172.21.15.104:0/1134556405 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f5e0414ad60 con 0x7f5dec003f40 2024-03-28T14:08:56.122 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:56.114+0000 7f5e08e55640 1 -- 172.21.15.104:0/1134556405 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f5e0414b040 con 0x7f5dec004d50 2024-03-28T14:08:56.122 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:56.114+0000 7f5e08e55640 1 -- 172.21.15.104:0/1134556405 --> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] -- mon_getmap magic: 0 -- 0x7f5dec00ba20 con 0x7f5e0414b840 2024-03-28T14:08:56.122 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:56.114+0000 7f5e09656640 1 --2- 172.21.15.104:0/1134556405 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5dec003f40 0x7f5dec004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:56.122 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:56.114+0000 7f5e09e57640 1 --2- 172.21.15.104:0/1134556405 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f5e0414b840 0x7f5dec0039c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:56.122 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:56.114+0000 7f5e0a658640 1 --2- 172.21.15.104:0/1134556405 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f5dec004d50 0x7f5dec0051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:56.122 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:56.114+0000 7f5e09656640 1 -- 172.21.15.104:0/1134556405 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f5e0414b840 msgr2=0x7f5dec0039c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T14:08:56.122 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:56.114+0000 7f5e09656640 1 --2- 172.21.15.104:0/1134556405 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f5e0414b840 0x7f5dec0039c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:56.122 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:56.114+0000 7f5e09656640 1 -- 172.21.15.104:0/1134556405 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f5dec004d50 msgr2=0x7f5dec0051d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T14:08:56.122 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:56.114+0000 7f5e09656640 1 --2- 172.21.15.104:0/1134556405 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f5dec004d50 0x7f5dec0051d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:56.122 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:56.114+0000 7f5e09656640 1 -- 172.21.15.104:0/1134556405 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5dec003530 con 0x7f5dec003f40 2024-03-28T14:08:56.125 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:56.118+0000 7f5e09656640 1 --2- 172.21.15.104:0/1134556405 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5dec003f40 0x7f5dec004810 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f5df8009af0 tx=0x7f5df801ca30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b7921d9855646112 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:08:56.125 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:56.118+0000 7f5e08e55640 1 -- 172.21.15.104:0/1134556405 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5df80b2070 con 0x7f5dec003f40 2024-03-28T14:08:56.125 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:56.118+0000 7f5e08e55640 1 -- 172.21.15.104:0/1134556405 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5df80a5510 con 0x7f5dec003f40 2024-03-28T14:08:56.126 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:56.118+0000 7f5e08e55640 1 -- 172.21.15.104:0/1134556405 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5df8009690 con 0x7f5dec003f40 2024-03-28T14:08:56.126 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:56.118+0000 7f5e0c0bb640 1 -- 172.21.15.104:0/1134556405 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5dec003f40 msgr2=0x7f5dec004810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:56.126 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:56.118+0000 7f5e0c0bb640 1 --2- 172.21.15.104:0/1134556405 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5dec003f40 0x7f5dec004810 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f5df8009af0 tx=0x7f5df801ca30 comp rx=0 tx=0).stop 2024-03-28T14:08:56.126 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:56.118+0000 7f5e0a658640 1 -- 172.21.15.104:0/1134556405 reap_dead start 2024-03-28T14:08:56.126 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:56.118+0000 7f5e0c0bb640 1 -- 172.21.15.104:0/1134556405 shutdown_connections 2024-03-28T14:08:56.126 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:56.118+0000 7f5e0c0bb640 1 -- 172.21.15.104:0/1134556405 >> 172.21.15.104:0/1134556405 conn(0x7f5e040b9b40 msgr2=0x7f5e040b9f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T14:08:56.126 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:56.118+0000 7f5e0c0bb640 1 -- 172.21.15.104:0/1134556405 shutdown_connections 2024-03-28T14:08:56.126 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:56.118+0000 7f5e0c0bb640 1 -- 172.21.15.104:0/1134556405 wait complete. 2024-03-28T14:08:56.126 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:56.118+0000 7f5e0c0bb640 1 Processor -- start 2024-03-28T14:08:56.126 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:56.118+0000 7f5e0c0bb640 1 -- start start 2024-03-28T14:08:56.126 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:56.118+0000 7f5e0c0bb640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f5e0413bb90 0x7f5e041e46f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:56.126 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:56.118+0000 7f5e0c0bb640 1 --2- >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f5e04131800 0x7f5e0412fe70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:56.127 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:56.118+0000 7f5e0c0bb640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5e0414d850 0x7f5e041303b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:56.127 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:56.118+0000 7f5e0c0bb640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f5e04152a70 con 0x7f5e0414d850 2024-03-28T14:08:56.127 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:56.118+0000 7f5e0c0bb640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f5e0414bc20 con 0x7f5e0413bb90 2024-03-28T14:08:56.127 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:56.118+0000 7f5e0c0bb640 1 -- --> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] -- mon_getmap magic: 0 -- 0x7f5e04152bf0 con 0x7f5e04131800 2024-03-28T14:08:56.127 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:56.118+0000 7f5e0a658640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5e0414d850 0x7f5e041303b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:56.127 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:56.118+0000 7f5e0a658640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5e0414d850 0x7f5e041303b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:38032/0 (socket says 172.21.15.104:38032) 2024-03-28T14:08:56.127 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:56.118+0000 7f5e0a658640 1 -- 172.21.15.104:0/3563161460 learned_addr learned my addr 172.21.15.104:0/3563161460 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-03-28T14:08:56.127 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:56.122+0000 7f5e09e57640 1 --2- 172.21.15.104:0/3563161460 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f5e0413bb90 0x7f5e041e46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:56.127 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:56.122+0000 7f5e0a658640 1 -- 172.21.15.104:0/3563161460 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f5e04131800 msgr2=0x7f5e0412fe70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:56.127 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:56.122+0000 7f5e0a658640 1 --2- 172.21.15.104:0/3563161460 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f5e04131800 0x7f5e0412fe70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:56.127 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:56.122+0000 7f5e0a658640 1 -- 172.21.15.104:0/3563161460 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f5e0413bb90 msgr2=0x7f5e041e46f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:56.127 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:56.122+0000 7f5e0a658640 1 --2- 172.21.15.104:0/3563161460 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f5e0413bb90 0x7f5e041e46f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:56.127 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:56.122+0000 7f5e0a658640 1 -- 172.21.15.104:0/3563161460 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5e041e3f60 con 0x7f5e0414d850 2024-03-28T14:08:56.127 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:56.122+0000 7f5e0a658640 1 --2- 172.21.15.104:0/3563161460 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5e0414d850 0x7f5e041303b0 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f5df4001ec0 tx=0x7f5df400eb90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:08:56.128 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:56.122+0000 7f5e027fc640 1 -- 172.21.15.104:0/3563161460 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5df4019070 con 0x7f5e0414d850 2024-03-28T14:08:56.128 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:56.122+0000 7f5e027fc640 1 -- 172.21.15.104:0/3563161460 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5df400c690 con 0x7f5e0414d850 2024-03-28T14:08:56.128 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:56.122+0000 7f5e0c0bb640 1 -- 172.21.15.104:0/3563161460 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5e0414dda0 con 0x7f5e0414d850 2024-03-28T14:08:56.128 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:56.122+0000 7f5e027fc640 1 -- 172.21.15.104:0/3563161460 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5df400ca80 con 0x7f5e0414d850 2024-03-28T14:08:56.128 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:56.122+0000 7f5e0c0bb640 1 -- 172.21.15.104:0/3563161460 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5e041e41d0 con 0x7f5e0414d850 2024-03-28T14:08:56.128 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:56.122+0000 7f5e0c0bb640 1 -- 172.21.15.104:0/3563161460 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5e0402cc50 con 0x7f5e0414d850 2024-03-28T14:08:56.128 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:56.122+0000 7f5e027fc640 1 -- 172.21.15.104:0/3563161460 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) ==== 87803+0+0 (secure 0 0 0) 0x7f5df401c070 con 0x7f5e0414d850 2024-03-28T14:08:56.129 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:56.122+0000 7f5e027fc640 1 --2- 172.21.15.104:0/3563161460 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f5df0067fc0 0x7f5df006a480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:56.129 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:56.122+0000 7f5e027fc640 1 -- 172.21.15.104:0/3563161460 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(31..31 src has 1..31) ==== 5795+0+0 (secure 0 0 0) 0x7f5df4085a40 con 0x7f5e0414d850 2024-03-28T14:08:56.129 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:56.122+0000 7f5e09e57640 1 --2- 172.21.15.104:0/3563161460 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f5df0067fc0 0x7f5df006a480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:56.129 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:56.122+0000 7f5e09e57640 1 --2- 172.21.15.104:0/3563161460 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f5df0067fc0 0x7f5df006a480 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f5dfc007520 tx=0x7f5dfc00b040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:08:56.130 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:56.126+0000 7f5e027fc640 1 -- 172.21.15.104:0/3563161460 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179117 (secure 0 0 0) 0x7f5df4056180 con 0x7f5e0414d850 2024-03-28T14:08:56.295 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:56.290+0000 7f5e0c0bb640 1 -- 172.21.15.104:0/3563161460 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) -- 0x7f5e040687c0 con 0x7f5e0414d850 2024-03-28T14:08:56.935 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:56.930+0000 7f5e027fc640 1 -- 172.21.15.104:0/3563161460 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v32) ==== 181+0+0 (secure 0 0 0) 0x7f5df4055a80 con 0x7f5e0414d850 2024-03-28T14:08:56.935 INFO:teuthology.orchestra.run.smithi104.stderr:pool 'cephfs_metadata' removed 2024-03-28T14:08:56.938 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:56.934+0000 7f5e0c0bb640 1 -- 172.21.15.104:0/3563161460 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f5df0067fc0 msgr2=0x7f5df006a480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:56.938 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:56.934+0000 7f5e0c0bb640 1 --2- 172.21.15.104:0/3563161460 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f5df0067fc0 0x7f5df006a480 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f5dfc007520 tx=0x7f5dfc00b040 comp rx=0 tx=0).stop 2024-03-28T14:08:56.938 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:56.934+0000 7f5e0c0bb640 1 -- 172.21.15.104:0/3563161460 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5e0414d850 msgr2=0x7f5e041303b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:56.938 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:56.934+0000 7f5e0c0bb640 1 --2- 172.21.15.104:0/3563161460 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5e0414d850 0x7f5e041303b0 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f5df4001ec0 tx=0x7f5df400eb90 comp rx=0 tx=0).stop 2024-03-28T14:08:56.938 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:56.934+0000 7f5e0c0bb640 1 -- 172.21.15.104:0/3563161460 shutdown_connections 2024-03-28T14:08:56.938 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:56.934+0000 7f5e0c0bb640 1 --2- 172.21.15.104:0/3563161460 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f5df0067fc0 0x7f5df006a480 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:56.938 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:56.934+0000 7f5e0c0bb640 1 --2- 172.21.15.104:0/3563161460 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5e0414d850 0x7f5e041303b0 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:56.938 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:56.934+0000 7f5e0c0bb640 1 --2- 172.21.15.104:0/3563161460 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f5e04131800 0x7f5e0412fe70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:56.939 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:56.934+0000 7f5e0c0bb640 1 --2- 172.21.15.104:0/3563161460 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f5e0413bb90 0x7f5e041e46f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:56.939 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:56.934+0000 7f5e0c0bb640 1 -- 172.21.15.104:0/3563161460 >> 172.21.15.104:0/3563161460 conn(0x7f5e040b9b40 msgr2=0x7f5e04148b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T14:08:56.939 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:56.934+0000 7f5e0c0bb640 1 -- 172.21.15.104:0/3563161460 shutdown_connections 2024-03-28T14:08:56.939 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:56.934+0000 7f5e0c0bb640 1 -- 172.21.15.104:0/3563161460 wait complete. 2024-03-28T14:08:56.964 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-03-28T14:08:57.091 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:57.086+0000 7f4f6e9cf640 1 Processor -- start 2024-03-28T14:08:57.091 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:57.086+0000 7f4f6e9cf640 1 -- start start 2024-03-28T14:08:57.092 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:57.086+0000 7f4f6e9cf640 1 --2- >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f4f6806c8f0 0x7f4f680b2740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:57.092 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:57.086+0000 7f4f6e9cf640 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f4f680b3730 0x7f4f68148c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:57.093 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:57.086+0000 7f4f6e9cf640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4f680b3b30 con 0x7f4f681491b0 2024-03-28T14:08:57.093 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:57.086+0000 7f4f6e9cf640 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 -- 0x7f4f680b0b30 con 0x7f4f680b3730 2024-03-28T14:08:57.093 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:57.086+0000 7f4f6e9cf640 1 -- --> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] -- mon_getmap magic: 0 -- 0x7f4f680b22b0 con 0x7f4f6806c8f0 2024-03-28T14:08:57.093 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:57.086+0000 7f4f677fe640 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f4f680b3730 0x7f4f68148c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:57.093 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:57.086+0000 7f4f6cf6c640 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f4f681491b0 0x7f4f68147b80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:40494/0 (socket says 172.21.15.104:40494) 2024-03-28T14:08:57.093 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:57.086+0000 7f4f6cf6c640 1 -- 172.21.15.104:0/2462625923 learned_addr learned my addr 172.21.15.104:0/2462625923 (peer_addr_for_me v1:172.21.15.104:0/0) 2024-03-28T14:08:57.093 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:57.090+0000 7f4f67fff640 1 --2- 172.21.15.104:0/2462625923 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f4f6806c8f0 0x7f4f680b2740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:57.093 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:57.090+0000 7f4f677fe640 1 -- 172.21.15.104:0/2462625923 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f4f6806c8f0 msgr2=0x7f4f680b2740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T14:08:57.094 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:57.090+0000 7f4f677fe640 1 --2- 172.21.15.104:0/2462625923 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f4f6806c8f0 0x7f4f680b2740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:57.094 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:57.090+0000 7f4f677fe640 1 -- 172.21.15.104:0/2462625923 >> v1:172.21.15.104:6789/0 conn(0x7f4f681491b0 legacy=0x7f4f68147b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:57.094 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:57.090+0000 7f4f677fe640 1 -- 172.21.15.104:0/2462625923 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4f6806ccd0 con 0x7f4f680b3730 2024-03-28T14:08:57.095 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:57.090+0000 7f4f677fe640 1 --2- 172.21.15.104:0/2462625923 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4f680b3730 0x7f4f68148c70 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f4f5c0097c0 tx=0x7f4f5c01c3e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9a3dc886b37b75c8 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:08:57.095 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:57.090+0000 7f4f66ffd640 1 -- 172.21.15.104:0/2462625923 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f5c0b1070 con 0x7f4f680b3730 2024-03-28T14:08:57.096 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:57.090+0000 7f4f66ffd640 1 -- 172.21.15.104:0/2462625923 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4f5c0a8050 con 0x7f4f680b3730 2024-03-28T14:08:57.096 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:57.090+0000 7f4f66ffd640 1 -- 172.21.15.104:0/2462625923 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f5c0ac860 con 0x7f4f680b3730 2024-03-28T14:08:57.096 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:57.090+0000 7f4f6e9cf640 1 -- 172.21.15.104:0/2462625923 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4f680b3730 msgr2=0x7f4f68148c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:57.096 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:57.090+0000 7f4f6e9cf640 1 --2- 172.21.15.104:0/2462625923 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4f680b3730 0x7f4f68148c70 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f4f5c0097c0 tx=0x7f4f5c01c3e0 comp rx=0 tx=0).stop 2024-03-28T14:08:57.096 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:57.090+0000 7f4f6e9cf640 1 -- 172.21.15.104:0/2462625923 shutdown_connections 2024-03-28T14:08:57.096 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:57.090+0000 7f4f6e9cf640 1 --2- 172.21.15.104:0/2462625923 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4f680b3730 0x7f4f68148c70 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:57.096 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:57.090+0000 7f4f6e9cf640 1 --2- 172.21.15.104:0/2462625923 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f4f6806c8f0 0x7f4f680b2740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:57.096 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:57.090+0000 7f4f6e9cf640 1 -- 172.21.15.104:0/2462625923 >> 172.21.15.104:0/2462625923 conn(0x7f4f680b8f00 msgr2=0x7f4f680b9310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T14:08:57.096 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:57.090+0000 7f4f6e9cf640 1 -- 172.21.15.104:0/2462625923 shutdown_connections 2024-03-28T14:08:57.096 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:57.090+0000 7f4f6e9cf640 1 -- 172.21.15.104:0/2462625923 wait complete. 2024-03-28T14:08:57.098 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:57.094+0000 7f4f6e9cf640 1 Processor -- start 2024-03-28T14:08:57.098 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:57.094+0000 7f4f6e9cf640 1 -- start start 2024-03-28T14:08:57.099 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:57.094+0000 7f4f6e9cf640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4f6806c8f0 0x7f4f6814e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:57.099 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:57.094+0000 7f4f6e9cf640 1 --2- >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f4f680b3730 0x7f4f681e3640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:57.099 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:57.094+0000 7f4f6e9cf640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f4f681491b0 0x7f4f681e3b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:57.099 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:57.094+0000 7f4f6e9cf640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f4f680b14e0 con 0x7f4f6806c8f0 2024-03-28T14:08:57.099 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:57.094+0000 7f4f6e9cf640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f4f681499d0 con 0x7f4f681491b0 2024-03-28T14:08:57.099 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:57.094+0000 7f4f6e9cf640 1 -- --> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] -- mon_getmap magic: 0 -- 0x7f4f680b1660 con 0x7f4f680b3730 2024-03-28T14:08:57.099 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:57.094+0000 7f4f67fff640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4f6806c8f0 0x7f4f6814e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:57.099 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:57.094+0000 7f4f67fff640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4f6806c8f0 0x7f4f6814e2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:49738/0 (socket says 172.21.15.104:49738) 2024-03-28T14:08:57.099 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:57.094+0000 7f4f67fff640 1 -- 172.21.15.104:0/3709649417 learned_addr learned my addr 172.21.15.104:0/3709649417 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-03-28T14:08:57.099 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:57.094+0000 7f4f67fff640 1 -- 172.21.15.104:0/3709649417 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f4f680b3730 msgr2=0x7f4f681e3640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:57.099 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:57.094+0000 7f4f67fff640 1 --2- 172.21.15.104:0/3709649417 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f4f680b3730 0x7f4f681e3640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:57.099 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:57.094+0000 7f4f67fff640 1 -- 172.21.15.104:0/3709649417 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f4f681491b0 msgr2=0x7f4f681e3b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:57.099 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:57.094+0000 7f4f67fff640 1 --2- 172.21.15.104:0/3709649417 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f4f681491b0 0x7f4f681e3b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:57.100 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:57.094+0000 7f4f67fff640 1 -- 172.21.15.104:0/3709649417 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4f6814fa30 con 0x7f4f6806c8f0 2024-03-28T14:08:57.100 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:57.094+0000 7f4f67fff640 1 --2- 172.21.15.104:0/3709649417 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4f6806c8f0 0x7f4f6814e2f0 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f4f5800d5f0 tx=0x7f4f5800dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:08:57.100 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:57.094+0000 7f4f64ff9640 1 -- 172.21.15.104:0/3709649417 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f580125f0 con 0x7f4f6806c8f0 2024-03-28T14:08:57.100 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:57.094+0000 7f4f64ff9640 1 -- 172.21.15.104:0/3709649417 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4f58019070 con 0x7f4f6806c8f0 2024-03-28T14:08:57.100 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:57.094+0000 7f4f6e9cf640 1 -- 172.21.15.104:0/3709649417 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4f6814f750 con 0x7f4f6806c8f0 2024-03-28T14:08:57.100 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:57.094+0000 7f4f64ff9640 1 -- 172.21.15.104:0/3709649417 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f580129b0 con 0x7f4f6806c8f0 2024-03-28T14:08:57.100 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:57.094+0000 7f4f6e9cf640 1 -- 172.21.15.104:0/3709649417 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4f681503d0 con 0x7f4f6806c8f0 2024-03-28T14:08:57.100 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:57.094+0000 7f4f6e9cf640 1 -- 172.21.15.104:0/3709649417 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4f680687c0 con 0x7f4f6806c8f0 2024-03-28T14:08:57.101 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:57.094+0000 7f4f64ff9640 1 -- 172.21.15.104:0/3709649417 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) ==== 87803+0+0 (secure 0 0 0) 0x7f4f58012b50 con 0x7f4f6806c8f0 2024-03-28T14:08:57.101 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:57.094+0000 7f4f64ff9640 1 --2- 172.21.15.104:0/3709649417 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f4f54067fc0 0x7f4f5406a480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:57.101 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:57.094+0000 7f4f64ff9640 1 -- 172.21.15.104:0/3709649417 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(32..32 src has 1..32) ==== 5357+0+0 (secure 0 0 0) 0x7f4f58086750 con 0x7f4f6806c8f0 2024-03-28T14:08:57.101 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:57.098+0000 7f4f677fe640 1 --2- 172.21.15.104:0/3709649417 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f4f54067fc0 0x7f4f5406a480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:57.101 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:57.098+0000 7f4f677fe640 1 --2- 172.21.15.104:0/3709649417 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f4f54067fc0 0x7f4f5406a480 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f4f5c0ae000 tx=0x7f4f5c004740 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:08:57.104 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:57.102+0000 7f4f64ff9640 1 -- 172.21.15.104:0/3709649417 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179117 (secure 0 0 0) 0x7f4f6814fa30 con 0x7f4f6806c8f0 2024-03-28T14:08:57.267 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:57.262+0000 7f4f6e9cf640 1 -- 172.21.15.104:0/3709649417 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) -- 0x7f4f68144930 con 0x7f4f6806c8f0 2024-03-28T14:08:57.944 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:57.938+0000 7f4f64ff9640 1 -- 172.21.15.104:0/3709649417 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v33) ==== 169+0+0 (secure 0 0 0) 0x7f4f580570f0 con 0x7f4f6806c8f0 2024-03-28T14:08:57.944 INFO:teuthology.orchestra.run.smithi104.stderr:pool 'cephfs_data' removed 2024-03-28T14:08:57.948 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:57.942+0000 7f4f6e9cf640 1 -- 172.21.15.104:0/3709649417 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f4f54067fc0 msgr2=0x7f4f5406a480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:57.948 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:57.942+0000 7f4f6e9cf640 1 --2- 172.21.15.104:0/3709649417 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f4f54067fc0 0x7f4f5406a480 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f4f5c0ae000 tx=0x7f4f5c004740 comp rx=0 tx=0).stop 2024-03-28T14:08:57.948 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:57.942+0000 7f4f6e9cf640 1 -- 172.21.15.104:0/3709649417 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4f6806c8f0 msgr2=0x7f4f6814e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:57.948 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:57.942+0000 7f4f6e9cf640 1 --2- 172.21.15.104:0/3709649417 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4f6806c8f0 0x7f4f6814e2f0 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f4f5800d5f0 tx=0x7f4f5800dab0 comp rx=0 tx=0).stop 2024-03-28T14:08:57.948 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:57.942+0000 7f4f6e9cf640 1 -- 172.21.15.104:0/3709649417 shutdown_connections 2024-03-28T14:08:57.948 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:57.942+0000 7f4f6e9cf640 1 --2- 172.21.15.104:0/3709649417 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f4f681491b0 0x7f4f681e3b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:57.948 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:57.942+0000 7f4f6e9cf640 1 --2- 172.21.15.104:0/3709649417 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f4f680b3730 0x7f4f681e3640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:57.949 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:57.942+0000 7f4f6e9cf640 1 --2- 172.21.15.104:0/3709649417 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f4f54067fc0 0x7f4f5406a480 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:57.949 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:57.942+0000 7f4f6e9cf640 1 --2- 172.21.15.104:0/3709649417 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4f6806c8f0 0x7f4f6814e2f0 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:57.949 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:57.942+0000 7f4f6e9cf640 1 -- 172.21.15.104:0/3709649417 >> 172.21.15.104:0/3709649417 conn(0x7f4f680b8f00 msgr2=0x7f4f680b6b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T14:08:57.949 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:57.942+0000 7f4f6e9cf640 1 -- 172.21.15.104:0/3709649417 shutdown_connections 2024-03-28T14:08:57.949 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:57.942+0000 7f4f6e9cf640 1 -- 172.21.15.104:0/3709649417 wait complete. 2024-03-28T14:08:57.972 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2024-03-28T14:08:58.094 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:58.090+0000 7fedbf66f640 1 Processor -- start 2024-03-28T14:08:58.095 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:58.090+0000 7fedbf66f640 1 -- start start 2024-03-28T14:08:58.096 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:58.090+0000 7fedbf66f640 1 --2- >> v2:172.21.15.171:3300/0 conn(0x7fedb80b4aa0 0x7fedb80af660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:58.096 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:58.090+0000 7fedbf66f640 1 --2- >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fedb80b29e0 0x7fedb814ef20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:58.096 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:58.090+0000 7fedbf66f640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fedb8148030 con 0x7fedb8147bf0 2024-03-28T14:08:58.096 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:58.090+0000 7fedbf66f640 1 -- --> v2:172.21.15.171:3300/0 -- mon_getmap magic: 0 -- 0x7fedb80b4e80 con 0x7fedb80b4aa0 2024-03-28T14:08:58.096 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:58.090+0000 7fedbf66f640 1 -- --> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] -- mon_getmap magic: 0 -- 0x7fedb80b0890 con 0x7fedb80b29e0 2024-03-28T14:08:58.096 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:58.090+0000 7fedbdc0c640 1 --2- >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fedb80b29e0 0x7fedb814ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:58.096 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:58.090+0000 7fedbd40b640 1 --2- >> v2:172.21.15.171:3300/0 conn(0x7fedb80b4aa0 0x7fedb80af660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:58.096 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:58.090+0000 7fedbdc0c640 1 --2- >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fedb80b29e0 0x7fedb814ef20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.171:3301/0 says I am v2:172.21.15.104:60644/0 (socket says 172.21.15.104:60644) 2024-03-28T14:08:58.096 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:58.090+0000 7fedbdc0c640 1 -- 172.21.15.104:0/46558137 learned_addr learned my addr 172.21.15.104:0/46558137 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-03-28T14:08:58.097 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:58.094+0000 7fedbd40b640 1 -- 172.21.15.104:0/46558137 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fedb80b29e0 msgr2=0x7fedb814ef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T14:08:58.097 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:58.094+0000 7fedbd40b640 1 --2- 172.21.15.104:0/46558137 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fedb80b29e0 0x7fedb814ef20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:58.097 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:58.094+0000 7fedbd40b640 1 -- 172.21.15.104:0/46558137 >> v1:172.21.15.171:6789/0 conn(0x7fedb8147bf0 legacy=0x7fedb80b22d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:58.097 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:58.094+0000 7fedbd40b640 1 -- 172.21.15.104:0/46558137 --> v2:172.21.15.171:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fedb80ae0a0 con 0x7fedb80b4aa0 2024-03-28T14:08:58.098 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:58.094+0000 7fedbd40b640 1 --2- 172.21.15.104:0/46558137 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fedb80b4aa0 0x7fedb80af660 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fedb40097c0 tx=0x7fedb401c410 comp rx=0 tx=0).ready entity=mon.1 client_cookie=eca4eb34cc0b9f8c server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:08:58.099 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:58.094+0000 7fedaffff640 1 -- 172.21.15.104:0/46558137 <== mon.1 v2:172.21.15.171:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fedb40b0070 con 0x7fedb80b4aa0 2024-03-28T14:08:58.099 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:58.094+0000 7fedaffff640 1 -- 172.21.15.104:0/46558137 <== mon.1 v2:172.21.15.171:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fedb40ab600 con 0x7fedb80b4aa0 2024-03-28T14:08:58.099 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:58.094+0000 7fedaffff640 1 -- 172.21.15.104:0/46558137 <== mon.1 v2:172.21.15.171:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fedb40ab9d0 con 0x7fedb80b4aa0 2024-03-28T14:08:58.100 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:58.094+0000 7fedbf66f640 1 -- 172.21.15.104:0/46558137 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fedb80b4aa0 msgr2=0x7fedb80af660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:58.100 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:58.094+0000 7fedbf66f640 1 --2- 172.21.15.104:0/46558137 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fedb80b4aa0 0x7fedb80af660 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fedb40097c0 tx=0x7fedb401c410 comp rx=0 tx=0).stop 2024-03-28T14:08:58.100 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:58.094+0000 7fedbf66f640 1 -- 172.21.15.104:0/46558137 shutdown_connections 2024-03-28T14:08:58.100 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:58.094+0000 7fedbf66f640 1 --2- 172.21.15.104:0/46558137 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fedb80b29e0 0x7fedb814ef20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:58.100 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:58.094+0000 7fedbf66f640 1 --2- 172.21.15.104:0/46558137 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fedb80b4aa0 0x7fedb80af660 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:58.100 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:58.094+0000 7fedbf66f640 1 -- 172.21.15.104:0/46558137 >> 172.21.15.104:0/46558137 conn(0x7fedb80b8f00 msgr2=0x7fedb80b9310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T14:08:58.101 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:58.094+0000 7fedbf66f640 1 -- 172.21.15.104:0/46558137 shutdown_connections 2024-03-28T14:08:58.101 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:58.094+0000 7fedbf66f640 1 -- 172.21.15.104:0/46558137 wait complete. 2024-03-28T14:08:58.103 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:58.098+0000 7fedbf66f640 1 Processor -- start 2024-03-28T14:08:58.104 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:58.098+0000 7fedbf66f640 1 -- start start 2024-03-28T14:08:58.104 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:58.098+0000 7fedbf66f640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fedb80b29e0 0x7fedb81437a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:58.104 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:58.098+0000 7fedbf66f640 1 --2- >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fedb80b4aa0 0x7fedb8145650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:58.104 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:58.098+0000 7fedbf66f640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fedb8147bf0 0x7fedb813cf90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:58.104 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:58.098+0000 7fedbf66f640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7fedb806dc40 con 0x7fedb8147bf0 2024-03-28T14:08:58.104 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:58.098+0000 7fedbf66f640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7fedb80ae9a0 con 0x7fedb80b29e0 2024-03-28T14:08:58.104 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:58.098+0000 7fedbf66f640 1 -- --> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] -- mon_getmap magic: 0 -- 0x7fedb8072810 con 0x7fedb80b4aa0 2024-03-28T14:08:58.104 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:58.098+0000 7fedbdc0c640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fedb8147bf0 0x7fedb813cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:58.104 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:58.098+0000 7fedbdc0c640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fedb8147bf0 0x7fedb813cf90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:49744/0 (socket says 172.21.15.104:49744) 2024-03-28T14:08:58.105 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:58.098+0000 7fedbdc0c640 1 -- 172.21.15.104:0/1323706575 learned_addr learned my addr 172.21.15.104:0/1323706575 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-03-28T14:08:58.105 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:58.098+0000 7fedbcc0a640 1 --2- 172.21.15.104:0/1323706575 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fedb80b4aa0 0x7fedb8145650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:58.105 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:58.098+0000 7fedbd40b640 1 --2- 172.21.15.104:0/1323706575 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fedb80b29e0 0x7fedb81437a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:58.105 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:58.098+0000 7fedbdc0c640 1 -- 172.21.15.104:0/1323706575 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fedb80b4aa0 msgr2=0x7fedb8145650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:58.105 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:58.098+0000 7fedbdc0c640 1 --2- 172.21.15.104:0/1323706575 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fedb80b4aa0 0x7fedb8145650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:58.105 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:58.098+0000 7fedbdc0c640 1 -- 172.21.15.104:0/1323706575 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fedb80b29e0 msgr2=0x7fedb81437a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:58.105 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:58.098+0000 7fedbdc0c640 1 --2- 172.21.15.104:0/1323706575 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fedb80b29e0 0x7fedb81437a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:58.105 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:58.098+0000 7fedbdc0c640 1 -- 172.21.15.104:0/1323706575 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fedb8146160 con 0x7fedb8147bf0 2024-03-28T14:08:58.105 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:58.098+0000 7fedbd40b640 1 --2- 172.21.15.104:0/1323706575 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fedb80b29e0 0x7fedb81437a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T14:08:58.105 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:58.098+0000 7fedbdc0c640 1 --2- 172.21.15.104:0/1323706575 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fedb8147bf0 0x7fedb813cf90 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7fedb000efc0 tx=0x7fedb000c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:08:58.106 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:58.098+0000 7fedadffb640 1 -- 172.21.15.104:0/1323706575 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fedb000cd20 con 0x7fedb8147bf0 2024-03-28T14:08:58.106 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:58.098+0000 7fedadffb640 1 -- 172.21.15.104:0/1323706575 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fedb00039a0 con 0x7fedb8147bf0 2024-03-28T14:08:58.106 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:58.098+0000 7fedbf66f640 1 -- 172.21.15.104:0/1323706575 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fedb81447d0 con 0x7fedb8147bf0 2024-03-28T14:08:58.106 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:58.098+0000 7fedadffb640 1 -- 172.21.15.104:0/1323706575 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fedb0003d90 con 0x7fedb8147bf0 2024-03-28T14:08:58.106 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:58.098+0000 7fedbf66f640 1 -- 172.21.15.104:0/1323706575 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fedb8143eb0 con 0x7fedb8147bf0 2024-03-28T14:08:58.106 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:58.098+0000 7fedbf66f640 1 -- 172.21.15.104:0/1323706575 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fedb8145200 con 0x7fedb8147bf0 2024-03-28T14:08:58.106 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:58.098+0000 7fedadffb640 1 -- 172.21.15.104:0/1323706575 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) ==== 87803+0+0 (secure 0 0 0) 0x7fedb0012050 con 0x7fedb8147bf0 2024-03-28T14:08:58.107 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:58.102+0000 7fedadffb640 1 --2- 172.21.15.104:0/1323706575 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7fed9c067fc0 0x7fed9c06a480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:58.107 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:58.102+0000 7fedadffb640 1 -- 172.21.15.104:0/1323706575 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(33..33 src has 1..33) ==== 4943+0+0 (secure 0 0 0) 0x7fedb0086660 con 0x7fedb8147bf0 2024-03-28T14:08:58.107 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:58.102+0000 7fedbd40b640 1 --2- 172.21.15.104:0/1323706575 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7fed9c067fc0 0x7fed9c06a480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:58.107 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:58.102+0000 7fedbd40b640 1 --2- 172.21.15.104:0/1323706575 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7fed9c067fc0 0x7fed9c06a480 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fedb40ad000 tx=0x7fedb4008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:08:58.109 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:58.106+0000 7fedadffb640 1 -- 172.21.15.104:0/1323706575 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179117 (secure 0 0 0) 0x7fedb8145200 con 0x7fedb8147bf0 2024-03-28T14:08:58.270 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:58.266+0000 7fedbf66f640 1 -- 172.21.15.104:0/1323706575 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) -- 0x7fedb80687c0 con 0x7fedb8147bf0 2024-03-28T14:08:58.276 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:58.274+0000 7fedadffb640 1 -- 172.21.15.104:0/1323706575 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v11) ==== 104+0+0 (secure 0 0 0) 0x7fedb0022020 con 0x7fedb8147bf0 2024-03-28T14:08:58.278 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:58.274+0000 7fedbf66f640 1 -- 172.21.15.104:0/1323706575 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7fed9c067fc0 msgr2=0x7fed9c06a480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:58.279 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:58.274+0000 7fedbf66f640 1 --2- 172.21.15.104:0/1323706575 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7fed9c067fc0 0x7fed9c06a480 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fedb40ad000 tx=0x7fedb4008040 comp rx=0 tx=0).stop 2024-03-28T14:08:58.279 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:58.274+0000 7fedbf66f640 1 -- 172.21.15.104:0/1323706575 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fedb8147bf0 msgr2=0x7fedb813cf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:58.279 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:58.274+0000 7fedbf66f640 1 --2- 172.21.15.104:0/1323706575 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fedb8147bf0 0x7fedb813cf90 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7fedb000efc0 tx=0x7fedb000c490 comp rx=0 tx=0).stop 2024-03-28T14:08:58.279 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:58.274+0000 7fedbf66f640 1 -- 172.21.15.104:0/1323706575 shutdown_connections 2024-03-28T14:08:58.279 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:58.274+0000 7fedbf66f640 1 --2- 172.21.15.104:0/1323706575 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7fed9c067fc0 0x7fed9c06a480 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:58.279 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:58.274+0000 7fedbf66f640 1 --2- 172.21.15.104:0/1323706575 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fedb8147bf0 0x7fedb813cf90 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:58.279 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:58.274+0000 7fedbf66f640 1 --2- 172.21.15.104:0/1323706575 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fedb80b4aa0 0x7fedb8145650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:58.279 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:58.274+0000 7fedbf66f640 1 --2- 172.21.15.104:0/1323706575 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fedb80b29e0 0x7fedb81437a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:58.279 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:58.274+0000 7fedbf66f640 1 -- 172.21.15.104:0/1323706575 >> 172.21.15.104:0/1323706575 conn(0x7fedb80b8f00 msgr2=0x7fedb80b1100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T14:08:58.279 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:58.274+0000 7fedbf66f640 1 -- 172.21.15.104:0/1323706575 shutdown_connections 2024-03-28T14:08:58.279 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:58.274+0000 7fedbf66f640 1 -- 172.21.15.104:0/1323706575 wait complete. 2024-03-28T14:08:58.297 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_cephfs_shell.TestCD.test_cd_with_args' 2024-03-28T14:08:58.421 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:58.414+0000 7fdbae42a640 1 Processor -- start 2024-03-28T14:08:58.421 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:58.414+0000 7fdbae42a640 1 -- start start 2024-03-28T14:08:58.421 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:58.418+0000 7fdbae42a640 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fdba806b830 0x7fdba8148000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:58.421 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:58.418+0000 7fdbae42a640 1 --2- >> v2:172.21.15.171:3300/0 conn(0x7fdba8141d40 0x7fdba81520c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:58.422 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:58.418+0000 7fdbae42a640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdba80bae90 con 0x7fdba8148540 2024-03-28T14:08:58.422 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:58.418+0000 7fdbae42a640 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 -- 0x7fdba80b82e0 con 0x7fdba806b830 2024-03-28T14:08:58.422 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:58.418+0000 7fdbae42a640 1 -- --> v2:172.21.15.171:3300/0 -- mon_getmap magic: 0 -- 0x7fdba8147b70 con 0x7fdba8141d40 2024-03-28T14:08:58.422 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:58.418+0000 7fdba7fff640 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fdba806b830 0x7fdba8148000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:58.422 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:58.418+0000 7fdba77fe640 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7fdba8148540 0x7fdba8141630 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:40510/0 (socket says 172.21.15.104:40510) 2024-03-28T14:08:58.422 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:58.418+0000 7fdba77fe640 1 -- 172.21.15.104:0/4127150919 learned_addr learned my addr 172.21.15.104:0/4127150919 (peer_addr_for_me v1:172.21.15.104:0/0) 2024-03-28T14:08:58.422 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:58.418+0000 7fdbac9c7640 1 --2- 172.21.15.104:0/4127150919 >> v2:172.21.15.171:3300/0 conn(0x7fdba8141d40 0x7fdba81520c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:58.423 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:58.418+0000 7fdba7fff640 1 -- 172.21.15.104:0/4127150919 >> v2:172.21.15.171:3300/0 conn(0x7fdba8141d40 msgr2=0x7fdba81520c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T14:08:58.423 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:58.418+0000 7fdba7fff640 1 --2- 172.21.15.104:0/4127150919 >> v2:172.21.15.171:3300/0 conn(0x7fdba8141d40 0x7fdba81520c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:58.423 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:58.418+0000 7fdba7fff640 1 -- 172.21.15.104:0/4127150919 >> v1:172.21.15.104:6789/0 conn(0x7fdba8148540 legacy=0x7fdba8141630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:58.423 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:58.418+0000 7fdba6ffd640 1 -- 172.21.15.104:0/4127150919 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2220561921 0 0) 0x7fdba80bae90 con 0x7fdba8148540 2024-03-28T14:08:58.423 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:58.418+0000 7fdba7fff640 1 -- 172.21.15.104:0/4127150919 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdba8147820 con 0x7fdba806b830 2024-03-28T14:08:58.423 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:58.418+0000 7fdbac9c7640 1 --2- 172.21.15.104:0/4127150919 >> v2:172.21.15.171:3300/0 conn(0x7fdba8141d40 0x7fdba81520c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T14:08:58.427 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:58.418+0000 7fdba7fff640 1 --2- 172.21.15.104:0/4127150919 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fdba806b830 0x7fdba8148000 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7fdba8147cf0 tx=0x7fdb9801c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e8f034254021c2c4 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:08:58.427 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:58.418+0000 7fdba6ffd640 1 -- 172.21.15.104:0/4127150919 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb980b1070 con 0x7fdba806b830 2024-03-28T14:08:58.427 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:58.418+0000 7fdba6ffd640 1 -- 172.21.15.104:0/4127150919 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdb980ac620 con 0x7fdba806b830 2024-03-28T14:08:58.427 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:58.418+0000 7fdba6ffd640 1 -- 172.21.15.104:0/4127150919 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb980ac9f0 con 0x7fdba806b830 2024-03-28T14:08:58.427 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:58.418+0000 7fdbae42a640 1 -- 172.21.15.104:0/4127150919 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fdba806b830 msgr2=0x7fdba8148000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:58.427 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:58.418+0000 7fdbae42a640 1 --2- 172.21.15.104:0/4127150919 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fdba806b830 0x7fdba8148000 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7fdba8147cf0 tx=0x7fdb9801c3a0 comp rx=0 tx=0).stop 2024-03-28T14:08:58.427 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:58.418+0000 7fdbae42a640 1 -- 172.21.15.104:0/4127150919 shutdown_connections 2024-03-28T14:08:58.428 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:58.418+0000 7fdbae42a640 1 --2- 172.21.15.104:0/4127150919 >> v2:172.21.15.171:3300/0 conn(0x7fdba8141d40 0x7fdba81520c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:58.428 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:58.418+0000 7fdbae42a640 1 --2- 172.21.15.104:0/4127150919 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fdba806b830 0x7fdba8148000 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:58.428 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:58.418+0000 7fdbae42a640 1 -- 172.21.15.104:0/4127150919 >> 172.21.15.104:0/4127150919 conn(0x7fdba80b9b50 msgr2=0x7fdba80b9f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T14:08:58.428 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:58.418+0000 7fdbae42a640 1 -- 172.21.15.104:0/4127150919 shutdown_connections 2024-03-28T14:08:58.428 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:58.418+0000 7fdbae42a640 1 -- 172.21.15.104:0/4127150919 wait complete. 2024-03-28T14:08:58.428 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:58.422+0000 7fdbae42a640 1 Processor -- start 2024-03-28T14:08:58.428 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:58.422+0000 7fdbae42a640 1 -- start start 2024-03-28T14:08:58.428 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:58.422+0000 7fdbae42a640 1 --2- >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fdba806b830 0x7fdba81449c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:58.428 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:58.422+0000 7fdbae42a640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fdba8141d40 0x7fdba80a24d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:58.428 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:58.422+0000 7fdbae42a640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fdba8148540 0x7fdba80a2a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:58.428 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:58.422+0000 7fdbae42a640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7fdba80b98e0 con 0x7fdba8141d40 2024-03-28T14:08:58.428 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:58.422+0000 7fdbae42a640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7fdba80b9760 con 0x7fdba8148540 2024-03-28T14:08:58.429 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:58.422+0000 7fdbae42a640 1 -- --> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] -- mon_getmap magic: 0 -- 0x7fdba8154e20 con 0x7fdba806b830 2024-03-28T14:08:58.429 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:58.422+0000 7fdba77fe640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fdba8141d40 0x7fdba80a24d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:58.429 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:58.422+0000 7fdba77fe640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fdba8141d40 0x7fdba80a24d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:49762/0 (socket says 172.21.15.104:49762) 2024-03-28T14:08:58.429 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:58.422+0000 7fdba77fe640 1 -- 172.21.15.104:0/3455913720 learned_addr learned my addr 172.21.15.104:0/3455913720 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-03-28T14:08:58.429 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:58.422+0000 7fdba7fff640 1 --2- 172.21.15.104:0/3455913720 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fdba806b830 0x7fdba81449c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:58.429 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:58.422+0000 7fdba77fe640 1 -- 172.21.15.104:0/3455913720 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fdba806b830 msgr2=0x7fdba81449c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:58.429 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:58.422+0000 7fdbac9c7640 1 --2- 172.21.15.104:0/3455913720 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fdba8148540 0x7fdba80a2a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:58.429 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:58.422+0000 7fdba77fe640 1 --2- 172.21.15.104:0/3455913720 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fdba806b830 0x7fdba81449c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:58.429 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:58.422+0000 7fdba77fe640 1 -- 172.21.15.104:0/3455913720 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fdba8148540 msgr2=0x7fdba80a2a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:58.429 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:58.422+0000 7fdba77fe640 1 --2- 172.21.15.104:0/3455913720 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fdba8148540 0x7fdba80a2a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:58.429 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:58.422+0000 7fdba77fe640 1 -- 172.21.15.104:0/3455913720 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdba8143fd0 con 0x7fdba8141d40 2024-03-28T14:08:58.429 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:58.422+0000 7fdba77fe640 1 --2- 172.21.15.104:0/3455913720 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fdba8141d40 0x7fdba80a24d0 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7fdb9c00c450 tx=0x7fdb9c00c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:08:58.429 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:58.422+0000 7fdba4ff9640 1 -- 172.21.15.104:0/3455913720 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb9c018070 con 0x7fdba8141d40 2024-03-28T14:08:58.429 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:58.422+0000 7fdba4ff9640 1 -- 172.21.15.104:0/3455913720 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdb9c0074e0 con 0x7fdba8141d40 2024-03-28T14:08:58.429 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:58.422+0000 7fdbae42a640 1 -- 172.21.15.104:0/3455913720 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdba8145370 con 0x7fdba8141d40 2024-03-28T14:08:58.430 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:58.422+0000 7fdba4ff9640 1 -- 172.21.15.104:0/3455913720 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb9c0049f0 con 0x7fdba8141d40 2024-03-28T14:08:58.430 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:58.422+0000 7fdbae42a640 1 -- 172.21.15.104:0/3455913720 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdba8143030 con 0x7fdba8141d40 2024-03-28T14:08:58.430 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:58.422+0000 7fdbae42a640 1 -- 172.21.15.104:0/3455913720 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdba8142b60 con 0x7fdba8141d40 2024-03-28T14:08:58.431 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:58.422+0000 7fdba4ff9640 1 -- 172.21.15.104:0/3455913720 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) ==== 87803+0+0 (secure 0 0 0) 0x7fdb9c01b070 con 0x7fdba8141d40 2024-03-28T14:08:58.431 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:58.426+0000 7fdba4ff9640 1 --2- 172.21.15.104:0/3455913720 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7fdb94067fc0 0x7fdb9406a480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:58.431 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:58.426+0000 7fdba4ff9640 1 -- 172.21.15.104:0/3455913720 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(33..33 src has 1..33) ==== 4943+0+0 (secure 0 0 0) 0x7fdb9c0854f0 con 0x7fdba8141d40 2024-03-28T14:08:58.431 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:58.426+0000 7fdba7fff640 1 --2- 172.21.15.104:0/3455913720 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7fdb94067fc0 0x7fdb9406a480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:58.431 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:58.426+0000 7fdba7fff640 1 --2- 172.21.15.104:0/3455913720 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7fdb94067fc0 0x7fdb9406a480 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fdb980ae000 tx=0x7fdb98008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:08:58.432 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:58.430+0000 7fdba4ff9640 1 -- 172.21.15.104:0/3455913720 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179117 (secure 0 0 0) 0x7fdb9c056030 con 0x7fdba8141d40 2024-03-28T14:08:58.652 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:58.646+0000 7fdbae42a640 1 -- 172.21.15.104:0/3455913720 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_cephfs_shell.TestCD.test_cd_with_args"]} v 0) -- 0x7fdba80687c0 con 0x7fdba8141d40 2024-03-28T14:08:58.946 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:58.942+0000 7fdba4ff9640 1 -- 172.21.15.104:0/3455913720 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_cephfs_shell.TestCD.test_cd_with_args"]}]=0 v80) ==== 134+0+0 (secure 0 0 0) 0x7fdb9c055930 con 0x7fdba8141d40 2024-03-28T14:08:58.949 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:58.946+0000 7fdbae42a640 1 -- 172.21.15.104:0/3455913720 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7fdb94067fc0 msgr2=0x7fdb9406a480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:58.949 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:58.946+0000 7fdbae42a640 1 --2- 172.21.15.104:0/3455913720 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7fdb94067fc0 0x7fdb9406a480 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fdb980ae000 tx=0x7fdb98008040 comp rx=0 tx=0).stop 2024-03-28T14:08:58.949 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:58.946+0000 7fdbae42a640 1 -- 172.21.15.104:0/3455913720 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fdba8141d40 msgr2=0x7fdba80a24d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:58.949 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:58.946+0000 7fdbae42a640 1 --2- 172.21.15.104:0/3455913720 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fdba8141d40 0x7fdba80a24d0 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7fdb9c00c450 tx=0x7fdb9c00c910 comp rx=0 tx=0).stop 2024-03-28T14:08:58.949 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:58.946+0000 7fdbae42a640 1 -- 172.21.15.104:0/3455913720 shutdown_connections 2024-03-28T14:08:58.950 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:58.946+0000 7fdbae42a640 1 --2- 172.21.15.104:0/3455913720 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fdba8148540 0x7fdba80a2a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:58.950 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:58.946+0000 7fdbae42a640 1 --2- 172.21.15.104:0/3455913720 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7fdb94067fc0 0x7fdb9406a480 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:58.950 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:58.946+0000 7fdbae42a640 1 --2- 172.21.15.104:0/3455913720 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fdba8141d40 0x7fdba80a24d0 unknown :-1 s=CLOSED pgs=431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:58.950 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:58.946+0000 7fdbae42a640 1 --2- 172.21.15.104:0/3455913720 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fdba806b830 0x7fdba81449c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:58.950 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:58.946+0000 7fdbae42a640 1 -- 172.21.15.104:0/3455913720 >> 172.21.15.104:0/3455913720 conn(0x7fdba80b9b50 msgr2=0x7fdba8140490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T14:08:58.950 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:58.946+0000 7fdbae42a640 1 -- 172.21.15.104:0/3455913720 shutdown_connections 2024-03-28T14:08:58.950 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:58.946+0000 7fdbae42a640 1 -- 172.21.15.104:0/3455913720 wait complete. 2024-03-28T14:08:58.970 INFO:tasks.cephfs_test_runner:Test that when cd is issued with an argument, CWD is changed ... ERROR 2024-03-28T14:08:58.971 INFO:tasks.cephfs_test_runner: 2024-03-28T14:08:58.971 INFO:tasks.cephfs_test_runner:====================================================================== 2024-03-28T14:08:58.971 INFO:tasks.cephfs_test_runner:ERROR: test_cd_with_args (tasks.cephfs.test_cephfs_shell.TestCD) 2024-03-28T14:08:58.971 INFO:tasks.cephfs_test_runner:Test that when cd is issued with an argument, CWD is changed 2024-03-28T14:08:58.971 INFO:tasks.cephfs_test_runner:---------------------------------------------------------------------- 2024-03-28T14:08:58.971 INFO:tasks.cephfs_test_runner:Traceback (most recent call last): 2024-03-28T14:08:58.971 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_ceph-c_e0e48ba6ee8c8b4c8f38d3b879d6eafbbb1efc15/qa/tasks/cephfs/test_cephfs_shell.py", line 602, in test_cd_with_args 2024-03-28T14:08:58.971 INFO:tasks.cephfs_test_runner: output = self.get_cephfs_shell_script_output(script) 2024-03-28T14:08:58.971 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_ceph-c_e0e48ba6ee8c8b4c8f38d3b879d6eafbbb1efc15/qa/tasks/cephfs/test_cephfs_shell.py", line 161, in get_cephfs_shell_script_output 2024-03-28T14:08:58.971 INFO:tasks.cephfs_test_runner: return ensure_str(self.run_cephfs_shell_script( 2024-03-28T14:08:58.971 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_ceph-c_e0e48ba6ee8c8b4c8f38d3b879d6eafbbb1efc15/qa/tasks/cephfs/test_cephfs_shell.py", line 153, in run_cephfs_shell_script 2024-03-28T14:08:58.971 INFO:tasks.cephfs_test_runner: return mount_x.client_remote.run(args=args, stdout=stdout, 2024-03-28T14:08:58.971 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/remote.py", line 523, in run 2024-03-28T14:08:58.972 INFO:tasks.cephfs_test_runner: r = self._runner(client=self.ssh, name=self.shortname, **kwargs) 2024-03-28T14:08:58.972 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 455, in run 2024-03-28T14:08:58.972 INFO:tasks.cephfs_test_runner: r.wait() 2024-03-28T14:08:58.972 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait 2024-03-28T14:08:58.972 INFO:tasks.cephfs_test_runner: self._raise_for_status() 2024-03-28T14:08:58.972 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status 2024-03-28T14:08:58.972 INFO:tasks.cephfs_test_runner: raise CommandFailedError( 2024-03-28T14:08:58.972 INFO:tasks.cephfs_test_runner:teuthology.exceptions.CommandFailedError: Command failed on smithi104 with status 127: 'cephfs-shell -b /tmp/test-cephfspmcncaaq' 2024-03-28T14:08:58.972 INFO:tasks.cephfs_test_runner: 2024-03-28T14:08:58.972 INFO:tasks.cephfs_test_runner:---------------------------------------------------------------------- 2024-03-28T14:08:58.972 INFO:tasks.cephfs_test_runner:Ran 1 test in 38.810s 2024-03-28T14:08:58.972 INFO:tasks.cephfs_test_runner: 2024-03-28T14:08:58.972 INFO:tasks.cephfs_test_runner:FAILED (errors=1) 2024-03-28T14:08:58.972 INFO:tasks.cephfs_test_runner: 2024-03-28T14:08:58.972 INFO:tasks.cephfs_test_runner:====================================================================== 2024-03-28T14:08:58.973 INFO:tasks.cephfs_test_runner:ERROR: test_cd_with_args (tasks.cephfs.test_cephfs_shell.TestCD) 2024-03-28T14:08:58.973 INFO:tasks.cephfs_test_runner:Test that when cd is issued with an argument, CWD is changed 2024-03-28T14:08:58.973 INFO:tasks.cephfs_test_runner:---------------------------------------------------------------------- 2024-03-28T14:08:58.973 INFO:tasks.cephfs_test_runner:Traceback (most recent call last): 2024-03-28T14:08:58.973 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_ceph-c_e0e48ba6ee8c8b4c8f38d3b879d6eafbbb1efc15/qa/tasks/cephfs/test_cephfs_shell.py", line 602, in test_cd_with_args 2024-03-28T14:08:58.973 INFO:tasks.cephfs_test_runner: output = self.get_cephfs_shell_script_output(script) 2024-03-28T14:08:58.973 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_ceph-c_e0e48ba6ee8c8b4c8f38d3b879d6eafbbb1efc15/qa/tasks/cephfs/test_cephfs_shell.py", line 161, in get_cephfs_shell_script_output 2024-03-28T14:08:58.973 INFO:tasks.cephfs_test_runner: return ensure_str(self.run_cephfs_shell_script( 2024-03-28T14:08:58.973 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_ceph-c_e0e48ba6ee8c8b4c8f38d3b879d6eafbbb1efc15/qa/tasks/cephfs/test_cephfs_shell.py", line 153, in run_cephfs_shell_script 2024-03-28T14:08:58.973 INFO:tasks.cephfs_test_runner: return mount_x.client_remote.run(args=args, stdout=stdout, 2024-03-28T14:08:58.973 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/remote.py", line 523, in run 2024-03-28T14:08:58.973 INFO:tasks.cephfs_test_runner: r = self._runner(client=self.ssh, name=self.shortname, **kwargs) 2024-03-28T14:08:58.973 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 455, in run 2024-03-28T14:08:58.973 INFO:tasks.cephfs_test_runner: r.wait() 2024-03-28T14:08:58.973 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait 2024-03-28T14:08:58.973 INFO:tasks.cephfs_test_runner: self._raise_for_status() 2024-03-28T14:08:58.974 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status 2024-03-28T14:08:58.974 INFO:tasks.cephfs_test_runner: raise CommandFailedError( 2024-03-28T14:08:58.974 INFO:tasks.cephfs_test_runner:teuthology.exceptions.CommandFailedError: Command failed on smithi104 with status 127: 'cephfs-shell -b /tmp/test-cephfspmcncaaq' 2024-03-28T14:08:58.974 INFO:tasks.cephfs_test_runner: 2024-03-28T14:08:58.974 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_e0e48ba6ee8c8b4c8f38d3b879d6eafbbb1efc15/qa/tasks/cephfs_test_runner.py", line 211, in task raise RuntimeError("Test failure: {0}".format(", ".join(bad_tests))) RuntimeError: Test failure: test_cd_with_args (tasks.cephfs.test_cephfs_shell.TestCD) 2024-03-28T14:08:59.216 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=7a9146a91cbe4b9c9cf66cd4591d6be4 Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_e0e48ba6ee8c8b4c8f38d3b879d6eafbbb1efc15/qa/tasks/cephfs_test_runner.py", line 211, in task raise RuntimeError("Test failure: {0}".format(", ".join(bad_tests))) RuntimeError: Test failure: test_cd_with_args (tasks.cephfs.test_cephfs_shell.TestCD) 2024-03-28T14:08:59.218 DEBUG:teuthology.run_tasks:Unwinding manager cephfs_test_runner 2024-03-28T14:08:59.227 DEBUG:teuthology.run_tasks:Unwinding manager ceph-fuse 2024-03-28T14:08:59.235 INFO:tasks.ceph_fuse:Unmounting ceph-fuse clients... 2024-03-28T14:08:59.236 DEBUG:teuthology.orchestra.run.smithi104:> set -ex 2024-03-28T14:08:59.236 DEBUG:teuthology.orchestra.run.smithi104:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-28T14:08:59.239 DEBUG:teuthology.orchestra.run.smithi104:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-28T14:08:59.250 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=16336516k,nr_inodes=4084129,mode=755,inode64 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3274696k,mode=755,inode64 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev,inode64 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=20838 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysusers.service ramfs ro,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3274692k,nr_inodes=818673,mode=700,uid=1000,gid=1271,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3274696k,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 2024-03-28T14:08:59.250 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T14:08:59.251 DEBUG:teuthology.orchestra.run.smithi104:> ip netns list 2024-03-28T14:08:59.251 DEBUG:teuthology.orchestra.run.smithi171:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-28T14:08:59.340 INFO:teuthology.orchestra.run.smithi104.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-03-28T14:08:59.341 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T14:08:59.341 DEBUG:teuthology.orchestra.run.smithi104:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-28T14:08:59.398 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T14:08:59.398 DEBUG:teuthology.orchestra.run.smithi104:> sudo ip link delete ceph-brx 2024-03-28T14:08:59.590 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-03-28T14:08:59.599 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-03-28T14:08:59.599 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-28T14:08:59.724 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:59.722+0000 7f8112a5a640 1 Processor -- start 2024-03-28T14:08:59.724 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:59.722+0000 7f8112a5a640 1 -- start start 2024-03-28T14:08:59.726 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:59.722+0000 7f8112a5a640 1 --2- >> v2:172.21.15.171:3300/0 conn(0x7f810c0af930 0x7f810c148140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:59.726 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:59.722+0000 7f8112a5a640 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f810c148680 0x7f810c14ef20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:59.726 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:59.722+0000 7f8112a5a640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f810c148ad0 con 0x7f810c06d970 2024-03-28T14:08:59.726 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:59.722+0000 7f8112a5a640 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 -- 0x7f810c0b5500 con 0x7f810c148680 2024-03-28T14:08:59.727 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:59.722+0000 7f8112a5a640 1 -- --> v2:172.21.15.171:3300/0 -- mon_getmap magic: 0 -- 0x7f810c0b02a0 con 0x7f810c0af930 2024-03-28T14:08:59.727 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:59.722+0000 7f8110ff7640 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f810c148680 0x7f810c14ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:59.727 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:59.722+0000 7f8110ff7640 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f810c148680 0x7f810c14ef20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:49764/0 (socket says 172.21.15.104:49764) 2024-03-28T14:08:59.727 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:59.722+0000 7f8110ff7640 1 -- 172.21.15.104:0/3768845697 learned_addr learned my addr 172.21.15.104:0/3768845697 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-03-28T14:08:59.727 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:59.722+0000 7f810b7fe640 1 --2- 172.21.15.104:0/3768845697 >> v2:172.21.15.171:3300/0 conn(0x7f810c0af930 0x7f810c148140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:59.727 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:59.722+0000 7f8110ff7640 1 -- 172.21.15.104:0/3768845697 >> v2:172.21.15.171:3300/0 conn(0x7f810c0af930 msgr2=0x7f810c148140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T14:08:59.727 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:59.722+0000 7f8110ff7640 1 --2- 172.21.15.104:0/3768845697 >> v2:172.21.15.171:3300/0 conn(0x7f810c0af930 0x7f810c148140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:59.727 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:59.722+0000 7f8110ff7640 1 -- 172.21.15.104:0/3768845697 >> v1:172.21.15.171:6789/0 conn(0x7f810c06d970 legacy=0x7f810c0addc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T14:08:59.728 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:59.722+0000 7f8110ff7640 1 -- 172.21.15.104:0/3768845697 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f810c06dd50 con 0x7f810c148680 2024-03-28T14:08:59.729 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:59.722+0000 7f8110ff7640 1 --2- 172.21.15.104:0/3768845697 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f810c148680 0x7f810c14ef20 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f810c0b4960 tx=0x7f810401c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2e84a877d48ecdb4 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:08:59.729 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:59.726+0000 7f810affd640 1 -- 172.21.15.104:0/3768845697 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f81040a55e0 con 0x7f810c148680 2024-03-28T14:08:59.729 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:59.726+0000 7f810affd640 1 -- 172.21.15.104:0/3768845697 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f81040a5780 con 0x7f810c148680 2024-03-28T14:08:59.729 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:59.726+0000 7f810affd640 1 -- 172.21.15.104:0/3768845697 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f81040a5b50 con 0x7f810c148680 2024-03-28T14:08:59.729 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:59.726+0000 7f8112a5a640 1 -- 172.21.15.104:0/3768845697 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f810c148680 msgr2=0x7f810c14ef20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:59.730 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:59.726+0000 7f8112a5a640 1 --2- 172.21.15.104:0/3768845697 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f810c148680 0x7f810c14ef20 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f810c0b4960 tx=0x7f810401c920 comp rx=0 tx=0).stop 2024-03-28T14:08:59.730 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:59.726+0000 7f8112a5a640 1 -- 172.21.15.104:0/3768845697 shutdown_connections 2024-03-28T14:08:59.730 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:59.726+0000 7f8112a5a640 1 --2- 172.21.15.104:0/3768845697 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f810c148680 0x7f810c14ef20 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:59.730 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:59.726+0000 7f8112a5a640 1 --2- 172.21.15.104:0/3768845697 >> v2:172.21.15.171:3300/0 conn(0x7f810c0af930 0x7f810c148140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:59.730 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:59.726+0000 7f8112a5a640 1 -- 172.21.15.104:0/3768845697 >> 172.21.15.104:0/3768845697 conn(0x7f810c0b58a0 msgr2=0x7f810c0b5cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T14:08:59.730 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:59.726+0000 7f8112a5a640 1 -- 172.21.15.104:0/3768845697 shutdown_connections 2024-03-28T14:08:59.730 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:59.726+0000 7f8112a5a640 1 -- 172.21.15.104:0/3768845697 wait complete. 2024-03-28T14:08:59.733 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:59.726+0000 7f8112a5a640 1 Processor -- start 2024-03-28T14:08:59.733 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:59.726+0000 7f8112a5a640 1 -- start start 2024-03-28T14:08:59.733 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:59.726+0000 7f8112a5a640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f810c06d970 0x7f810c13a290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:59.733 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:59.726+0000 7f8112a5a640 1 --2- >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f810c0af930 0x7f810c1ddc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:59.733 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:59.726+0000 7f8112a5a640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f810c148680 0x7f810c131070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:59.733 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:59.726+0000 7f8112a5a640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f810c0ae4d0 con 0x7f810c148680 2024-03-28T14:08:59.733 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:59.726+0000 7f8112a5a640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f810c09ed80 con 0x7f810c06d970 2024-03-28T14:08:59.734 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:59.726+0000 7f8112a5a640 1 -- --> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] -- mon_getmap magic: 0 -- 0x7f810c13d9a0 con 0x7f810c0af930 2024-03-28T14:08:59.734 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:59.726+0000 7f8110ff7640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f810c148680 0x7f810c131070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:59.734 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:59.726+0000 7f8110ff7640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f810c148680 0x7f810c131070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:49770/0 (socket says 172.21.15.104:49770) 2024-03-28T14:08:59.734 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:59.726+0000 7f8110ff7640 1 -- 172.21.15.104:0/4071225962 learned_addr learned my addr 172.21.15.104:0/4071225962 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-03-28T14:08:59.734 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:59.726+0000 7f8110ff7640 1 -- 172.21.15.104:0/4071225962 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f810c0af930 msgr2=0x7f810c1ddc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:59.734 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:59.726+0000 7f8110ff7640 1 --2- 172.21.15.104:0/4071225962 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f810c0af930 0x7f810c1ddc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:59.734 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:59.726+0000 7f8110ff7640 1 -- 172.21.15.104:0/4071225962 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f810c06d970 msgr2=0x7f810c13a290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:59.735 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:59.726+0000 7f810bfff640 1 --2- 172.21.15.104:0/4071225962 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f810c06d970 0x7f810c13a290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:59.735 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:59.726+0000 7f8110ff7640 1 --2- 172.21.15.104:0/4071225962 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f810c06d970 0x7f810c13a290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:59.735 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:59.726+0000 7f8110ff7640 1 -- 172.21.15.104:0/4071225962 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f810c13e6f0 con 0x7f810c148680 2024-03-28T14:08:59.735 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:59.726+0000 7f8110ff7640 1 --2- 172.21.15.104:0/4071225962 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f810c148680 0x7f810c131070 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7f81040af040 tx=0x7f81040196c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:08:59.735 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:59.726+0000 7f8108ff9640 1 -- 172.21.15.104:0/4071225962 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f81040bc020 con 0x7f810c148680 2024-03-28T14:08:59.735 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:59.726+0000 7f8108ff9640 1 -- 172.21.15.104:0/4071225962 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f81040b8670 con 0x7f810c148680 2024-03-28T14:08:59.735 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:59.726+0000 7f8108ff9640 1 -- 172.21.15.104:0/4071225962 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8104009510 con 0x7f810c148680 2024-03-28T14:08:59.736 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:59.726+0000 7f8112a5a640 1 -- 172.21.15.104:0/4071225962 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f810c13d3e0 con 0x7f810c148680 2024-03-28T14:08:59.736 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:59.726+0000 7f8112a5a640 1 -- 172.21.15.104:0/4071225962 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f810c0b7c60 con 0x7f810c148680 2024-03-28T14:08:59.736 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:59.730+0000 7f8112a5a640 1 -- 172.21.15.104:0/4071225962 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f810c0b7e60 con 0x7f810c148680 2024-03-28T14:08:59.736 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:59.730+0000 7f8108ff9640 1 -- 172.21.15.104:0/4071225962 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) ==== 87803+0+0 (secure 0 0 0) 0x7f81040bf020 con 0x7f810c148680 2024-03-28T14:08:59.737 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:59.730+0000 7f8108ff9640 1 --2- 172.21.15.104:0/4071225962 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f80ec067fc0 0x7f80ec06a480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:08:59.737 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:59.730+0000 7f8108ff9640 1 -- 172.21.15.104:0/4071225962 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(33..33 src has 1..33) ==== 4943+0+0 (secure 0 0 0) 0x7f8104121120 con 0x7f810c148680 2024-03-28T14:08:59.737 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:59.730+0000 7f810bfff640 1 --2- 172.21.15.104:0/4071225962 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f80ec067fc0 0x7f80ec06a480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:08:59.738 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:59.730+0000 7f810bfff640 1 --2- 172.21.15.104:0/4071225962 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f80ec067fc0 0x7f80ec06a480 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f80fc00a750 tx=0x7f80fc008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:08:59.738 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:59.734+0000 7f8108ff9640 1 -- 172.21.15.104:0/4071225962 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179117 (secure 0 0 0) 0x7f810c0b7e60 con 0x7f810c148680 2024-03-28T14:08:59.896 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:59.890+0000 7f8112a5a640 1 -- 172.21.15.104:0/4071225962 --> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f810c0687c0 con 0x7f80ec067fc0 2024-03-28T14:08:59.898 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:59.894+0000 7f8108ff9640 1 -- 172.21.15.104:0/4071225962 <== mgr.4100 v2:172.21.15.171:6832/1522752412 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+13458 (secure 0 0 0) 0x7f810c0687c0 con 0x7f80ec067fc0 2024-03-28T14:08:59.898 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-03-28T14:08:59.898 INFO:teuthology.orchestra.run.smithi104.stderr:dumped all 2024-03-28T14:08:59.901 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:59.898+0000 7f8112a5a640 1 -- 172.21.15.104:0/4071225962 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f80ec067fc0 msgr2=0x7f80ec06a480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:59.901 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:59.898+0000 7f8112a5a640 1 --2- 172.21.15.104:0/4071225962 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f80ec067fc0 0x7f80ec06a480 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f80fc00a750 tx=0x7f80fc008040 comp rx=0 tx=0).stop 2024-03-28T14:08:59.901 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:59.898+0000 7f8112a5a640 1 -- 172.21.15.104:0/4071225962 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f810c148680 msgr2=0x7f810c131070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:08:59.901 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:59.898+0000 7f8112a5a640 1 --2- 172.21.15.104:0/4071225962 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f810c148680 0x7f810c131070 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7f81040af040 tx=0x7f81040196c0 comp rx=0 tx=0).stop 2024-03-28T14:08:59.901 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:59.898+0000 7f8112a5a640 1 -- 172.21.15.104:0/4071225962 shutdown_connections 2024-03-28T14:08:59.901 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:59.898+0000 7f8112a5a640 1 --2- 172.21.15.104:0/4071225962 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f80ec067fc0 0x7f80ec06a480 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:59.901 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:59.898+0000 7f8112a5a640 1 --2- 172.21.15.104:0/4071225962 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f810c148680 0x7f810c131070 unknown :-1 s=CLOSED pgs=433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:59.901 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:59.898+0000 7f8112a5a640 1 --2- 172.21.15.104:0/4071225962 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f810c0af930 0x7f810c1ddc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:59.902 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:59.898+0000 7f8112a5a640 1 --2- 172.21.15.104:0/4071225962 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f810c06d970 0x7f810c13a290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:08:59.902 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:59.898+0000 7f8112a5a640 1 -- 172.21.15.104:0/4071225962 >> 172.21.15.104:0/4071225962 conn(0x7f810c0b58a0 msgr2=0x7f810c0b4030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T14:08:59.902 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:59.898+0000 7f8112a5a640 1 -- 172.21.15.104:0/4071225962 shutdown_connections 2024-03-28T14:08:59.902 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:08:59.898+0000 7f8112a5a640 1 -- 172.21.15.104:0/4071225962 wait complete. 2024-03-28T14:08:59.917 INFO:teuthology.orchestra.run.smithi104.stdout:{"pg_ready":true,"pg_map":{"version":63,"stamp":"2024-03-28T14:08:59.598545+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":220096,"kb_used_data":3932,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754754624,"statfs":{"total":773094113280,"available":772868734976,"internally_reserved":0,"allocated":4026368,"data_stored":2270761,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":221236785},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-10891,"num_objects":-22,"num_object_clones":0,"num_object_copies":-44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-55,"num_write_kb":-27,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":-14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.002292"},"pg_stats":[{"pgid":"1.0","version":"17'117","reported_seq":188,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-28T14:08:55.142304+0000","last_change":"2024-03-28T14:07:49.234448+0000","last_active":"2024-03-28T14:08:55.142304+0000","last_peered":"2024-03-28T14:08:55.142304+0000","last_clean":"2024-03-28T14:08:55.142304+0000","last_became_active":"2024-03-28T14:07:49.233895+0000","last_became_peered":"2024-03-28T14:07:49.233895+0000","last_unstale":"2024-03-28T14:08:55.142304+0000","last_undegraded":"2024-03-28T14:08:55.142304+0000","last_fullsized":"2024-03-28T14:08:55.142304+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:07:48.069157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:07:48.069157+0000","last_clean_scrub_stamp":"2024-03-28T14:07:48.069157+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:11:39.977557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574866,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27944,"kb_used_data":924,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343896,"statfs":{"total":96636764160,"available":96608149504,"internally_reserved":0,"allocated":946176,"data_stored":725222,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574867,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27360,"kb_used_data":336,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344480,"statfs":{"total":96636764160,"available":96608747520,"internally_reserved":0,"allocated":344064,"data_stored":129299,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574865,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27360,"kb_used_data":336,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344480,"statfs":{"total":96636764160,"available":96608747520,"internally_reserved":0,"allocated":344064,"data_stored":129355,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574866,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27368,"kb_used_data":348,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344472,"statfs":{"total":96636764160,"available":96608739328,"internally_reserved":0,"allocated":356352,"data_stored":134868,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574868,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27372,"kb_used_data":356,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344468,"statfs":{"total":96636764160,"available":96608735232,"internally_reserved":0,"allocated":364544,"data_stored":142985,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574866,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27364,"kb_used_data":344,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344476,"statfs":{"total":96636764160,"available":96608743424,"internally_reserved":0,"allocated":352256,"data_stored":134834,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574864,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27388,"kb_used_data":368,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344452,"statfs":{"total":96636764160,"available":96608718848,"internally_reserved":0,"allocated":376832,"data_stored":149066,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574867,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27940,"kb_used_data":920,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343900,"statfs":{"total":96636764160,"available":96608153600,"internally_reserved":0,"allocated":942080,"data_stored":725132,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-28T14:08:59.917 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-28T14:09:00.040 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:00.038+0000 7fc6635fa640 1 Processor -- start 2024-03-28T14:09:00.041 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:00.038+0000 7fc6635fa640 1 -- start start 2024-03-28T14:09:00.044 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:00.038+0000 7fc6635fa640 1 --2- >> v2:172.21.15.171:3300/0 conn(0x7fc65c06d970 0x7fc65c13f3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:09:00.044 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:00.038+0000 7fc6635fa640 1 --2- >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fc65c0b5460 0x7fc65c13f920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:09:00.044 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:00.038+0000 7fc6635fa640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc65c06dd50 con 0x7fc65c0732a0 2024-03-28T14:09:00.045 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:00.038+0000 7fc6635fa640 1 -- --> v2:172.21.15.171:3300/0 -- mon_getmap magic: 0 -- 0x7fc65c0b02a0 con 0x7fc65c06d970 2024-03-28T14:09:00.045 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:00.038+0000 7fc6635fa640 1 -- --> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] -- mon_getmap magic: 0 -- 0x7fc65c0b2d30 con 0x7fc65c0b5460 2024-03-28T14:09:00.045 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:00.038+0000 7fc660b95640 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7fc65c0732a0 0x7fc65c0b0c30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:40522/0 (socket says 172.21.15.104:40522) 2024-03-28T14:09:00.045 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:00.038+0000 7fc660b95640 1 -- 172.21.15.104:0/2315875801 learned_addr learned my addr 172.21.15.104:0/2315875801 (peer_addr_for_me v1:172.21.15.104:0/0) 2024-03-28T14:09:00.045 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:00.038+0000 7fc661396640 1 --2- 172.21.15.104:0/2315875801 >> v2:172.21.15.171:3300/0 conn(0x7fc65c06d970 0x7fc65c13f3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:09:00.046 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:00.038+0000 7fc661b97640 1 --2- 172.21.15.104:0/2315875801 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fc65c0b5460 0x7fc65c13f920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:09:00.046 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:00.038+0000 7fc65bfff640 1 -- 172.21.15.104:0/2315875801 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3683893176 0 0) 0x7fc65c06dd50 con 0x7fc65c0732a0 2024-03-28T14:09:00.046 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:00.038+0000 7fc65bfff640 1 -- 172.21.15.104:0/2315875801 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc644003530 con 0x7fc65c0732a0 2024-03-28T14:09:00.046 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:00.038+0000 7fc65bfff640 1 -- 172.21.15.104:0/2315875801 <== mon.0 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 210500802 0 0) 0x7fc644003530 con 0x7fc65c0732a0 2024-03-28T14:09:00.046 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:00.038+0000 7fc65bfff640 1 -- 172.21.15.104:0/2315875801 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fc65c0b5460 msgr2=0x7fc65c13f920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T14:09:00.046 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:00.038+0000 7fc65bfff640 1 --2- 172.21.15.104:0/2315875801 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fc65c0b5460 0x7fc65c13f920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:09:00.047 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:00.038+0000 7fc65bfff640 1 -- 172.21.15.104:0/2315875801 >> v2:172.21.15.171:3300/0 conn(0x7fc65c06d970 msgr2=0x7fc65c13f3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T14:09:00.047 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:00.038+0000 7fc65bfff640 1 --2- 172.21.15.104:0/2315875801 >> v2:172.21.15.171:3300/0 conn(0x7fc65c06d970 0x7fc65c13f3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:09:00.047 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:00.038+0000 7fc65bfff640 1 -- 172.21.15.104:0/2315875801 --> v1:172.21.15.104:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc65c0b3010 con 0x7fc65c0732a0 2024-03-28T14:09:00.047 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:00.038+0000 7fc65bfff640 1 -- 172.21.15.104:0/2315875801 <== mon.0 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3745580849 0 0) 0x7fc650001a00 con 0x7fc65c0732a0 2024-03-28T14:09:00.047 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:00.038+0000 7fc65bfff640 1 -- 172.21.15.104:0/2315875801 >> v1:172.21.15.104:6789/0 conn(0x7fc65c0732a0 legacy=0x7fc65c0b0c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:09:00.047 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:00.038+0000 7fc65bfff640 1 --2- 172.21.15.104:0/2315875801 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fc6440039c0 0x7fc644003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:09:00.047 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:00.038+0000 7fc65bfff640 1 --2- 172.21.15.104:0/2315875801 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc644004810 0x7fc644004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:09:00.048 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:00.038+0000 7fc65bfff640 1 --2- 172.21.15.104:0/2315875801 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fc65c0b5460 0x7fc6440051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:09:00.048 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:00.038+0000 7fc65bfff640 1 -- 172.21.15.104:0/2315875801 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7fc65c0b02a0 con 0x7fc644004810 2024-03-28T14:09:00.048 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:00.038+0000 7fc65bfff640 1 -- 172.21.15.104:0/2315875801 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7fc65c0b2d30 con 0x7fc6440039c0 2024-03-28T14:09:00.048 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:00.038+0000 7fc65bfff640 1 -- 172.21.15.104:0/2315875801 --> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] -- mon_getmap magic: 0 -- 0x7fc64400b9d0 con 0x7fc65c0b5460 2024-03-28T14:09:00.048 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:00.038+0000 7fc660b95640 1 --2- 172.21.15.104:0/2315875801 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc644004810 0x7fc644004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:09:00.049 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:00.038+0000 7fc661396640 1 --2- 172.21.15.104:0/2315875801 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fc6440039c0 0x7fc644003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:09:00.049 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:00.038+0000 7fc661b97640 1 --2- 172.21.15.104:0/2315875801 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fc65c0b5460 0x7fc6440051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:09:00.049 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:00.038+0000 7fc660b95640 1 -- 172.21.15.104:0/2315875801 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fc65c0b5460 msgr2=0x7fc6440051d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T14:09:00.049 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:00.038+0000 7fc660b95640 1 --2- 172.21.15.104:0/2315875801 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fc65c0b5460 0x7fc6440051d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:09:00.049 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:00.038+0000 7fc660b95640 1 -- 172.21.15.104:0/2315875801 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fc6440039c0 msgr2=0x7fc644003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T14:09:00.050 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:00.038+0000 7fc660b95640 1 --2- 172.21.15.104:0/2315875801 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fc6440039c0 0x7fc644003de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:09:00.050 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:00.038+0000 7fc660b95640 1 -- 172.21.15.104:0/2315875801 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc644003530 con 0x7fc644004810 2024-03-28T14:09:00.050 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:00.042+0000 7fc660b95640 1 --2- 172.21.15.104:0/2315875801 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc644004810 0x7fc644004c90 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7fc65c09eb70 tx=0x7fc65001d2f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d039ea1cd38de976 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:09:00.050 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:00.042+0000 7fc65bfff640 1 -- 172.21.15.104:0/2315875801 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc65000a4c0 con 0x7fc644004810 2024-03-28T14:09:00.050 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:00.042+0000 7fc65bfff640 1 -- 172.21.15.104:0/2315875801 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc6500ad630 con 0x7fc644004810 2024-03-28T14:09:00.050 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:00.042+0000 7fc65bfff640 1 -- 172.21.15.104:0/2315875801 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6500ada20 con 0x7fc644004810 2024-03-28T14:09:00.054 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:00.042+0000 7fc6635fa640 1 -- 172.21.15.104:0/2315875801 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc644004810 msgr2=0x7fc644004c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:09:00.054 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:00.042+0000 7fc6635fa640 1 --2- 172.21.15.104:0/2315875801 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc644004810 0x7fc644004c90 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7fc65c09eb70 tx=0x7fc65001d2f0 comp rx=0 tx=0).stop 2024-03-28T14:09:00.054 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:00.042+0000 7fc661b97640 1 -- 172.21.15.104:0/2315875801 reap_dead start 2024-03-28T14:09:00.054 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:00.042+0000 7fc6635fa640 1 -- 172.21.15.104:0/2315875801 shutdown_connections 2024-03-28T14:09:00.054 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:00.042+0000 7fc6635fa640 1 -- 172.21.15.104:0/2315875801 >> 172.21.15.104:0/2315875801 conn(0x7fc65c0b58a0 msgr2=0x7fc65c0b5cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T14:09:00.054 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:00.042+0000 7fc6635fa640 1 -- 172.21.15.104:0/2315875801 shutdown_connections 2024-03-28T14:09:00.054 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:00.042+0000 7fc6635fa640 1 -- 172.21.15.104:0/2315875801 wait complete. 2024-03-28T14:09:00.055 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:00.042+0000 7fc6635fa640 1 Processor -- start 2024-03-28T14:09:00.055 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:00.042+0000 7fc6635fa640 1 -- start start 2024-03-28T14:09:00.055 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:00.042+0000 7fc6635fa640 1 --2- >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fc644004810 0x7fc65c09ae30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:09:00.055 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:00.042+0000 7fc6635fa640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc65c09e1e0 0x7fc65c09e660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:09:00.055 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:00.042+0000 7fc6635fa640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fc65c1de8a0 0x7fc65c1ded20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:09:00.055 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:00.042+0000 7fc6635fa640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7fc65c150880 con 0x7fc65c09e1e0 2024-03-28T14:09:00.055 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:00.042+0000 7fc6635fa640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7fc65c13fd30 con 0x7fc65c1de8a0 2024-03-28T14:09:00.055 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:00.042+0000 7fc6635fa640 1 -- --> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] -- mon_getmap magic: 0 -- 0x7fc65c150a00 con 0x7fc644004810 2024-03-28T14:09:00.056 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:00.042+0000 7fc660b95640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc65c09e1e0 0x7fc65c09e660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:09:00.056 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:00.042+0000 7fc660b95640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc65c09e1e0 0x7fc65c09e660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:49782/0 (socket says 172.21.15.104:49782) 2024-03-28T14:09:00.056 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:00.042+0000 7fc660b95640 1 -- 172.21.15.104:0/1002074100 learned_addr learned my addr 172.21.15.104:0/1002074100 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-03-28T14:09:00.056 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:00.042+0000 7fc661396640 1 --2- 172.21.15.104:0/1002074100 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fc644004810 0x7fc65c09ae30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:09:00.056 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:00.042+0000 7fc661b97640 1 --2- 172.21.15.104:0/1002074100 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fc65c1de8a0 0x7fc65c1ded20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:09:00.056 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:00.042+0000 7fc660b95640 1 -- 172.21.15.104:0/1002074100 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fc644004810 msgr2=0x7fc65c09ae30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:09:00.056 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:00.042+0000 7fc660b95640 1 --2- 172.21.15.104:0/1002074100 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fc644004810 0x7fc65c09ae30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:09:00.056 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:00.042+0000 7fc660b95640 1 -- 172.21.15.104:0/1002074100 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fc65c1de8a0 msgr2=0x7fc65c1ded20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:09:00.056 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:00.042+0000 7fc660b95640 1 --2- 172.21.15.104:0/1002074100 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fc65c1de8a0 0x7fc65c1ded20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:09:00.057 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:00.042+0000 7fc660b95640 1 -- 172.21.15.104:0/1002074100 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc65c09c530 con 0x7fc65c09e1e0 2024-03-28T14:09:00.057 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:00.042+0000 7fc660b95640 1 --2- 172.21.15.104:0/1002074100 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc65c09e1e0 0x7fc65c09e660 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7fc65001d120 tx=0x7fc644005840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:09:00.057 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:00.042+0000 7fc659ffb640 1 -- 172.21.15.104:0/1002074100 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6500bb070 con 0x7fc65c09e1e0 2024-03-28T14:09:00.057 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:00.042+0000 7fc659ffb640 1 -- 172.21.15.104:0/1002074100 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc6500ad430 con 0x7fc65c09e1e0 2024-03-28T14:09:00.057 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:00.042+0000 7fc659ffb640 1 -- 172.21.15.104:0/1002074100 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6500019e0 con 0x7fc65c09e1e0 2024-03-28T14:09:00.057 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:00.042+0000 7fc6635fa640 1 -- 172.21.15.104:0/1002074100 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc65c09ca80 con 0x7fc65c09e1e0 2024-03-28T14:09:00.057 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:00.042+0000 7fc6635fa640 1 -- 172.21.15.104:0/1002074100 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc65c142a30 con 0x7fc65c09e1e0 2024-03-28T14:09:00.058 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:00.046+0000 7fc6635fa640 1 -- 172.21.15.104:0/1002074100 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc65c0687c0 con 0x7fc65c09e1e0 2024-03-28T14:09:00.058 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:00.046+0000 7fc659ffb640 1 -- 172.21.15.104:0/1002074100 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) ==== 87803+0+0 (secure 0 0 0) 0x7fc6500be070 con 0x7fc65c09e1e0 2024-03-28T14:09:00.058 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:00.046+0000 7fc659ffb640 1 --2- 172.21.15.104:0/1002074100 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7fc648067fc0 0x7fc64806a480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:09:00.058 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:00.046+0000 7fc659ffb640 1 -- 172.21.15.104:0/1002074100 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(33..33 src has 1..33) ==== 4943+0+0 (secure 0 0 0) 0x7fc65011ffc0 con 0x7fc65c09e1e0 2024-03-28T14:09:00.058 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:00.046+0000 7fc661396640 1 --2- 172.21.15.104:0/1002074100 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7fc648067fc0 0x7fc64806a480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:09:00.058 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:00.046+0000 7fc661396640 1 --2- 172.21.15.104:0/1002074100 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7fc648067fc0 0x7fc64806a480 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fc654009ab0 tx=0x7fc654008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:09:00.058 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:00.050+0000 7fc659ffb640 1 -- 172.21.15.104:0/1002074100 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179117 (secure 0 0 0) 0x7fc650017030 con 0x7fc65c09e1e0 2024-03-28T14:09:00.211 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:00.206+0000 7fc6635fa640 1 -- 172.21.15.104:0/1002074100 --> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fc65c1431b0 con 0x7fc648067fc0 2024-03-28T14:09:00.213 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:00.214+0000 7fc659ffb640 1 -- 172.21.15.104:0/1002074100 <== mgr.4100 v2:172.21.15.171:6832/1522752412 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+13458 (secure 0 0 0) 0x7fc65c1431b0 con 0x7fc648067fc0 2024-03-28T14:09:00.213 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-03-28T14:09:00.214 INFO:teuthology.orchestra.run.smithi104.stderr:dumped all 2024-03-28T14:09:00.216 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:00.214+0000 7fc6635fa640 1 -- 172.21.15.104:0/1002074100 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7fc648067fc0 msgr2=0x7fc64806a480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:09:00.216 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:00.214+0000 7fc6635fa640 1 --2- 172.21.15.104:0/1002074100 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7fc648067fc0 0x7fc64806a480 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fc654009ab0 tx=0x7fc654008040 comp rx=0 tx=0).stop 2024-03-28T14:09:00.216 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:00.214+0000 7fc6635fa640 1 -- 172.21.15.104:0/1002074100 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc65c09e1e0 msgr2=0x7fc65c09e660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:09:00.216 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:00.214+0000 7fc6635fa640 1 --2- 172.21.15.104:0/1002074100 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc65c09e1e0 0x7fc65c09e660 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7fc65001d120 tx=0x7fc644005840 comp rx=0 tx=0).stop 2024-03-28T14:09:00.216 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:00.214+0000 7fc6635fa640 1 -- 172.21.15.104:0/1002074100 shutdown_connections 2024-03-28T14:09:00.216 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:00.214+0000 7fc6635fa640 1 --2- 172.21.15.104:0/1002074100 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fc65c1de8a0 0x7fc65c1ded20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:09:00.216 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:00.214+0000 7fc6635fa640 1 --2- 172.21.15.104:0/1002074100 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7fc648067fc0 0x7fc64806a480 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:09:00.216 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:00.214+0000 7fc6635fa640 1 --2- 172.21.15.104:0/1002074100 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc65c09e1e0 0x7fc65c09e660 unknown :-1 s=CLOSED pgs=436 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:09:00.217 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:00.214+0000 7fc6635fa640 1 --2- 172.21.15.104:0/1002074100 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fc644004810 0x7fc65c09ae30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:09:00.217 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:00.214+0000 7fc6635fa640 1 -- 172.21.15.104:0/1002074100 >> 172.21.15.104:0/1002074100 conn(0x7fc65c0b58a0 msgr2=0x7fc65c0b8280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T14:09:00.217 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:00.214+0000 7fc6635fa640 1 -- 172.21.15.104:0/1002074100 shutdown_connections 2024-03-28T14:09:00.217 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:00.214+0000 7fc6635fa640 1 -- 172.21.15.104:0/1002074100 wait complete. 2024-03-28T14:09:00.232 INFO:teuthology.orchestra.run.smithi104.stdout:{"pg_ready":true,"pg_map":{"version":63,"stamp":"2024-03-28T14:08:59.598545+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":220096,"kb_used_data":3932,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754754624,"statfs":{"total":773094113280,"available":772868734976,"internally_reserved":0,"allocated":4026368,"data_stored":2270761,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":221236785},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-10891,"num_objects":-22,"num_object_clones":0,"num_object_copies":-44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-55,"num_write_kb":-27,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":-14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.002292"},"pg_stats":[{"pgid":"1.0","version":"17'117","reported_seq":188,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-28T14:08:55.142304+0000","last_change":"2024-03-28T14:07:49.234448+0000","last_active":"2024-03-28T14:08:55.142304+0000","last_peered":"2024-03-28T14:08:55.142304+0000","last_clean":"2024-03-28T14:08:55.142304+0000","last_became_active":"2024-03-28T14:07:49.233895+0000","last_became_peered":"2024-03-28T14:07:49.233895+0000","last_unstale":"2024-03-28T14:08:55.142304+0000","last_undegraded":"2024-03-28T14:08:55.142304+0000","last_fullsized":"2024-03-28T14:08:55.142304+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:07:48.069157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:07:48.069157+0000","last_clean_scrub_stamp":"2024-03-28T14:07:48.069157+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:11:39.977557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574866,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27944,"kb_used_data":924,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343896,"statfs":{"total":96636764160,"available":96608149504,"internally_reserved":0,"allocated":946176,"data_stored":725222,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574867,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27360,"kb_used_data":336,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344480,"statfs":{"total":96636764160,"available":96608747520,"internally_reserved":0,"allocated":344064,"data_stored":129299,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574865,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27360,"kb_used_data":336,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344480,"statfs":{"total":96636764160,"available":96608747520,"internally_reserved":0,"allocated":344064,"data_stored":129355,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574866,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27368,"kb_used_data":348,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344472,"statfs":{"total":96636764160,"available":96608739328,"internally_reserved":0,"allocated":356352,"data_stored":134868,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574868,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27372,"kb_used_data":356,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344468,"statfs":{"total":96636764160,"available":96608735232,"internally_reserved":0,"allocated":364544,"data_stored":142985,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574866,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27364,"kb_used_data":344,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344476,"statfs":{"total":96636764160,"available":96608743424,"internally_reserved":0,"allocated":352256,"data_stored":134834,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574864,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27388,"kb_used_data":368,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344452,"statfs":{"total":96636764160,"available":96608718848,"internally_reserved":0,"allocated":376832,"data_stored":149066,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574867,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27940,"kb_used_data":920,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343900,"statfs":{"total":96636764160,"available":96608153600,"internally_reserved":0,"allocated":942080,"data_stored":725132,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-28T14:09:00.233 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-03-28T14:09:00.233 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-28T14:09:00.351 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:00.346+0000 7f1230594640 1 Processor -- start 2024-03-28T14:09:00.352 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:00.346+0000 7f1230594640 1 -- start start 2024-03-28T14:09:00.353 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:00.350+0000 7f1230594640 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f122806d970 0x7f12280addc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:09:00.353 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:00.350+0000 7f1230594640 1 --2- >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f12280af930 0x7f1228148140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:09:00.353 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:00.350+0000 7f1230594640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f122806dd50 con 0x7f12280ae300 2024-03-28T14:09:00.353 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:00.350+0000 7f1230594640 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 -- 0x7f12280b5500 con 0x7f122806d970 2024-03-28T14:09:00.354 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:00.350+0000 7f1230594640 1 -- --> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] -- mon_getmap magic: 0 -- 0x7f12280b02a0 con 0x7f12280af930 2024-03-28T14:09:00.354 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:00.350+0000 7f122e330640 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f122806d970 0x7f12280addc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:09:00.354 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:00.350+0000 7f122eb31640 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f12280ae300 0x7f12280b1c30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:40536/0 (socket says 172.21.15.104:40536) 2024-03-28T14:09:00.354 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:00.350+0000 7f122eb31640 1 -- 172.21.15.104:0/3633970457 learned_addr learned my addr 172.21.15.104:0/3633970457 (peer_addr_for_me v1:172.21.15.104:0/0) 2024-03-28T14:09:00.354 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:00.350+0000 7f122db2f640 1 --2- 172.21.15.104:0/3633970457 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f12280af930 0x7f1228148140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:09:00.355 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:00.350+0000 7f122e330640 1 -- 172.21.15.104:0/3633970457 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f12280af930 msgr2=0x7f1228148140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T14:09:00.355 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:00.350+0000 7f122e330640 1 --2- 172.21.15.104:0/3633970457 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f12280af930 0x7f1228148140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:09:00.355 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:00.350+0000 7f122e330640 1 -- 172.21.15.104:0/3633970457 >> v1:172.21.15.104:6789/0 conn(0x7f12280ae300 legacy=0x7f12280b1c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:09:00.355 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:00.350+0000 7f122d32e640 1 -- 172.21.15.104:0/3633970457 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3527235890 0 0) 0x7f122806dd50 con 0x7f12280ae300 2024-03-28T14:09:00.355 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:00.350+0000 7f122e330640 1 -- 172.21.15.104:0/3633970457 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f12280b17a0 con 0x7f122806d970 2024-03-28T14:09:00.356 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:00.350+0000 7f122e330640 1 --2- 172.21.15.104:0/3633970457 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f122806d970 0x7f12280addc0 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7f12240097c0 tx=0x7f122401c4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=952b38d83c99a9e server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:09:00.356 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:00.350+0000 7f122d32e640 1 -- 172.21.15.104:0/3633970457 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f12240b0070 con 0x7f122806d970 2024-03-28T14:09:00.357 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:00.350+0000 7f122d32e640 1 -- 172.21.15.104:0/3633970457 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f12240ab600 con 0x7f122806d970 2024-03-28T14:09:00.357 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:00.350+0000 7f122d32e640 1 -- 172.21.15.104:0/3633970457 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f12240ab9d0 con 0x7f122806d970 2024-03-28T14:09:00.357 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:00.354+0000 7f1230594640 1 -- 172.21.15.104:0/3633970457 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f122806d970 msgr2=0x7f12280addc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:09:00.357 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:00.354+0000 7f1230594640 1 --2- 172.21.15.104:0/3633970457 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f122806d970 0x7f12280addc0 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7f12240097c0 tx=0x7f122401c4a0 comp rx=0 tx=0).stop 2024-03-28T14:09:00.357 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:00.354+0000 7f1230594640 1 -- 172.21.15.104:0/3633970457 shutdown_connections 2024-03-28T14:09:00.357 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:00.354+0000 7f1230594640 1 --2- 172.21.15.104:0/3633970457 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f12280af930 0x7f1228148140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:09:00.357 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:00.354+0000 7f1230594640 1 --2- 172.21.15.104:0/3633970457 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f122806d970 0x7f12280addc0 unknown :-1 s=CLOSED pgs=438 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:09:00.357 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:00.354+0000 7f1230594640 1 -- 172.21.15.104:0/3633970457 >> 172.21.15.104:0/3633970457 conn(0x7f12280b58a0 msgr2=0x7f12280b5cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T14:09:00.358 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:00.354+0000 7f1230594640 1 -- 172.21.15.104:0/3633970457 shutdown_connections 2024-03-28T14:09:00.358 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:00.354+0000 7f1230594640 1 -- 172.21.15.104:0/3633970457 wait complete. 2024-03-28T14:09:00.360 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:00.354+0000 7f1230594640 1 Processor -- start 2024-03-28T14:09:00.360 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:00.354+0000 7f1230594640 1 -- start start 2024-03-28T14:09:00.361 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:00.354+0000 7f1230594640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f122806d970 0x7f1228071d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:09:00.361 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:00.354+0000 7f1230594640 1 --2- >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f12280ae300 0x7f12281e10a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:09:00.361 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:00.354+0000 7f1230594640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f12280af930 0x7f122813f040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:09:00.361 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:00.354+0000 7f1230594640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f1228151f40 con 0x7f12280af930 2024-03-28T14:09:00.361 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:00.354+0000 7f1230594640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f122809ed80 con 0x7f122806d970 2024-03-28T14:09:00.361 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:00.354+0000 7f1230594640 1 -- --> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] -- mon_getmap magic: 0 -- 0x7f12281520c0 con 0x7f12280ae300 2024-03-28T14:09:00.361 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:00.354+0000 7f122eb31640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f12280af930 0x7f122813f040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:09:00.361 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:00.354+0000 7f122eb31640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f12280af930 0x7f122813f040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:49794/0 (socket says 172.21.15.104:49794) 2024-03-28T14:09:00.362 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:00.354+0000 7f122eb31640 1 -- 172.21.15.104:0/3468242729 learned_addr learned my addr 172.21.15.104:0/3468242729 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-03-28T14:09:00.362 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:00.354+0000 7f122e330640 1 --2- 172.21.15.104:0/3468242729 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f122806d970 0x7f1228071d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:09:00.362 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:00.354+0000 7f122db2f640 1 --2- 172.21.15.104:0/3468242729 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f12280ae300 0x7f12281e10a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:09:00.362 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:00.354+0000 7f122eb31640 1 -- 172.21.15.104:0/3468242729 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f12280ae300 msgr2=0x7f12281e10a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:09:00.362 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:00.354+0000 7f122eb31640 1 --2- 172.21.15.104:0/3468242729 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f12280ae300 0x7f12281e10a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:09:00.362 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:00.354+0000 7f122eb31640 1 -- 172.21.15.104:0/3468242729 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f122806d970 msgr2=0x7f1228071d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:09:00.362 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:00.354+0000 7f122eb31640 1 --2- 172.21.15.104:0/3468242729 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f122806d970 0x7f1228071d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:09:00.362 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:00.354+0000 7f122eb31640 1 -- 172.21.15.104:0/3468242729 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f122809be00 con 0x7f12280af930 2024-03-28T14:09:00.362 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:00.354+0000 7f122e330640 1 --2- 172.21.15.104:0/3468242729 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f122806d970 0x7f1228071d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T14:09:00.363 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:00.354+0000 7f122eb31640 1 --2- 172.21.15.104:0/3468242729 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f12280af930 0x7f122813f040 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f122000c780 tx=0x7f122000cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:09:00.363 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:00.354+0000 7f121effd640 1 -- 172.21.15.104:0/3468242729 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1220018070 con 0x7f12280af930 2024-03-28T14:09:00.363 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:00.354+0000 7f121effd640 1 -- 172.21.15.104:0/3468242729 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f122000a640 con 0x7f12280af930 2024-03-28T14:09:00.363 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:00.354+0000 7f1230594640 1 -- 172.21.15.104:0/3468242729 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1228141d00 con 0x7f12280af930 2024-03-28T14:09:00.363 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:00.354+0000 7f121effd640 1 -- 172.21.15.104:0/3468242729 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f122000aab0 con 0x7f12280af930 2024-03-28T14:09:00.363 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:00.354+0000 7f1230594640 1 -- 172.21.15.104:0/3468242729 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f12281e08a0 con 0x7f12280af930 2024-03-28T14:09:00.363 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:00.354+0000 7f1230594640 1 -- 172.21.15.104:0/3468242729 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f12281e0690 con 0x7f12280af930 2024-03-28T14:09:00.363 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:00.358+0000 7f121effd640 1 -- 172.21.15.104:0/3468242729 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) ==== 87803+0+0 (secure 0 0 0) 0x7f122001b070 con 0x7f12280af930 2024-03-28T14:09:00.364 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:00.358+0000 7f121effd640 1 --2- 172.21.15.104:0/3468242729 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f120c067fc0 0x7f120c06a480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:09:00.364 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:00.358+0000 7f121effd640 1 -- 172.21.15.104:0/3468242729 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(33..33 src has 1..33) ==== 4943+0+0 (secure 0 0 0) 0x7f1220085730 con 0x7f12280af930 2024-03-28T14:09:00.364 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:00.358+0000 7f122e330640 1 --2- 172.21.15.104:0/3468242729 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f120c067fc0 0x7f120c06a480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:09:00.364 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:00.358+0000 7f122e330640 1 --2- 172.21.15.104:0/3468242729 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f120c067fc0 0x7f120c06a480 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f12240ad000 tx=0x7f1224008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:09:00.364 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:00.362+0000 7f121effd640 1 -- 172.21.15.104:0/3468242729 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179117 (secure 0 0 0) 0x7f1228141d00 con 0x7f12280af930 2024-03-28T14:09:00.522 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:00.518+0000 7f1230594640 1 -- 172.21.15.104:0/3468242729 --> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f12280687c0 con 0x7f120c067fc0 2024-03-28T14:09:00.524 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:00.522+0000 7f121effd640 1 -- 172.21.15.104:0/3468242729 <== mgr.4100 v2:172.21.15.171:6832/1522752412 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+13458 (secure 0 0 0) 0x7f12280687c0 con 0x7f120c067fc0 2024-03-28T14:09:00.525 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-03-28T14:09:00.525 INFO:teuthology.orchestra.run.smithi104.stderr:dumped all 2024-03-28T14:09:00.527 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:00.522+0000 7f1230594640 1 -- 172.21.15.104:0/3468242729 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f120c067fc0 msgr2=0x7f120c06a480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:09:00.527 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:00.522+0000 7f1230594640 1 --2- 172.21.15.104:0/3468242729 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f120c067fc0 0x7f120c06a480 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f12240ad000 tx=0x7f1224008040 comp rx=0 tx=0).stop 2024-03-28T14:09:00.527 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:00.522+0000 7f1230594640 1 -- 172.21.15.104:0/3468242729 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f12280af930 msgr2=0x7f122813f040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:09:00.527 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:00.522+0000 7f1230594640 1 --2- 172.21.15.104:0/3468242729 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f12280af930 0x7f122813f040 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f122000c780 tx=0x7f122000cc40 comp rx=0 tx=0).stop 2024-03-28T14:09:00.527 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:00.522+0000 7f1230594640 1 -- 172.21.15.104:0/3468242729 shutdown_connections 2024-03-28T14:09:00.527 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:00.522+0000 7f1230594640 1 --2- 172.21.15.104:0/3468242729 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f120c067fc0 0x7f120c06a480 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:09:00.528 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:00.522+0000 7f1230594640 1 --2- 172.21.15.104:0/3468242729 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f12280af930 0x7f122813f040 unknown :-1 s=CLOSED pgs=439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:09:00.528 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:00.522+0000 7f1230594640 1 --2- 172.21.15.104:0/3468242729 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f12280ae300 0x7f12281e10a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:09:00.528 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:00.522+0000 7f1230594640 1 --2- 172.21.15.104:0/3468242729 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f122806d970 0x7f1228071d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:09:00.528 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:00.522+0000 7f1230594640 1 -- 172.21.15.104:0/3468242729 >> 172.21.15.104:0/3468242729 conn(0x7f12280b58a0 msgr2=0x7f1228067330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T14:09:00.528 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:00.522+0000 7f1230594640 1 -- 172.21.15.104:0/3468242729 shutdown_connections 2024-03-28T14:09:00.528 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:00.522+0000 7f1230594640 1 -- 172.21.15.104:0/3468242729 wait complete. 2024-03-28T14:09:00.543 INFO:teuthology.orchestra.run.smithi104.stdout:{"pg_ready":true,"pg_map":{"version":63,"stamp":"2024-03-28T14:08:59.598545+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":220096,"kb_used_data":3932,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754754624,"statfs":{"total":773094113280,"available":772868734976,"internally_reserved":0,"allocated":4026368,"data_stored":2270761,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":221236785},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-10891,"num_objects":-22,"num_object_clones":0,"num_object_copies":-44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-55,"num_write_kb":-27,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":-14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.002292"},"pg_stats":[{"pgid":"1.0","version":"17'117","reported_seq":188,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-28T14:08:55.142304+0000","last_change":"2024-03-28T14:07:49.234448+0000","last_active":"2024-03-28T14:08:55.142304+0000","last_peered":"2024-03-28T14:08:55.142304+0000","last_clean":"2024-03-28T14:08:55.142304+0000","last_became_active":"2024-03-28T14:07:49.233895+0000","last_became_peered":"2024-03-28T14:07:49.233895+0000","last_unstale":"2024-03-28T14:08:55.142304+0000","last_undegraded":"2024-03-28T14:08:55.142304+0000","last_fullsized":"2024-03-28T14:08:55.142304+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:07:48.069157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:07:48.069157+0000","last_clean_scrub_stamp":"2024-03-28T14:07:48.069157+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:11:39.977557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574866,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27944,"kb_used_data":924,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343896,"statfs":{"total":96636764160,"available":96608149504,"internally_reserved":0,"allocated":946176,"data_stored":725222,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574867,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27360,"kb_used_data":336,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344480,"statfs":{"total":96636764160,"available":96608747520,"internally_reserved":0,"allocated":344064,"data_stored":129299,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574865,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27360,"kb_used_data":336,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344480,"statfs":{"total":96636764160,"available":96608747520,"internally_reserved":0,"allocated":344064,"data_stored":129355,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574866,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27368,"kb_used_data":348,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344472,"statfs":{"total":96636764160,"available":96608739328,"internally_reserved":0,"allocated":356352,"data_stored":134868,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574868,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27372,"kb_used_data":356,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344468,"statfs":{"total":96636764160,"available":96608735232,"internally_reserved":0,"allocated":364544,"data_stored":142985,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574866,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27364,"kb_used_data":344,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344476,"statfs":{"total":96636764160,"available":96608743424,"internally_reserved":0,"allocated":352256,"data_stored":134834,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574864,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27388,"kb_used_data":368,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344452,"statfs":{"total":96636764160,"available":96608718848,"internally_reserved":0,"allocated":376832,"data_stored":149066,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574867,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27940,"kb_used_data":920,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343900,"statfs":{"total":96636764160,"available":96608153600,"internally_reserved":0,"allocated":942080,"data_stored":725132,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-28T14:09:00.544 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-28T14:09:00.660 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:00.658+0000 7f466c672640 1 Processor -- start 2024-03-28T14:09:00.661 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:00.658+0000 7f466c672640 1 -- start start 2024-03-28T14:09:00.663 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:00.658+0000 7f466c672640 1 --2- >> v2:172.21.15.171:3300/0 conn(0x7f466413faf0 0x7f46640b0c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:09:00.663 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:00.658+0000 7f466c672640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f466406dd50 con 0x7f466406d970 2024-03-28T14:09:00.663 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:00.658+0000 7f466c672640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f46640b2d30 con 0x7f46640732a0 2024-03-28T14:09:00.664 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:00.658+0000 7f466c672640 1 -- --> v2:172.21.15.171:3300/0 -- mon_getmap magic: 0 -- 0x7f46640b02a0 con 0x7f466413faf0 2024-03-28T14:09:00.664 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:00.658+0000 7f466a40e640 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f466406d970 0x7f466413f3e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:40538/0 (socket says 172.21.15.104:40538) 2024-03-28T14:09:00.664 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:00.658+0000 7f466a40e640 1 -- 172.21.15.104:0/2545803112 learned_addr learned my addr 172.21.15.104:0/2545803112 (peer_addr_for_me v1:172.21.15.104:0/0) 2024-03-28T14:09:00.664 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:00.658+0000 7f4669c0d640 1 --2- 172.21.15.104:0/2545803112 >> v2:172.21.15.171:3300/0 conn(0x7f466413faf0 0x7f46640b0c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:09:00.664 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:00.658+0000 7f466940c640 1 -- 172.21.15.104:0/2545803112 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1248547777 0 0) 0x7f466406dd50 con 0x7f466406d970 2024-03-28T14:09:00.664 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:00.658+0000 7f466940c640 1 -- 172.21.15.104:0/2545803112 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4650003530 con 0x7f466406d970 2024-03-28T14:09:00.664 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:00.658+0000 7f466940c640 1 -- 172.21.15.104:0/2545803112 <== mon.0 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3349196626 0 0) 0x7f4650003530 con 0x7f466406d970 2024-03-28T14:09:00.664 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:00.658+0000 7f466940c640 1 -- 172.21.15.104:0/2545803112 >> v2:172.21.15.171:3300/0 conn(0x7f466413faf0 msgr2=0x7f46640b0c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T14:09:00.665 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:00.658+0000 7f466940c640 1 --2- 172.21.15.104:0/2545803112 >> v2:172.21.15.171:3300/0 conn(0x7f466413faf0 0x7f46640b0c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:09:00.665 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:00.658+0000 7f466940c640 1 -- 172.21.15.104:0/2545803112 >> v1:172.21.15.171:6789/0 conn(0x7f46640732a0 legacy=0x7f466414fe60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:09:00.665 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:00.658+0000 7f466940c640 1 -- 172.21.15.104:0/2545803112 --> v1:172.21.15.104:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f46640b3010 con 0x7f466406d970 2024-03-28T14:09:00.665 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:00.658+0000 7f466940c640 1 -- 172.21.15.104:0/2545803112 <== mon.0 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3745580849 0 0) 0x7f4660001680 con 0x7f466406d970 2024-03-28T14:09:00.665 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:00.658+0000 7f466940c640 1 -- 172.21.15.104:0/2545803112 >> v1:172.21.15.104:6789/0 conn(0x7f466406d970 legacy=0x7f466413f3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:09:00.665 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:00.658+0000 7f466940c640 1 --2- 172.21.15.104:0/2545803112 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f46500039c0 0x7f4650003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:09:00.665 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:00.658+0000 7f466940c640 1 --2- 172.21.15.104:0/2545803112 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4650004810 0x7f4650004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:09:00.665 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:00.658+0000 7f466940c640 1 --2- 172.21.15.104:0/2545803112 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f46500051d0 0x7f465000b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:09:00.666 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:00.658+0000 7f466940c640 1 -- 172.21.15.104:0/2545803112 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f46640b02a0 con 0x7f4650004810 2024-03-28T14:09:00.666 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:00.658+0000 7f466940c640 1 -- 172.21.15.104:0/2545803112 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f465000c170 con 0x7f46500039c0 2024-03-28T14:09:00.666 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:00.658+0000 7f466940c640 1 -- 172.21.15.104:0/2545803112 --> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] -- mon_getmap magic: 0 -- 0x7f465000c370 con 0x7f46500051d0 2024-03-28T14:09:00.666 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:00.658+0000 7f4669c0d640 1 --2- 172.21.15.104:0/2545803112 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4650004810 0x7f4650004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:09:00.666 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:00.658+0000 7f466a40e640 1 --2- 172.21.15.104:0/2545803112 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f46500039c0 0x7f4650003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:09:00.666 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:00.658+0000 7f466ac0f640 1 --2- 172.21.15.104:0/2545803112 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f46500051d0 0x7f465000b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:09:00.666 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:00.658+0000 7f4669c0d640 1 -- 172.21.15.104:0/2545803112 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f46500051d0 msgr2=0x7f465000b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T14:09:00.666 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:00.658+0000 7f4669c0d640 1 --2- 172.21.15.104:0/2545803112 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f46500051d0 0x7f465000b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:09:00.667 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:00.658+0000 7f4669c0d640 1 -- 172.21.15.104:0/2545803112 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f46500039c0 msgr2=0x7f4650003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T14:09:00.667 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:00.658+0000 7f4669c0d640 1 --2- 172.21.15.104:0/2545803112 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f46500039c0 0x7f4650003de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:09:00.667 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:00.658+0000 7f466ac0f640 1 -- 172.21.15.104:0/2545803112 reap_dead start 2024-03-28T14:09:00.667 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:00.658+0000 7f4669c0d640 1 -- 172.21.15.104:0/2545803112 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4650003530 con 0x7f4650004810 2024-03-28T14:09:00.671 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:00.662+0000 7f4669c0d640 1 --2- 172.21.15.104:0/2545803112 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4650004810 0x7f4650004c90 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7f466409eb70 tx=0x7f465401ca00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b3582485f22de5e3 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:09:00.671 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:00.662+0000 7f466940c640 1 -- 172.21.15.104:0/2545803112 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46540b2070 con 0x7f4650004810 2024-03-28T14:09:00.671 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:00.662+0000 7f466940c640 1 -- 172.21.15.104:0/2545803112 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f46540a55b0 con 0x7f4650004810 2024-03-28T14:09:00.671 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:00.662+0000 7f466940c640 1 -- 172.21.15.104:0/2545803112 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46540096b0 con 0x7f4650004810 2024-03-28T14:09:00.671 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:00.662+0000 7f466c672640 1 -- 172.21.15.104:0/2545803112 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4650004810 msgr2=0x7f4650004c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:09:00.671 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:00.662+0000 7f466c672640 1 --2- 172.21.15.104:0/2545803112 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4650004810 0x7f4650004c90 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7f466409eb70 tx=0x7f465401ca00 comp rx=0 tx=0).stop 2024-03-28T14:09:00.671 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:00.662+0000 7f466c672640 1 -- 172.21.15.104:0/2545803112 shutdown_connections 2024-03-28T14:09:00.672 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:00.662+0000 7f466c672640 1 --2- 172.21.15.104:0/2545803112 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4650004810 0x7f4650004c90 unknown :-1 s=CLOSED pgs=441 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:09:00.672 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:00.662+0000 7f466c672640 1 -- 172.21.15.104:0/2545803112 >> 172.21.15.104:0/2545803112 conn(0x7f46640b58a0 msgr2=0x7f46640b5cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T14:09:00.672 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:00.662+0000 7f466c672640 1 -- 172.21.15.104:0/2545803112 shutdown_connections 2024-03-28T14:09:00.672 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:00.662+0000 7f466c672640 1 -- 172.21.15.104:0/2545803112 wait complete. 2024-03-28T14:09:00.672 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:00.662+0000 7f466c672640 1 Processor -- start 2024-03-28T14:09:00.672 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:00.662+0000 7f466c672640 1 -- start start 2024-03-28T14:09:00.672 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:00.662+0000 7f466c672640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4650004810 0x7f4664071140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:09:00.672 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:00.662+0000 7f466c672640 1 --2- >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f46640addc0 0x7f46641e3f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:09:00.672 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:00.662+0000 7f466c672640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f46641e44b0 0x7f46641e4960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:09:00.673 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:00.662+0000 7f466c672640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f46640b15c0 con 0x7f4650004810 2024-03-28T14:09:00.673 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:00.662+0000 7f466c672640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f46640b1440 con 0x7f46641e44b0 2024-03-28T14:09:00.673 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:00.662+0000 7f466c672640 1 -- --> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] -- mon_getmap magic: 0 -- 0x7f46640b37a0 con 0x7f46640addc0 2024-03-28T14:09:00.673 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:00.662+0000 7f466a40e640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4650004810 0x7f4664071140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:09:00.673 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:00.662+0000 7f466a40e640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4650004810 0x7f4664071140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:49818/0 (socket says 172.21.15.104:49818) 2024-03-28T14:09:00.673 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:00.662+0000 7f466a40e640 1 -- 172.21.15.104:0/807925702 learned_addr learned my addr 172.21.15.104:0/807925702 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-03-28T14:09:00.673 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:00.662+0000 7f466ac0f640 1 --2- 172.21.15.104:0/807925702 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f46641e44b0 0x7f46641e4960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:09:00.673 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:00.662+0000 7f4669c0d640 1 --2- 172.21.15.104:0/807925702 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f46640addc0 0x7f46641e3f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:09:00.674 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:00.662+0000 7f466a40e640 1 -- 172.21.15.104:0/807925702 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f46640addc0 msgr2=0x7f46641e3f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:09:00.674 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:00.662+0000 7f466a40e640 1 --2- 172.21.15.104:0/807925702 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f46640addc0 0x7f46641e3f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:09:00.674 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:00.662+0000 7f466a40e640 1 -- 172.21.15.104:0/807925702 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f46641e44b0 msgr2=0x7f46641e4960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:09:00.674 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:00.662+0000 7f466a40e640 1 --2- 172.21.15.104:0/807925702 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f46641e44b0 0x7f46641e4960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:09:00.674 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:00.662+0000 7f466a40e640 1 -- 172.21.15.104:0/807925702 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4664149a60 con 0x7f4650004810 2024-03-28T14:09:00.674 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:00.666+0000 7f466a40e640 1 --2- 172.21.15.104:0/807925702 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4650004810 0x7f4664071140 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f466000e7b0 tx=0x7f466000ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:09:00.674 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:00.666+0000 7f465affd640 1 -- 172.21.15.104:0/807925702 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f466000ee60 con 0x7f4650004810 2024-03-28T14:09:00.674 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:00.666+0000 7f465affd640 1 -- 172.21.15.104:0/807925702 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4660018070 con 0x7f4650004810 2024-03-28T14:09:00.675 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:00.666+0000 7f466c672640 1 -- 172.21.15.104:0/807925702 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f46640af6c0 con 0x7f4650004810 2024-03-28T14:09:00.675 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:00.666+0000 7f465affd640 1 -- 172.21.15.104:0/807925702 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f466000cb30 con 0x7f4650004810 2024-03-28T14:09:00.675 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:00.666+0000 7f466c672640 1 -- 172.21.15.104:0/807925702 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4664070a70 con 0x7f4650004810 2024-03-28T14:09:00.675 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:00.666+0000 7f466c672640 1 -- 172.21.15.104:0/807925702 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f46640687c0 con 0x7f4650004810 2024-03-28T14:09:00.675 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:00.666+0000 7f465affd640 1 -- 172.21.15.104:0/807925702 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) ==== 87803+0+0 (secure 0 0 0) 0x7f466000ccd0 con 0x7f4650004810 2024-03-28T14:09:00.675 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:00.666+0000 7f465affd640 1 --2- 172.21.15.104:0/807925702 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f4648067ff0 0x7f464806a4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:09:00.675 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:00.666+0000 7f465affd640 1 -- 172.21.15.104:0/807925702 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(33..33 src has 1..33) ==== 4943+0+0 (secure 0 0 0) 0x7f4660085c80 con 0x7f4650004810 2024-03-28T14:09:00.675 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:00.666+0000 7f4669c0d640 1 --2- 172.21.15.104:0/807925702 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f4648067ff0 0x7f464806a4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:09:00.675 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:00.666+0000 7f4669c0d640 1 --2- 172.21.15.104:0/807925702 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f4648067ff0 0x7f464806a4b0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f46540af000 tx=0x7f4654008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:09:00.676 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:00.670+0000 7f465affd640 1 -- 172.21.15.104:0/807925702 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179117 (secure 0 0 0) 0x7f46600567c0 con 0x7f4650004810 2024-03-28T14:09:00.832 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:00.830+0000 7f466c672640 1 -- 172.21.15.104:0/807925702 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f4664144830 con 0x7f4650004810 2024-03-28T14:09:00.833 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:00.830+0000 7f465affd640 1 -- 172.21.15.104:0/807925702 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v33) ==== 74+0+13046 (secure 0 0 0) 0x7f46600560c0 con 0x7f4650004810 2024-03-28T14:09:00.834 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-03-28T14:09:00.834 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":33,"fsid":"428a90f2-3351-435f-8fcf-f7e66c78a13e","created":"2024-03-28T14:07:34.862458+0000","modified":"2024-03-28T14:08:57.936359+0000","last_up_change":"2024-03-28T14:07:46.028122+0000","last_in_change":"2024-03-28T14:07:37.157486+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-28T14:07:47.622988+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"1cb70540-4e42-402f-a446-aaec4ca83007","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6800","nonce":3745690918},{"type":"v1","addr":"172.21.15.104:6801","nonce":3745690918}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":3745690918},{"type":"v1","addr":"172.21.15.104:6803","nonce":3745690918}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":3745690918},{"type":"v1","addr":"172.21.15.104:6807","nonce":3745690918}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":3745690918},{"type":"v1","addr":"172.21.15.104:6805","nonce":3745690918}]},"public_addr":"172.21.15.104:6801/3745690918","cluster_addr":"172.21.15.104:6803/3745690918","heartbeat_back_addr":"172.21.15.104:6807/3745690918","heartbeat_front_addr":"172.21.15.104:6805/3745690918","state":["exists","up"]},{"osd":1,"uuid":"3ab7afdd-b980-418c-81f2-004e16de3dca","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":19438674},{"type":"v1","addr":"172.21.15.104:6810","nonce":19438674}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":19438674},{"type":"v1","addr":"172.21.15.104:6814","nonce":19438674}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":19438674},{"type":"v1","addr":"172.21.15.104:6823","nonce":19438674}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":19438674},{"type":"v1","addr":"172.21.15.104:6819","nonce":19438674}]},"public_addr":"172.21.15.104:6810/19438674","cluster_addr":"172.21.15.104:6814/19438674","heartbeat_back_addr":"172.21.15.104:6823/19438674","heartbeat_front_addr":"172.21.15.104:6819/19438674","state":["exists","up"]},{"osd":2,"uuid":"3e2651c8-ecae-4a9e-a053-27240a79eac2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":2352012626},{"type":"v1","addr":"172.21.15.104:6825","nonce":2352012626}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6826","nonce":2352012626},{"type":"v1","addr":"172.21.15.104:6827","nonce":2352012626}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":2352012626},{"type":"v1","addr":"172.21.15.104:6831","nonce":2352012626}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6828","nonce":2352012626},{"type":"v1","addr":"172.21.15.104:6829","nonce":2352012626}]},"public_addr":"172.21.15.104:6825/2352012626","cluster_addr":"172.21.15.104:6827/2352012626","heartbeat_back_addr":"172.21.15.104:6831/2352012626","heartbeat_front_addr":"172.21.15.104:6829/2352012626","state":["exists","up"]},{"osd":3,"uuid":"925c7e0a-38ef-47cd-b288-8bec8456fb17","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6809","nonce":3568347381},{"type":"v1","addr":"172.21.15.104:6811","nonce":3568347381}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6813","nonce":3568347381},{"type":"v1","addr":"172.21.15.104:6815","nonce":3568347381}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":3568347381},{"type":"v1","addr":"172.21.15.104:6822","nonce":3568347381}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6817","nonce":3568347381},{"type":"v1","addr":"172.21.15.104:6818","nonce":3568347381}]},"public_addr":"172.21.15.104:6811/3568347381","cluster_addr":"172.21.15.104:6815/3568347381","heartbeat_back_addr":"172.21.15.104:6822/3568347381","heartbeat_front_addr":"172.21.15.104:6818/3568347381","state":["exists","up"]},{"osd":4,"uuid":"4a8fac86-e595-4a0e-96e3-57edb3cc3f57","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6800","nonce":2107001077},{"type":"v1","addr":"172.21.15.171:6801","nonce":2107001077}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6802","nonce":2107001077},{"type":"v1","addr":"172.21.15.171:6803","nonce":2107001077}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6806","nonce":2107001077},{"type":"v1","addr":"172.21.15.171:6807","nonce":2107001077}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6804","nonce":2107001077},{"type":"v1","addr":"172.21.15.171:6805","nonce":2107001077}]},"public_addr":"172.21.15.171:6801/2107001077","cluster_addr":"172.21.15.171:6803/2107001077","heartbeat_back_addr":"172.21.15.171:6807/2107001077","heartbeat_front_addr":"172.21.15.171:6805/2107001077","state":["exists","up"]},{"osd":5,"uuid":"e48d50b0-beae-40ae-b15a-9a84dbfe88ab","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6808","nonce":2934826925},{"type":"v1","addr":"172.21.15.171:6810","nonce":2934826925}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6812","nonce":2934826925},{"type":"v1","addr":"172.21.15.171:6814","nonce":2934826925}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6819","nonce":2934826925},{"type":"v1","addr":"172.21.15.171:6821","nonce":2934826925}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6816","nonce":2934826925},{"type":"v1","addr":"172.21.15.171:6817","nonce":2934826925}]},"public_addr":"172.21.15.171:6810/2934826925","cluster_addr":"172.21.15.171:6814/2934826925","heartbeat_back_addr":"172.21.15.171:6821/2934826925","heartbeat_front_addr":"172.21.15.171:6817/2934826925","state":["exists","up"]},{"osd":6,"uuid":"5e7c3292-9936-4f35-b267-aed9d4e49323","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6824","nonce":436742919},{"type":"v1","addr":"172.21.15.171:6825","nonce":436742919}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6826","nonce":436742919},{"type":"v1","addr":"172.21.15.171:6827","nonce":436742919}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6830","nonce":436742919},{"type":"v1","addr":"172.21.15.171:6831","nonce":436742919}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6828","nonce":436742919},{"type":"v1","addr":"172.21.15.171:6829","nonce":436742919}]},"public_addr":"172.21.15.171:6825/436742919","cluster_addr":"172.21.15.171:6827/436742919","heartbeat_back_addr":"172.21.15.171:6831/436742919","heartbeat_front_addr":"172.21.15.171:6829/436742919","state":["exists","up"]},{"osd":7,"uuid":"339b8414-d7f0-4fb0-a446-efaf635cb7ff","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6809","nonce":2055631175},{"type":"v1","addr":"172.21.15.171:6811","nonce":2055631175}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6813","nonce":2055631175},{"type":"v1","addr":"172.21.15.171:6815","nonce":2055631175}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6822","nonce":2055631175},{"type":"v1","addr":"172.21.15.171:6823","nonce":2055631175}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6818","nonce":2055631175},{"type":"v1","addr":"172.21.15.171:6820","nonce":2055631175}]},"public_addr":"172.21.15.171:6811/2055631175","cluster_addr":"172.21.15.171:6815/2055631175","heartbeat_back_addr":"172.21.15.171:6823/2055631175","heartbeat_front_addr":"172.21.15.171:6820/2055631175","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T14:07:42.668881+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T14:07:42.708397+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T14:07:42.634191+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T14:07:42.733662+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T14:07:42.651541+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T14:07:42.783181+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T14:07:42.766310+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T14:07:42.670797+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.104:6833/1094990707":"2024-03-29T14:08:54.921407+0000","172.21.15.104:6832/1094990707":"2024-03-29T14:08:54.921407+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-28T14:09:00.836 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:00.830+0000 7f466c672640 1 -- 172.21.15.104:0/807925702 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f4648067ff0 msgr2=0x7f464806a4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:09:00.836 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:00.830+0000 7f466c672640 1 --2- 172.21.15.104:0/807925702 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f4648067ff0 0x7f464806a4b0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f46540af000 tx=0x7f4654008040 comp rx=0 tx=0).stop 2024-03-28T14:09:00.836 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:00.830+0000 7f466c672640 1 -- 172.21.15.104:0/807925702 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4650004810 msgr2=0x7f4664071140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:09:00.836 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:00.830+0000 7f466c672640 1 --2- 172.21.15.104:0/807925702 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4650004810 0x7f4664071140 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f466000e7b0 tx=0x7f466000ec70 comp rx=0 tx=0).stop 2024-03-28T14:09:00.836 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:00.830+0000 7f466c672640 1 -- 172.21.15.104:0/807925702 shutdown_connections 2024-03-28T14:09:00.836 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:00.830+0000 7f466c672640 1 --2- 172.21.15.104:0/807925702 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f46641e44b0 0x7f46641e4960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:09:00.836 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:00.830+0000 7f466c672640 1 --2- 172.21.15.104:0/807925702 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f46640addc0 0x7f46641e3f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:09:00.837 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:00.830+0000 7f466c672640 1 --2- 172.21.15.104:0/807925702 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f4648067ff0 0x7f464806a4b0 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:09:00.837 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:00.830+0000 7f466c672640 1 --2- 172.21.15.104:0/807925702 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4650004810 0x7f4664071140 unknown :-1 s=CLOSED pgs=442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:09:00.837 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:00.830+0000 7f466c672640 1 -- 172.21.15.104:0/807925702 >> 172.21.15.104:0/807925702 conn(0x7f46640b58a0 msgr2=0x7f466413bff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T14:09:00.837 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:00.830+0000 7f466c672640 1 -- 172.21.15.104:0/807925702 shutdown_connections 2024-03-28T14:09:00.837 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:00.830+0000 7f466c672640 1 -- 172.21.15.104:0/807925702 wait complete. 2024-03-28T14:09:01.852 INFO:tasks.ceph:Scrubbing osd.0 2024-03-28T14:09:01.853 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 config set osd_debug_deep_scrub_sleep 0 2024-03-28T14:09:01.990 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:01.986+0000 7fd561b5b640 1 Processor -- start 2024-03-28T14:09:01.991 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:01.986+0000 7fd561b5b640 1 -- start start 2024-03-28T14:09:01.992 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:01.986+0000 7fd561b5b640 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fd55c02c460 0x7fd55c13e510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:09:01.992 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:01.986+0000 7fd561b5b640 1 --2- >> v2:172.21.15.171:3300/0 conn(0x7fd55c13ea50 0x7fd55c144c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:09:01.992 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:01.986+0000 7fd561b5b640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd55c0b5230 con 0x7fd55c0b49f0 2024-03-28T14:09:01.992 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:01.986+0000 7fd561b5b640 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 -- 0x7fd55c0b5e60 con 0x7fd55c02c460 2024-03-28T14:09:01.992 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:01.986+0000 7fd561b5b640 1 -- --> v2:172.21.15.171:3300/0 -- mon_getmap magic: 0 -- 0x7fd55c1447c0 con 0x7fd55c13ea50 2024-03-28T14:09:01.993 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:01.986+0000 7fd55b7fe640 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fd55c02c460 0x7fd55c13e510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:09:01.993 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:01.986+0000 7fd55b7fe640 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fd55c02c460 0x7fd55c13e510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:49824/0 (socket says 172.21.15.104:49824) 2024-03-28T14:09:01.993 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:01.986+0000 7fd55b7fe640 1 -- 172.21.15.104:0/3730760145 learned_addr learned my addr 172.21.15.104:0/3730760145 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-03-28T14:09:01.993 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:01.990+0000 7fd55b7fe640 1 -- 172.21.15.104:0/3730760145 >> v2:172.21.15.171:3300/0 conn(0x7fd55c13ea50 msgr2=0x7fd55c144c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T14:09:01.993 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:01.990+0000 7fd55b7fe640 1 --2- 172.21.15.104:0/3730760145 >> v2:172.21.15.171:3300/0 conn(0x7fd55c13ea50 0x7fd55c144c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:09:01.993 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:01.990+0000 7fd55b7fe640 1 -- 172.21.15.104:0/3730760145 >> v1:172.21.15.171:6789/0 conn(0x7fd55c0b49f0 legacy=0x7fd55c147a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T14:09:01.993 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:01.990+0000 7fd55b7fe640 1 -- 172.21.15.104:0/3730760145 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd55c02c840 con 0x7fd55c02c460 2024-03-28T14:09:01.994 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:01.990+0000 7fd55b7fe640 1 --2- 172.21.15.104:0/3730760145 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd55c02c460 0x7fd55c13e510 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7fd55c070940 tx=0x7fd54c01eb20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4465c12457408d29 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:09:01.994 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:01.990+0000 7fd55affd640 1 -- 172.21.15.104:0/3730760145 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd54c0b4070 con 0x7fd55c02c460 2024-03-28T14:09:01.994 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:01.990+0000 7fd55affd640 1 -- 172.21.15.104:0/3730760145 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd54c0a79f0 con 0x7fd55c02c460 2024-03-28T14:09:01.994 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:01.990+0000 7fd55affd640 1 -- 172.21.15.104:0/3730760145 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd54c0a7dc0 con 0x7fd55c02c460 2024-03-28T14:09:01.995 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:01.990+0000 7fd561b5b640 1 -- 172.21.15.104:0/3730760145 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd55c02c460 msgr2=0x7fd55c13e510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:09:01.996 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:01.990+0000 7fd561b5b640 1 --2- 172.21.15.104:0/3730760145 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd55c02c460 0x7fd55c13e510 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7fd55c070940 tx=0x7fd54c01eb20 comp rx=0 tx=0).stop 2024-03-28T14:09:01.996 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:01.990+0000 7fd561b5b640 1 -- 172.21.15.104:0/3730760145 shutdown_connections 2024-03-28T14:09:01.996 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:01.990+0000 7fd561b5b640 1 --2- 172.21.15.104:0/3730760145 >> v2:172.21.15.171:3300/0 conn(0x7fd55c13ea50 0x7fd55c144c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:09:01.996 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:01.990+0000 7fd561b5b640 1 --2- 172.21.15.104:0/3730760145 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd55c02c460 0x7fd55c13e510 secure :-1 s=CLOSED pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7fd55c070940 tx=0x7fd54c01eb20 comp rx=0 tx=0).stop 2024-03-28T14:09:01.996 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:01.990+0000 7fd561b5b640 1 -- 172.21.15.104:0/3730760145 >> 172.21.15.104:0/3730760145 conn(0x7fd55c0b5ff0 msgr2=0x7fd55c0b63d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T14:09:01.996 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:01.990+0000 7fd561b5b640 1 -- 172.21.15.104:0/3730760145 shutdown_connections 2024-03-28T14:09:01.996 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:01.990+0000 7fd561b5b640 1 -- 172.21.15.104:0/3730760145 wait complete. 2024-03-28T14:09:01.996 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:01.994+0000 7fd561b5b640 1 Processor -- start 2024-03-28T14:09:01.997 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:01.994+0000 7fd561b5b640 1 -- start start 2024-03-28T14:09:01.999 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:01.994+0000 7fd561b5b640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd55c0b49f0 0x7fd55c1395f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:09:02.000 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:01.994+0000 7fd561b5b640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fd55c13ea50 0x7fd55c13b910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:09:02.000 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:01.994+0000 7fd561b5b640 1 --2- >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fd55c1dc0a0 0x7fd55c1377c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:09:02.000 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:01.994+0000 7fd561b5b640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7fd55c06cee0 con 0x7fd55c0b49f0 2024-03-28T14:09:02.000 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:01.994+0000 7fd561b5b640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7fd55c09eb20 con 0x7fd55c13ea50 2024-03-28T14:09:02.000 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:01.994+0000 7fd561b5b640 1 -- --> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] -- mon_getmap magic: 0 -- 0x7fd55c06d060 con 0x7fd55c1dc0a0 2024-03-28T14:09:02.000 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:01.994+0000 7fd55b7fe640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd55c0b49f0 0x7fd55c1395f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:09:02.000 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:01.994+0000 7fd55b7fe640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd55c0b49f0 0x7fd55c1395f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:49834/0 (socket says 172.21.15.104:49834) 2024-03-28T14:09:02.000 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:01.994+0000 7fd55b7fe640 1 -- 172.21.15.104:0/704052546 learned_addr learned my addr 172.21.15.104:0/704052546 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-03-28T14:09:02.001 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:01.994+0000 7fd55b7fe640 1 -- 172.21.15.104:0/704052546 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fd55c1dc0a0 msgr2=0x7fd55c1377c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:09:02.001 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:01.994+0000 7fd55b7fe640 1 --2- 172.21.15.104:0/704052546 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fd55c1dc0a0 0x7fd55c1377c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:09:02.001 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:01.994+0000 7fd55b7fe640 1 -- 172.21.15.104:0/704052546 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fd55c13ea50 msgr2=0x7fd55c13b910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:09:02.001 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:01.994+0000 7fd553fff640 1 --2- 172.21.15.104:0/704052546 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fd55c13ea50 0x7fd55c13b910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:09:02.001 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:01.994+0000 7fd55b7fe640 1 --2- 172.21.15.104:0/704052546 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fd55c13ea50 0x7fd55c13b910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:09:02.001 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:01.994+0000 7fd55b7fe640 1 -- 172.21.15.104:0/704052546 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd55c13ca00 con 0x7fd55c0b49f0 2024-03-28T14:09:02.001 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:01.994+0000 7fd55b7fe640 1 --2- 172.21.15.104:0/704052546 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd55c0b49f0 0x7fd55c1395f0 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7fd54c00b2c0 tx=0x7fd54c01b8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:09:02.001 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:01.994+0000 7fd558ff9640 1 -- 172.21.15.104:0/704052546 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd54c0be070 con 0x7fd55c0b49f0 2024-03-28T14:09:02.002 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:01.994+0000 7fd558ff9640 1 -- 172.21.15.104:0/704052546 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd54c0af840 con 0x7fd55c0b49f0 2024-03-28T14:09:02.002 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:01.994+0000 7fd558ff9640 1 -- 172.21.15.104:0/704052546 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd54c0b4040 con 0x7fd55c0b49f0 2024-03-28T14:09:02.002 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:01.994+0000 7fd561b5b640 1 -- 172.21.15.104:0/704052546 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd55c13cce0 con 0x7fd55c0b49f0 2024-03-28T14:09:02.002 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:01.994+0000 7fd561b5b640 1 -- 172.21.15.104:0/704052546 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd55c1476c0 con 0x7fd55c0b49f0 2024-03-28T14:09:02.002 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:01.994+0000 7fd561b5b640 1 -- 172.21.15.104:0/704052546 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fd55c0687c0 con 0x7fd55c0b49f0 2024-03-28T14:09:02.002 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:01.994+0000 7fd558ff9640 1 -- 172.21.15.104:0/704052546 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) ==== 87803+0+0 (secure 0 0 0) 0x7fd54c0c1070 con 0x7fd55c0b49f0 2024-03-28T14:09:02.002 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:01.994+0000 7fd558ff9640 1 --2- 172.21.15.104:0/704052546 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7fd53c067fc0 0x7fd53c06a480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:09:02.002 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:01.994+0000 7fd558ff9640 1 -- 172.21.15.104:0/704052546 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(33..33 src has 1..33) ==== 4943+0+0 (secure 0 0 0) 0x7fd54c1230e0 con 0x7fd55c0b49f0 2024-03-28T14:09:02.003 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:01.994+0000 7fd558ff9640 1 --2- 172.21.15.104:0/704052546 >> [v2:172.21.15.104:6800/3745690918,v1:172.21.15.104:6801/3745690918] conn(0x7fd53c06de80 0x7fd53c0702a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:09:02.003 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:01.994+0000 7fd558ff9640 1 -- 172.21.15.104:0/704052546 --> [v2:172.21.15.104:6800/3745690918,v1:172.21.15.104:6801/3745690918] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fd54c00d140 con 0x7fd53c06de80 2024-03-28T14:09:02.003 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:01.994+0000 7fd558ff9640 1 -- 172.21.15.104:0/704052546 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7fd54c12a050 con 0x7fd55c0b49f0 2024-03-28T14:09:02.003 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:01.994+0000 7fd55bfff640 1 --2- 172.21.15.104:0/704052546 >> [v2:172.21.15.104:6800/3745690918,v1:172.21.15.104:6801/3745690918] conn(0x7fd53c06de80 0x7fd53c0702a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:09:02.003 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:01.994+0000 7fd553fff640 1 --2- 172.21.15.104:0/704052546 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7fd53c067fc0 0x7fd53c06a480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:09:02.003 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:01.998+0000 7fd55bfff640 1 --2- 172.21.15.104:0/704052546 >> [v2:172.21.15.104:6800/3745690918,v1:172.21.15.104:6801/3745690918] conn(0x7fd53c06de80 0x7fd53c0702a0 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:09:02.004 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:01.998+0000 7fd553fff640 1 --2- 172.21.15.104:0/704052546 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7fd53c067fc0 0x7fd53c06a480 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fd55c14fd10 tx=0x7fd544008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:09:02.004 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:01.998+0000 7fd558ff9640 1 -- 172.21.15.104:0/704052546 <== osd.0 v2:172.21.15.104:6800/3745690918 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fd54c00d140 con 0x7fd53c06de80 2024-03-28T14:09:02.014 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:02.010+0000 7fd561b5b640 1 -- 172.21.15.104:0/704052546 --> [v2:172.21.15.104:6800/3745690918,v1:172.21.15.104:6801/3745690918] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7fd55c0b3920 con 0x7fd53c06de80 2024-03-28T14:09:02.014 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:02.010+0000 7fd558ff9640 1 -- 172.21.15.104:0/704052546 <== osd.0 v2:172.21.15.104:6800/3745690918 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7fd55c0b3920 con 0x7fd53c06de80 2024-03-28T14:09:02.014 INFO:teuthology.orchestra.run.smithi104.stdout:{ 2024-03-28T14:09:02.014 INFO:teuthology.orchestra.run.smithi104.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-03-28T14:09:02.014 INFO:teuthology.orchestra.run.smithi104.stdout:} 2024-03-28T14:09:02.015 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:02.010+0000 7fd561b5b640 1 -- 172.21.15.104:0/704052546 >> [v2:172.21.15.104:6800/3745690918,v1:172.21.15.104:6801/3745690918] conn(0x7fd53c06de80 msgr2=0x7fd53c0702a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:09:02.015 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:02.010+0000 7fd561b5b640 1 --2- 172.21.15.104:0/704052546 >> [v2:172.21.15.104:6800/3745690918,v1:172.21.15.104:6801/3745690918] conn(0x7fd53c06de80 0x7fd53c0702a0 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:09:02.016 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:02.010+0000 7fd561b5b640 1 -- 172.21.15.104:0/704052546 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7fd53c067fc0 msgr2=0x7fd53c06a480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:09:02.016 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:02.010+0000 7fd561b5b640 1 --2- 172.21.15.104:0/704052546 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7fd53c067fc0 0x7fd53c06a480 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fd55c14fd10 tx=0x7fd544008040 comp rx=0 tx=0).stop 2024-03-28T14:09:02.016 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:02.010+0000 7fd561b5b640 1 -- 172.21.15.104:0/704052546 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd55c0b49f0 msgr2=0x7fd55c1395f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:09:02.016 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:02.010+0000 7fd561b5b640 1 --2- 172.21.15.104:0/704052546 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd55c0b49f0 0x7fd55c1395f0 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7fd54c00b2c0 tx=0x7fd54c01b8c0 comp rx=0 tx=0).stop 2024-03-28T14:09:02.016 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:02.010+0000 7fd55bfff640 1 -- 172.21.15.104:0/704052546 reap_dead start 2024-03-28T14:09:02.017 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:02.010+0000 7fd561b5b640 1 -- 172.21.15.104:0/704052546 shutdown_connections 2024-03-28T14:09:02.017 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:02.010+0000 7fd561b5b640 1 -- 172.21.15.104:0/704052546 >> 172.21.15.104:0/704052546 conn(0x7fd55c0b5ff0 msgr2=0x7fd55c0b9d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T14:09:02.017 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:02.010+0000 7fd561b5b640 1 -- 172.21.15.104:0/704052546 shutdown_connections 2024-03-28T14:09:02.017 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:02.010+0000 7fd561b5b640 1 -- 172.21.15.104:0/704052546 wait complete. 2024-03-28T14:09:02.030 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 0 2024-03-28T14:09:02.142 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:02.138+0000 7ff6c8430640 1 Processor -- start 2024-03-28T14:09:02.142 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:02.138+0000 7ff6c8430640 1 -- start start 2024-03-28T14:09:02.143 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:02.138+0000 7ff6c8430640 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7ff6c006c8f0 0x7ff6c00ae0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:09:02.144 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:02.138+0000 7ff6c8430640 1 --2- >> v2:172.21.15.171:3300/0 conn(0x7ff6c014b340 0x7ff6c014b7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:09:02.144 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:02.138+0000 7ff6c8430640 1 --2- >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7ff6c00b3730 0x7ff6c00b3b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:09:02.144 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:02.138+0000 7ff6c8430640 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 -- 0x7ff6c00b0230 con 0x7ff6c006c8f0 2024-03-28T14:09:02.144 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:02.138+0000 7ff6c8430640 1 -- --> v2:172.21.15.171:3300/0 -- mon_getmap magic: 0 -- 0x7ff6c00b2e80 con 0x7ff6c014b340 2024-03-28T14:09:02.144 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:02.138+0000 7ff6c8430640 1 -- --> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] -- mon_getmap magic: 0 -- 0x7ff6c00af940 con 0x7ff6c00b3730 2024-03-28T14:09:02.144 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:02.138+0000 7ff6c61cc640 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7ff6c006c8f0 0x7ff6c00ae0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:09:02.144 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:02.138+0000 7ff6c61cc640 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7ff6c006c8f0 0x7ff6c00ae0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:49842/0 (socket says 172.21.15.104:49842) 2024-03-28T14:09:02.144 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:02.138+0000 7ff6c61cc640 1 -- 172.21.15.104:0/670918064 learned_addr learned my addr 172.21.15.104:0/670918064 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-03-28T14:09:02.145 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:02.138+0000 7ff6c69cd640 1 --2- 172.21.15.104:0/670918064 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7ff6c00b3730 0x7ff6c00b3b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:09:02.145 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:02.138+0000 7ff6c61cc640 1 -- 172.21.15.104:0/670918064 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7ff6c00b3730 msgr2=0x7ff6c00b3b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T14:09:02.145 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:02.138+0000 7ff6c61cc640 1 --2- 172.21.15.104:0/670918064 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7ff6c00b3730 0x7ff6c00b3b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:09:02.145 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:02.138+0000 7ff6c61cc640 1 -- 172.21.15.104:0/670918064 >> v2:172.21.15.171:3300/0 conn(0x7ff6c014b340 msgr2=0x7ff6c014b7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T14:09:02.145 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:02.138+0000 7ff6c59cb640 1 --2- 172.21.15.104:0/670918064 >> v2:172.21.15.171:3300/0 conn(0x7ff6c014b340 0x7ff6c014b7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:09:02.145 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:02.138+0000 7ff6c61cc640 1 --2- 172.21.15.104:0/670918064 >> v2:172.21.15.171:3300/0 conn(0x7ff6c014b340 0x7ff6c014b7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:09:02.145 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:02.138+0000 7ff6c61cc640 1 -- 172.21.15.104:0/670918064 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff6c006ccd0 con 0x7ff6c006c8f0 2024-03-28T14:09:02.146 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:02.142+0000 7ff6c61cc640 1 --2- 172.21.15.104:0/670918064 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff6c006c8f0 0x7ff6c00ae0f0 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7ff6c00aef30 tx=0x7ff6bc01c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6581c02c3cd9c115 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:09:02.147 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:02.142+0000 7ff6c51ca640 1 -- 172.21.15.104:0/670918064 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6bc0b0070 con 0x7ff6c006c8f0 2024-03-28T14:09:02.147 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:02.142+0000 7ff6c51ca640 1 -- 172.21.15.104:0/670918064 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff6bc0ab600 con 0x7ff6c006c8f0 2024-03-28T14:09:02.147 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:02.142+0000 7ff6c51ca640 1 -- 172.21.15.104:0/670918064 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6bc0ab9d0 con 0x7ff6c006c8f0 2024-03-28T14:09:02.147 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:02.142+0000 7ff6c8430640 1 -- 172.21.15.104:0/670918064 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff6c006c8f0 msgr2=0x7ff6c00ae0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:09:02.147 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:02.142+0000 7ff6c8430640 1 --2- 172.21.15.104:0/670918064 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff6c006c8f0 0x7ff6c00ae0f0 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7ff6c00aef30 tx=0x7ff6bc01c520 comp rx=0 tx=0).stop 2024-03-28T14:09:02.147 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:02.142+0000 7ff6c8430640 1 -- 172.21.15.104:0/670918064 shutdown_connections 2024-03-28T14:09:02.147 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:02.142+0000 7ff6c8430640 1 --2- 172.21.15.104:0/670918064 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7ff6c00b3730 0x7ff6c00b3b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:09:02.148 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:02.142+0000 7ff6c8430640 1 --2- 172.21.15.104:0/670918064 >> v2:172.21.15.171:3300/0 conn(0x7ff6c014b340 0x7ff6c014b7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:09:02.148 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:02.142+0000 7ff6c8430640 1 --2- 172.21.15.104:0/670918064 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff6c006c8f0 0x7ff6c00ae0f0 unknown :-1 s=CLOSED pgs=445 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:09:02.148 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:02.142+0000 7ff6c8430640 1 -- 172.21.15.104:0/670918064 >> 172.21.15.104:0/670918064 conn(0x7ff6c00b8f00 msgr2=0x7ff6c00b9310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T14:09:02.148 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:02.142+0000 7ff6c8430640 1 -- 172.21.15.104:0/670918064 shutdown_connections 2024-03-28T14:09:02.148 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:02.142+0000 7ff6c8430640 1 -- 172.21.15.104:0/670918064 wait complete. 2024-03-28T14:09:02.151 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:02.142+0000 7ff6c8430640 1 Processor -- start 2024-03-28T14:09:02.151 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:02.142+0000 7ff6c8430640 1 -- start start 2024-03-28T14:09:02.151 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:02.146+0000 7ff6c8430640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff6c006c8f0 0x7ff6c0071d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:09:02.151 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:02.146+0000 7ff6c8430640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7ff6c00b3730 0x7ff6c01e45f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:09:02.152 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:02.146+0000 7ff6c8430640 1 --2- >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7ff6c014b340 0x7ff6c009ab60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:09:02.152 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:02.146+0000 7ff6c8430640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7ff6c00aff50 con 0x7ff6c006c8f0 2024-03-28T14:09:02.152 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:02.146+0000 7ff6c8430640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7ff6c00afdd0 con 0x7ff6c00b3730 2024-03-28T14:09:02.152 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:02.146+0000 7ff6c8430640 1 -- --> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] -- mon_getmap magic: 0 -- 0x7ff6c00b4c20 con 0x7ff6c014b340 2024-03-28T14:09:02.152 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:02.146+0000 7ff6c61cc640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff6c006c8f0 0x7ff6c0071d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:09:02.152 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:02.146+0000 7ff6c61cc640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff6c006c8f0 0x7ff6c0071d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:49850/0 (socket says 172.21.15.104:49850) 2024-03-28T14:09:02.152 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:02.146+0000 7ff6c61cc640 1 -- 172.21.15.104:0/2918719552 learned_addr learned my addr 172.21.15.104:0/2918719552 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-03-28T14:09:02.152 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:02.146+0000 7ff6c59cb640 1 --2- 172.21.15.104:0/2918719552 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7ff6c00b3730 0x7ff6c01e45f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:09:02.152 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:02.146+0000 7ff6c61cc640 1 -- 172.21.15.104:0/2918719552 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7ff6c014b340 msgr2=0x7ff6c009ab60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:09:02.153 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:02.146+0000 7ff6c69cd640 1 --2- 172.21.15.104:0/2918719552 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7ff6c014b340 0x7ff6c009ab60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:09:02.153 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:02.146+0000 7ff6c61cc640 1 --2- 172.21.15.104:0/2918719552 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7ff6c014b340 0x7ff6c009ab60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:09:02.153 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:02.146+0000 7ff6c61cc640 1 -- 172.21.15.104:0/2918719552 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7ff6c00b3730 msgr2=0x7ff6c01e45f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:09:02.153 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:02.146+0000 7ff6c61cc640 1 --2- 172.21.15.104:0/2918719552 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7ff6c00b3730 0x7ff6c01e45f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:09:02.153 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:02.146+0000 7ff6c61cc640 1 -- 172.21.15.104:0/2918719552 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff6c009dc00 con 0x7ff6c006c8f0 2024-03-28T14:09:02.153 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:02.146+0000 7ff6c59cb640 1 --2- 172.21.15.104:0/2918719552 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7ff6c00b3730 0x7ff6c01e45f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T14:09:02.153 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:02.146+0000 7ff6c61cc640 1 --2- 172.21.15.104:0/2918719552 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff6c006c8f0 0x7ff6c0071d80 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7ff6bc0ae040 tx=0x7ff6bc01ad60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:09:02.153 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:02.146+0000 7ff6b6ffd640 1 -- 172.21.15.104:0/2918719552 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6bc0ba020 con 0x7ff6c006c8f0 2024-03-28T14:09:02.154 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:02.146+0000 7ff6b6ffd640 1 -- 172.21.15.104:0/2918719552 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff6bc0b6420 con 0x7ff6c006c8f0 2024-03-28T14:09:02.154 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:02.146+0000 7ff6b6ffd640 1 -- 172.21.15.104:0/2918719552 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6bc0b0040 con 0x7ff6c006c8f0 2024-03-28T14:09:02.154 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:02.146+0000 7ff6c8430640 1 -- 172.21.15.104:0/2918719552 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff6c009b590 con 0x7ff6c006c8f0 2024-03-28T14:09:02.154 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:02.146+0000 7ff6c8430640 1 -- 172.21.15.104:0/2918719552 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff6c01e40d0 con 0x7ff6c006c8f0 2024-03-28T14:09:02.154 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:02.146+0000 7ff6c8430640 1 -- 172.21.15.104:0/2918719552 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff6c00687c0 con 0x7ff6c006c8f0 2024-03-28T14:09:02.154 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:02.146+0000 7ff6b6ffd640 1 -- 172.21.15.104:0/2918719552 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) ==== 87803+0+0 (secure 0 0 0) 0x7ff6bc0bd020 con 0x7ff6c006c8f0 2024-03-28T14:09:02.154 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:02.146+0000 7ff6b6ffd640 1 --2- 172.21.15.104:0/2918719552 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7ff6a4067fc0 0x7ff6a406a480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:09:02.154 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:02.146+0000 7ff6b6ffd640 1 -- 172.21.15.104:0/2918719552 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(33..33 src has 1..33) ==== 4943+0+0 (secure 0 0 0) 0x7ff6bc1203e0 con 0x7ff6c006c8f0 2024-03-28T14:09:02.155 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:02.146+0000 7ff6c59cb640 1 --2- 172.21.15.104:0/2918719552 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7ff6a4067fc0 0x7ff6a406a480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:09:02.155 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:02.146+0000 7ff6c59cb640 1 --2- 172.21.15.104:0/2918719552 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7ff6a4067fc0 0x7ff6a406a480 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7ff6b0006fd0 tx=0x7ff6b0008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:09:02.155 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:02.150+0000 7ff6b6ffd640 1 -- 172.21.15.104:0/2918719552 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179117 (secure 0 0 0) 0x7ff6bc0c4030 con 0x7ff6c006c8f0 2024-03-28T14:09:02.314 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:02.310+0000 7ff6c8430640 1 -- 172.21.15.104:0/2918719552 --> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "0", "target": ["mon-mgr", ""]}) -- 0x7ff6c00b03c0 con 0x7ff6a4067fc0 2024-03-28T14:09:02.315 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:02.310+0000 7ff6b6ffd640 1 -- 172.21.15.104:0/2918719552 <== mgr.4100 v2:172.21.15.171:6832/1522752412 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 0 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7ff6c00b03c0 con 0x7ff6a4067fc0 2024-03-28T14:09:02.315 INFO:teuthology.orchestra.run.smithi104.stderr:instructed osd(s) 0 to deep-scrub 2024-03-28T14:09:02.317 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:02.314+0000 7ff6c8430640 1 -- 172.21.15.104:0/2918719552 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7ff6a4067fc0 msgr2=0x7ff6a406a480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:09:02.318 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:02.314+0000 7ff6c8430640 1 --2- 172.21.15.104:0/2918719552 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7ff6a4067fc0 0x7ff6a406a480 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7ff6b0006fd0 tx=0x7ff6b0008040 comp rx=0 tx=0).stop 2024-03-28T14:09:02.318 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:02.314+0000 7ff6c8430640 1 -- 172.21.15.104:0/2918719552 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff6c006c8f0 msgr2=0x7ff6c0071d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:09:02.318 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:02.314+0000 7ff6c8430640 1 --2- 172.21.15.104:0/2918719552 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff6c006c8f0 0x7ff6c0071d80 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7ff6bc0ae040 tx=0x7ff6bc01ad60 comp rx=0 tx=0).stop 2024-03-28T14:09:02.318 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:02.314+0000 7ff6c8430640 1 -- 172.21.15.104:0/2918719552 shutdown_connections 2024-03-28T14:09:02.318 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:02.314+0000 7ff6c8430640 1 --2- 172.21.15.104:0/2918719552 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7ff6c014b340 0x7ff6c009ab60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:09:02.318 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:02.314+0000 7ff6c8430640 1 --2- 172.21.15.104:0/2918719552 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7ff6c00b3730 0x7ff6c01e45f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:09:02.318 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:02.314+0000 7ff6c8430640 1 --2- 172.21.15.104:0/2918719552 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7ff6a4067fc0 0x7ff6a406a480 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:09:02.318 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:02.314+0000 7ff6c8430640 1 --2- 172.21.15.104:0/2918719552 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff6c006c8f0 0x7ff6c0071d80 unknown :-1 s=CLOSED pgs=446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:09:02.319 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:02.314+0000 7ff6c8430640 1 -- 172.21.15.104:0/2918719552 >> 172.21.15.104:0/2918719552 conn(0x7ff6c00b8f00 msgr2=0x7ff6c00b92e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T14:09:02.319 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:02.314+0000 7ff6c8430640 1 -- 172.21.15.104:0/2918719552 shutdown_connections 2024-03-28T14:09:02.319 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:02.314+0000 7ff6c8430640 1 -- 172.21.15.104:0/2918719552 wait complete. 2024-03-28T14:09:02.334 INFO:tasks.ceph:Scrubbing osd.1 2024-03-28T14:09:02.334 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 config set osd_debug_deep_scrub_sleep 0 2024-03-28T14:09:02.453 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:02.450+0000 7fc01a9c1640 1 Processor -- start 2024-03-28T14:09:02.453 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:02.450+0000 7fc01a9c1640 1 -- start start 2024-03-28T14:09:02.455 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:02.450+0000 7fc01a9c1640 1 --2- >> v2:172.21.15.171:3300/0 conn(0x7fc01406c8f0 0x7fc0140b2740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:09:02.455 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:02.450+0000 7fc01a9c1640 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fc0140b3730 0x7fc014148c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:09:02.455 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:02.450+0000 7fc01a9c1640 1 --2- >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fc0141491b0 0x7fc014149620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:09:02.455 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:02.450+0000 7fc01a9c1640 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 -- 0x7fc0140b0b30 con 0x7fc0140b3730 2024-03-28T14:09:02.455 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:02.450+0000 7fc01a9c1640 1 -- --> v2:172.21.15.171:3300/0 -- mon_getmap magic: 0 -- 0x7fc0140b22b0 con 0x7fc01406c8f0 2024-03-28T14:09:02.455 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:02.450+0000 7fc01a9c1640 1 -- --> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] -- mon_getmap magic: 0 -- 0x7fc0140b3b60 con 0x7fc0141491b0 2024-03-28T14:09:02.456 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:02.450+0000 7fc0137fe640 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fc0140b3730 0x7fc014148c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:09:02.456 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:02.450+0000 7fc0137fe640 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fc0140b3730 0x7fc014148c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:49852/0 (socket says 172.21.15.104:49852) 2024-03-28T14:09:02.456 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:02.450+0000 7fc0137fe640 1 -- 172.21.15.104:0/2444791108 learned_addr learned my addr 172.21.15.104:0/2444791108 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-03-28T14:09:02.456 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:02.450+0000 7fc018f5e640 1 --2- 172.21.15.104:0/2444791108 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fc0141491b0 0x7fc014149620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:09:02.456 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:02.450+0000 7fc013fff640 1 --2- 172.21.15.104:0/2444791108 >> v2:172.21.15.171:3300/0 conn(0x7fc01406c8f0 0x7fc0140b2740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:09:02.456 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:02.450+0000 7fc0137fe640 1 -- 172.21.15.104:0/2444791108 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fc0141491b0 msgr2=0x7fc014149620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T14:09:02.456 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:02.450+0000 7fc0137fe640 1 --2- 172.21.15.104:0/2444791108 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fc0141491b0 0x7fc014149620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:09:02.457 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:02.450+0000 7fc0137fe640 1 -- 172.21.15.104:0/2444791108 >> v2:172.21.15.171:3300/0 conn(0x7fc01406c8f0 msgr2=0x7fc0140b2740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T14:09:02.457 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:02.450+0000 7fc0137fe640 1 --2- 172.21.15.104:0/2444791108 >> v2:172.21.15.171:3300/0 conn(0x7fc01406c8f0 0x7fc0140b2740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:09:02.457 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:02.450+0000 7fc0137fe640 1 -- 172.21.15.104:0/2444791108 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc01406ccd0 con 0x7fc0140b3730 2024-03-28T14:09:02.457 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:02.450+0000 7fc018f5e640 1 --2- 172.21.15.104:0/2444791108 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fc0141491b0 0x7fc014149620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T14:09:02.457 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:02.450+0000 7fc013fff640 1 --2- 172.21.15.104:0/2444791108 >> v2:172.21.15.171:3300/0 conn(0x7fc01406c8f0 0x7fc0140b2740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T14:09:02.460 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:02.454+0000 7fc0137fe640 1 --2- 172.21.15.104:0/2444791108 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc0140b3730 0x7fc014148c70 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7fc01406e1f0 tx=0x7fc00401c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4bebc9389a21684e server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:09:02.460 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:02.454+0000 7fc012ffd640 1 -- 172.21.15.104:0/2444791108 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc0040ab650 con 0x7fc0140b3730 2024-03-28T14:09:02.460 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:02.454+0000 7fc012ffd640 1 -- 172.21.15.104:0/2444791108 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc0040b0070 con 0x7fc0140b3730 2024-03-28T14:09:02.461 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:02.454+0000 7fc012ffd640 1 -- 172.21.15.104:0/2444791108 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc0040aba10 con 0x7fc0140b3730 2024-03-28T14:09:02.461 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:02.454+0000 7fc01a9c1640 1 -- 172.21.15.104:0/2444791108 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc0140b3730 msgr2=0x7fc014148c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:09:02.461 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:02.454+0000 7fc01a9c1640 1 --2- 172.21.15.104:0/2444791108 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc0140b3730 0x7fc014148c70 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7fc01406e1f0 tx=0x7fc00401c490 comp rx=0 tx=0).stop 2024-03-28T14:09:02.461 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:02.454+0000 7fc01a9c1640 1 -- 172.21.15.104:0/2444791108 shutdown_connections 2024-03-28T14:09:02.461 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:02.454+0000 7fc01a9c1640 1 --2- 172.21.15.104:0/2444791108 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fc0141491b0 0x7fc014149620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:09:02.461 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:02.454+0000 7fc01a9c1640 1 --2- 172.21.15.104:0/2444791108 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc0140b3730 0x7fc014148c70 unknown :-1 s=CLOSED pgs=447 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:09:02.461 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:02.454+0000 7fc01a9c1640 1 --2- 172.21.15.104:0/2444791108 >> v2:172.21.15.171:3300/0 conn(0x7fc01406c8f0 0x7fc0140b2740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:09:02.461 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:02.454+0000 7fc01a9c1640 1 -- 172.21.15.104:0/2444791108 >> 172.21.15.104:0/2444791108 conn(0x7fc0140b8f00 msgr2=0x7fc0140b9310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T14:09:02.462 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:02.454+0000 7fc01a9c1640 1 -- 172.21.15.104:0/2444791108 shutdown_connections 2024-03-28T14:09:02.462 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:02.454+0000 7fc01a9c1640 1 -- 172.21.15.104:0/2444791108 wait complete. 2024-03-28T14:09:02.462 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:02.454+0000 7fc01a9c1640 1 Processor -- start 2024-03-28T14:09:02.462 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:02.454+0000 7fc01a9c1640 1 -- start start 2024-03-28T14:09:02.462 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:02.454+0000 7fc01a9c1640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc01406c8f0 0x7fc01414aa60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:09:02.462 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:02.454+0000 7fc01a9c1640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fc0140b3730 0x7fc014144130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:09:02.462 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:02.454+0000 7fc01a9c1640 1 --2- >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fc0141491b0 0x7fc014144670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:09:02.462 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:02.454+0000 7fc01a9c1640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7fc0140b1660 con 0x7fc01406c8f0 2024-03-28T14:09:02.462 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:02.454+0000 7fc01a9c1640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7fc0140b14e0 con 0x7fc0140b3730 2024-03-28T14:09:02.463 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:02.454+0000 7fc01a9c1640 1 -- --> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] -- mon_getmap magic: 0 -- 0x7fc01406dc70 con 0x7fc0141491b0 2024-03-28T14:09:02.463 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:02.454+0000 7fc013fff640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc01406c8f0 0x7fc01414aa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:09:02.463 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:02.454+0000 7fc013fff640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc01406c8f0 0x7fc01414aa60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:49856/0 (socket says 172.21.15.104:49856) 2024-03-28T14:09:02.463 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:02.454+0000 7fc013fff640 1 -- 172.21.15.104:0/1129173628 learned_addr learned my addr 172.21.15.104:0/1129173628 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-03-28T14:09:02.463 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:02.454+0000 7fc018f5e640 1 --2- 172.21.15.104:0/1129173628 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fc0141491b0 0x7fc014144670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:09:02.466 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:02.454+0000 7fc013fff640 1 -- 172.21.15.104:0/1129173628 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fc0141491b0 msgr2=0x7fc014144670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:09:02.466 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:02.454+0000 7fc013fff640 1 --2- 172.21.15.104:0/1129173628 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fc0141491b0 0x7fc014144670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:09:02.466 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:02.454+0000 7fc0137fe640 1 --2- 172.21.15.104:0/1129173628 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fc0140b3730 0x7fc014144130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:09:02.466 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:02.454+0000 7fc013fff640 1 -- 172.21.15.104:0/1129173628 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fc0140b3730 msgr2=0x7fc014144130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:09:02.466 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:02.454+0000 7fc013fff640 1 --2- 172.21.15.104:0/1129173628 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fc0140b3730 0x7fc014144130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:09:02.466 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:02.454+0000 7fc013fff640 1 -- 172.21.15.104:0/1129173628 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc01414d420 con 0x7fc01406c8f0 2024-03-28T14:09:02.466 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:02.454+0000 7fc013fff640 1 --2- 172.21.15.104:0/1129173628 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc01406c8f0 0x7fc01414aa60 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7fc00800c7f0 tx=0x7fc00800ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:09:02.466 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:02.454+0000 7fc010ff9640 1 -- 172.21.15.104:0/1129173628 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc008015070 con 0x7fc01406c8f0 2024-03-28T14:09:02.467 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:02.454+0000 7fc010ff9640 1 -- 172.21.15.104:0/1129173628 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc008007820 con 0x7fc01406c8f0 2024-03-28T14:09:02.467 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:02.454+0000 7fc01a9c1640 1 -- 172.21.15.104:0/1129173628 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc01414d690 con 0x7fc01406c8f0 2024-03-28T14:09:02.467 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:02.454+0000 7fc010ff9640 1 -- 172.21.15.104:0/1129173628 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc008007c10 con 0x7fc01406c8f0 2024-03-28T14:09:02.467 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:02.454+0000 7fc01a9c1640 1 -- 172.21.15.104:0/1129173628 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc014149c80 con 0x7fc01406c8f0 2024-03-28T14:09:02.467 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:02.458+0000 7fc01a9c1640 1 -- 172.21.15.104:0/1129173628 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fc0140687c0 con 0x7fc01406c8f0 2024-03-28T14:09:02.469 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:02.458+0000 7fc010ff9640 1 -- 172.21.15.104:0/1129173628 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) ==== 87803+0+0 (secure 0 0 0) 0x7fc008018070 con 0x7fc01406c8f0 2024-03-28T14:09:02.469 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:02.458+0000 7fc010ff9640 1 --2- 172.21.15.104:0/1129173628 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7fc000067fc0 0x7fc00006a480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:09:02.469 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:02.458+0000 7fc010ff9640 1 -- 172.21.15.104:0/1129173628 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(33..33 src has 1..33) ==== 4943+0+0 (secure 0 0 0) 0x7fc008086820 con 0x7fc01406c8f0 2024-03-28T14:09:02.469 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:02.458+0000 7fc010ff9640 1 --2- 172.21.15.104:0/1129173628 >> [v2:172.21.15.104:6808/19438674,v1:172.21.15.104:6810/19438674] conn(0x7fc00006de80 0x7fc0000702a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:09:02.469 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:02.458+0000 7fc010ff9640 1 -- 172.21.15.104:0/1129173628 --> [v2:172.21.15.104:6808/19438674,v1:172.21.15.104:6810/19438674] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fc000070950 con 0x7fc00006de80 2024-03-28T14:09:02.469 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:02.458+0000 7fc010ff9640 1 -- 172.21.15.104:0/1129173628 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7fc008086cb0 con 0x7fc01406c8f0 2024-03-28T14:09:02.469 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:02.458+0000 7fc018f5e640 1 --2- 172.21.15.104:0/1129173628 >> [v2:172.21.15.104:6808/19438674,v1:172.21.15.104:6810/19438674] conn(0x7fc00006de80 0x7fc0000702a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:09:02.469 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:02.458+0000 7fc0137fe640 1 --2- 172.21.15.104:0/1129173628 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7fc000067fc0 0x7fc00006a480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:09:02.470 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:02.458+0000 7fc018f5e640 1 --2- 172.21.15.104:0/1129173628 >> [v2:172.21.15.104:6808/19438674,v1:172.21.15.104:6810/19438674] conn(0x7fc00006de80 0x7fc0000702a0 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:09:02.470 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:02.458+0000 7fc0137fe640 1 --2- 172.21.15.104:0/1129173628 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7fc000067fc0 0x7fc00006a480 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fc0040af000 tx=0x7fc004008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:09:02.470 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:02.458+0000 7fc010ff9640 1 -- 172.21.15.104:0/1129173628 <== osd.1 v2:172.21.15.104:6808/19438674 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fc000070950 con 0x7fc00006de80 2024-03-28T14:09:02.476 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:02.470+0000 7fc01a9c1640 1 -- 172.21.15.104:0/1129173628 --> [v2:172.21.15.104:6808/19438674,v1:172.21.15.104:6810/19438674] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7fc0140b66e0 con 0x7fc00006de80 2024-03-28T14:09:02.476 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:02.474+0000 7fc010ff9640 1 -- 172.21.15.104:0/1129173628 <== osd.1 v2:172.21.15.104:6808/19438674 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7fc0140b66e0 con 0x7fc00006de80 2024-03-28T14:09:02.476 INFO:teuthology.orchestra.run.smithi104.stdout:{ 2024-03-28T14:09:02.476 INFO:teuthology.orchestra.run.smithi104.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-03-28T14:09:02.476 INFO:teuthology.orchestra.run.smithi104.stdout:} 2024-03-28T14:09:02.477 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:02.474+0000 7fc01a9c1640 1 -- 172.21.15.104:0/1129173628 >> [v2:172.21.15.104:6808/19438674,v1:172.21.15.104:6810/19438674] conn(0x7fc00006de80 msgr2=0x7fc0000702a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:09:02.477 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:02.474+0000 7fc01a9c1640 1 --2- 172.21.15.104:0/1129173628 >> [v2:172.21.15.104:6808/19438674,v1:172.21.15.104:6810/19438674] conn(0x7fc00006de80 0x7fc0000702a0 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:09:02.478 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:02.474+0000 7fc01a9c1640 1 -- 172.21.15.104:0/1129173628 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7fc000067fc0 msgr2=0x7fc00006a480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:09:02.478 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:02.474+0000 7fc01a9c1640 1 --2- 172.21.15.104:0/1129173628 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7fc000067fc0 0x7fc00006a480 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fc0040af000 tx=0x7fc004008040 comp rx=0 tx=0).stop 2024-03-28T14:09:02.478 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:02.474+0000 7fc01a9c1640 1 -- 172.21.15.104:0/1129173628 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc01406c8f0 msgr2=0x7fc01414aa60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:09:02.478 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:02.474+0000 7fc01a9c1640 1 --2- 172.21.15.104:0/1129173628 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc01406c8f0 0x7fc01414aa60 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7fc00800c7f0 tx=0x7fc00800ccb0 comp rx=0 tx=0).stop 2024-03-28T14:09:02.478 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:02.474+0000 7fc018f5e640 1 -- 172.21.15.104:0/1129173628 reap_dead start 2024-03-28T14:09:02.478 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:02.474+0000 7fc01a9c1640 1 -- 172.21.15.104:0/1129173628 shutdown_connections 2024-03-28T14:09:02.478 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:02.474+0000 7fc01a9c1640 1 -- 172.21.15.104:0/1129173628 >> 172.21.15.104:0/1129173628 conn(0x7fc0140b8f00 msgr2=0x7fc014067330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T14:09:02.479 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:02.474+0000 7fc01a9c1640 1 -- 172.21.15.104:0/1129173628 shutdown_connections 2024-03-28T14:09:02.479 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:02.474+0000 7fc01a9c1640 1 -- 172.21.15.104:0/1129173628 wait complete. 2024-03-28T14:09:02.491 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 1 2024-03-28T14:09:02.596 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:02.594+0000 7fe78f712640 1 Processor -- start 2024-03-28T14:09:02.597 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:02.594+0000 7fe78f712640 1 -- start start 2024-03-28T14:09:02.598 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:02.594+0000 7fe78f712640 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fe7880b2740 0x7fe7880b3730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:09:02.598 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:02.594+0000 7fe78f712640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe78806ccd0 con 0x7fe788149380 2024-03-28T14:09:02.598 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:02.594+0000 7fe78f712640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe7880b1980 con 0x7fe78806c8f0 2024-03-28T14:09:02.598 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:02.594+0000 7fe78f712640 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 -- 0x7fe7880b2b70 con 0x7fe7880b2740 2024-03-28T14:09:02.598 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:02.594+0000 7fe78ccad640 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fe7880b2740 0x7fe7880b3730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:09:02.598 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:02.594+0000 7fe78dcaf640 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7fe788149380 0x7fe7881501b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:40542/0 (socket says 172.21.15.104:40542) 2024-03-28T14:09:02.599 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:02.594+0000 7fe78dcaf640 1 -- 172.21.15.104:0/1445394784 learned_addr learned my addr 172.21.15.104:0/1445394784 (peer_addr_for_me v1:172.21.15.104:0/0) 2024-03-28T14:09:02.599 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:02.594+0000 7fe78ccad640 1 -- 172.21.15.104:0/1445394784 >> v1:172.21.15.171:6789/0 conn(0x7fe78806c8f0 legacy=0x7fe788148c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T14:09:02.599 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:02.594+0000 7fe78ccad640 1 -- 172.21.15.104:0/1445394784 >> v1:172.21.15.104:6789/0 conn(0x7fe788149380 legacy=0x7fe7881501b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:09:02.599 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:02.594+0000 7fe78ccad640 1 -- 172.21.15.104:0/1445394784 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe7880b0850 con 0x7fe7880b2740 2024-03-28T14:09:02.600 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:02.598+0000 7fe78ccad640 1 --2- 172.21.15.104:0/1445394784 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe7880b2740 0x7fe7880b3730 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7fe7840097c0 tx=0x7fe78401c3d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=863bb7ed5c63c6bd server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:09:02.600 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:02.598+0000 7fe77ffff640 1 -- 172.21.15.104:0/1445394784 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7840b0070 con 0x7fe7880b2740 2024-03-28T14:09:02.600 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:02.598+0000 7fe77ffff640 1 -- 172.21.15.104:0/1445394784 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe7840ab600 con 0x7fe7880b2740 2024-03-28T14:09:02.600 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:02.598+0000 7fe77ffff640 1 -- 172.21.15.104:0/1445394784 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7840a4e60 con 0x7fe7880b2740 2024-03-28T14:09:02.601 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:02.598+0000 7fe78f712640 1 -- 172.21.15.104:0/1445394784 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe7880b2740 msgr2=0x7fe7880b3730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:09:02.601 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:02.598+0000 7fe78f712640 1 --2- 172.21.15.104:0/1445394784 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe7880b2740 0x7fe7880b3730 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7fe7840097c0 tx=0x7fe78401c3d0 comp rx=0 tx=0).stop 2024-03-28T14:09:02.601 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:02.598+0000 7fe78f712640 1 -- 172.21.15.104:0/1445394784 shutdown_connections 2024-03-28T14:09:02.602 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:02.598+0000 7fe78f712640 1 --2- 172.21.15.104:0/1445394784 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe7880b2740 0x7fe7880b3730 unknown :-1 s=CLOSED pgs=450 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:09:02.602 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:02.598+0000 7fe78f712640 1 -- 172.21.15.104:0/1445394784 >> 172.21.15.104:0/1445394784 conn(0x7fe7880b8f00 msgr2=0x7fe7880b9310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T14:09:02.602 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:02.598+0000 7fe78f712640 1 -- 172.21.15.104:0/1445394784 shutdown_connections 2024-03-28T14:09:02.602 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:02.598+0000 7fe78f712640 1 -- 172.21.15.104:0/1445394784 wait complete. 2024-03-28T14:09:02.604 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:02.598+0000 7fe78f712640 1 Processor -- start 2024-03-28T14:09:02.604 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:02.598+0000 7fe78f712640 1 -- start start 2024-03-28T14:09:02.604 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:02.598+0000 7fe78f712640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fe78806c8f0 0x7fe788071e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:09:02.605 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:02.598+0000 7fe78f712640 1 --2- >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fe7880b2740 0x7fe7880add50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:09:02.605 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:02.598+0000 7fe78f712640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe788149380 0x7fe788070db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:09:02.605 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:02.598+0000 7fe78f712640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7fe788154680 con 0x7fe788149380 2024-03-28T14:09:02.605 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:02.598+0000 7fe78f712640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7fe788154500 con 0x7fe78806c8f0 2024-03-28T14:09:02.605 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:02.598+0000 7fe78f712640 1 -- --> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] -- mon_getmap magic: 0 -- 0x7fe788154800 con 0x7fe7880b2740 2024-03-28T14:09:02.605 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:02.598+0000 7fe78dcaf640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe788149380 0x7fe788070db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:09:02.605 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:02.598+0000 7fe78dcaf640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe788149380 0x7fe788070db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:49872/0 (socket says 172.21.15.104:49872) 2024-03-28T14:09:02.605 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:02.598+0000 7fe78dcaf640 1 -- 172.21.15.104:0/4144017514 learned_addr learned my addr 172.21.15.104:0/4144017514 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-03-28T14:09:02.606 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:02.598+0000 7fe78dcaf640 1 -- 172.21.15.104:0/4144017514 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fe7880b2740 msgr2=0x7fe7880add50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:09:02.606 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:02.598+0000 7fe78dcaf640 1 --2- 172.21.15.104:0/4144017514 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fe7880b2740 0x7fe7880add50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:09:02.606 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:02.598+0000 7fe78dcaf640 1 -- 172.21.15.104:0/4144017514 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fe78806c8f0 msgr2=0x7fe788071e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:09:02.606 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:02.598+0000 7fe78dcaf640 1 --2- 172.21.15.104:0/4144017514 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fe78806c8f0 0x7fe788071e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:09:02.606 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:02.598+0000 7fe78dcaf640 1 -- 172.21.15.104:0/4144017514 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe7880724a0 con 0x7fe788149380 2024-03-28T14:09:02.606 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:02.598+0000 7fe78dcaf640 1 --2- 172.21.15.104:0/4144017514 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe788149380 0x7fe788070db0 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7fe78000cad0 tx=0x7fe78000cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:09:02.606 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:02.598+0000 7fe77dffb640 1 -- 172.21.15.104:0/4144017514 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe780017070 con 0x7fe788149380 2024-03-28T14:09:02.606 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:02.598+0000 7fe77dffb640 1 -- 172.21.15.104:0/4144017514 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe78000ac30 con 0x7fe788149380 2024-03-28T14:09:02.607 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:02.598+0000 7fe77dffb640 1 -- 172.21.15.104:0/4144017514 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe780012620 con 0x7fe788149380 2024-03-28T14:09:02.607 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:02.598+0000 7fe78f712640 1 -- 172.21.15.104:0/4144017514 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe7880ae290 con 0x7fe788149380 2024-03-28T14:09:02.607 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:02.598+0000 7fe78f712640 1 -- 172.21.15.104:0/4144017514 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe7880712f0 con 0x7fe788149380 2024-03-28T14:09:02.607 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:02.602+0000 7fe78f712640 1 -- 172.21.15.104:0/4144017514 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe7880687c0 con 0x7fe788149380 2024-03-28T14:09:02.608 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:02.602+0000 7fe77dffb640 1 -- 172.21.15.104:0/4144017514 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) ==== 87803+0+0 (secure 0 0 0) 0x7fe78001a070 con 0x7fe788149380 2024-03-28T14:09:02.608 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:02.602+0000 7fe77dffb640 1 --2- 172.21.15.104:0/4144017514 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7fe76c067ff0 0x7fe76c06a4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:09:02.608 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:02.602+0000 7fe77dffb640 1 -- 172.21.15.104:0/4144017514 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(33..33 src has 1..33) ==== 4943+0+0 (secure 0 0 0) 0x7fe780085ea0 con 0x7fe788149380 2024-03-28T14:09:02.608 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:02.602+0000 7fe78d4ae640 1 --2- 172.21.15.104:0/4144017514 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7fe76c067ff0 0x7fe76c06a4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:09:02.608 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:02.602+0000 7fe78d4ae640 1 --2- 172.21.15.104:0/4144017514 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7fe76c067ff0 0x7fe76c06a4b0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fe778009650 tx=0x7fe778006bb0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:09:02.609 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:02.606+0000 7fe77dffb640 1 -- 172.21.15.104:0/4144017514 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179117 (secure 0 0 0) 0x7fe7880724a0 con 0x7fe788149380 2024-03-28T14:09:02.758 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:02.754+0000 7fe78f712640 1 -- 172.21.15.104:0/4144017514 --> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "1", "target": ["mon-mgr", ""]}) -- 0x7fe78806ff50 con 0x7fe76c067ff0 2024-03-28T14:09:02.758 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:02.754+0000 7fe77dffb640 1 -- 172.21.15.104:0/4144017514 <== mgr.4100 v2:172.21.15.171:6832/1522752412 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 1 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7fe78806ff50 con 0x7fe76c067ff0 2024-03-28T14:09:02.758 INFO:teuthology.orchestra.run.smithi104.stderr:instructed osd(s) 1 to deep-scrub 2024-03-28T14:09:02.760 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:02.758+0000 7fe78f712640 1 -- 172.21.15.104:0/4144017514 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7fe76c067ff0 msgr2=0x7fe76c06a4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:09:02.761 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:02.758+0000 7fe78f712640 1 --2- 172.21.15.104:0/4144017514 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7fe76c067ff0 0x7fe76c06a4b0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fe778009650 tx=0x7fe778006bb0 comp rx=0 tx=0).stop 2024-03-28T14:09:02.761 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:02.758+0000 7fe78f712640 1 -- 172.21.15.104:0/4144017514 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe788149380 msgr2=0x7fe788070db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:09:02.761 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:02.758+0000 7fe78f712640 1 --2- 172.21.15.104:0/4144017514 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe788149380 0x7fe788070db0 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7fe78000cad0 tx=0x7fe78000cf90 comp rx=0 tx=0).stop 2024-03-28T14:09:02.761 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:02.758+0000 7fe78f712640 1 -- 172.21.15.104:0/4144017514 shutdown_connections 2024-03-28T14:09:02.761 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:02.758+0000 7fe78f712640 1 --2- 172.21.15.104:0/4144017514 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7fe76c067ff0 0x7fe76c06a4b0 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:09:02.761 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:02.758+0000 7fe78f712640 1 --2- 172.21.15.104:0/4144017514 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe788149380 0x7fe788070db0 unknown :-1 s=CLOSED pgs=451 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:09:02.761 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:02.758+0000 7fe78f712640 1 --2- 172.21.15.104:0/4144017514 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fe7880b2740 0x7fe7880add50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:09:02.761 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:02.758+0000 7fe78f712640 1 --2- 172.21.15.104:0/4144017514 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fe78806c8f0 0x7fe788071e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:09:02.762 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:02.758+0000 7fe78f712640 1 -- 172.21.15.104:0/4144017514 >> 172.21.15.104:0/4144017514 conn(0x7fe7880b8f00 msgr2=0x7fe7881439a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T14:09:02.762 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:02.758+0000 7fe78f712640 1 -- 172.21.15.104:0/4144017514 shutdown_connections 2024-03-28T14:09:02.762 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:02.758+0000 7fe78f712640 1 -- 172.21.15.104:0/4144017514 wait complete. 2024-03-28T14:09:02.775 INFO:tasks.ceph:Scrubbing osd.2 2024-03-28T14:09:02.775 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 config set osd_debug_deep_scrub_sleep 0 2024-03-28T14:09:02.888 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:02.882+0000 7f2458eaa640 1 Processor -- start 2024-03-28T14:09:02.888 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:02.882+0000 7f2458eaa640 1 -- start start 2024-03-28T14:09:02.890 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:02.886+0000 7f2458eaa640 1 --2- >> v2:172.21.15.171:3300/0 conn(0x7f24540bb8c0 0x7f24540b6580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:09:02.890 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:02.886+0000 7f2458eaa640 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f245406da00 0x7f2454141640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:09:02.890 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:02.886+0000 7f2458eaa640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f24540bbca0 con 0x7f24540b5b50 2024-03-28T14:09:02.890 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:02.886+0000 7f2458eaa640 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 -- 0x7f24540b1ae0 con 0x7f245406da00 2024-03-28T14:09:02.890 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:02.886+0000 7f2458eaa640 1 -- --> v2:172.21.15.171:3300/0 -- mon_getmap magic: 0 -- 0x7f24540b1da0 con 0x7f24540bb8c0 2024-03-28T14:09:02.890 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:02.886+0000 7f244bfff640 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f24540b5b50 0x7f24540b72f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:40556/0 (socket says 172.21.15.104:40556) 2024-03-28T14:09:02.891 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:02.886+0000 7f244bfff640 1 -- 172.21.15.104:0/3990280899 learned_addr learned my addr 172.21.15.104:0/3990280899 (peer_addr_for_me v1:172.21.15.104:0/0) 2024-03-28T14:09:02.891 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:02.886+0000 7f2452d9d640 1 --2- 172.21.15.104:0/3990280899 >> v2:172.21.15.104:3300/0 conn(0x7f245406da00 0x7f2454141640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:09:02.891 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:02.886+0000 7f245259c640 1 --2- 172.21.15.104:0/3990280899 >> v2:172.21.15.171:3300/0 conn(0x7f24540bb8c0 0x7f24540b6580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:09:02.891 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:02.886+0000 7f2451d9b640 1 -- 172.21.15.104:0/3990280899 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3591459603 0 0) 0x7f24540bbca0 con 0x7f24540b5b50 2024-03-28T14:09:02.891 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:02.886+0000 7f2451d9b640 1 -- 172.21.15.104:0/3990280899 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2440003530 con 0x7f24540b5b50 2024-03-28T14:09:02.891 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:02.886+0000 7f2452d9d640 1 -- 172.21.15.104:0/3990280899 >> v2:172.21.15.171:3300/0 conn(0x7f24540bb8c0 msgr2=0x7f24540b6580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T14:09:02.891 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:02.886+0000 7f2452d9d640 1 --2- 172.21.15.104:0/3990280899 >> v2:172.21.15.171:3300/0 conn(0x7f24540bb8c0 0x7f24540b6580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:09:02.891 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:02.886+0000 7f2452d9d640 1 -- 172.21.15.104:0/3990280899 >> v1:172.21.15.104:6789/0 conn(0x7f24540b5b50 legacy=0x7f24540b72f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:09:02.891 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:02.886+0000 7f2452d9d640 1 -- 172.21.15.104:0/3990280899 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f24540b6dd0 con 0x7f245406da00 2024-03-28T14:09:02.892 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:02.886+0000 7f245259c640 1 --2- 172.21.15.104:0/3990280899 >> v2:172.21.15.171:3300/0 conn(0x7f24540bb8c0 0x7f24540b6580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T14:09:02.895 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:02.886+0000 7f2452d9d640 1 --2- 172.21.15.104:0/3990280899 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f245406da00 0x7f2454141640 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7f24540bbec0 tx=0x7f244c01c930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4ba2c7692919b388 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:09:02.895 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:02.886+0000 7f2451d9b640 1 -- 172.21.15.104:0/3990280899 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f244c0b2070 con 0x7f245406da00 2024-03-28T14:09:02.895 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:02.886+0000 7f2451d9b640 1 -- 172.21.15.104:0/3990280899 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f244c0a57f0 con 0x7f245406da00 2024-03-28T14:09:02.895 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:02.886+0000 7f2451d9b640 1 -- 172.21.15.104:0/3990280899 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f244c0a5bc0 con 0x7f245406da00 2024-03-28T14:09:02.895 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:02.890+0000 7f2458eaa640 1 -- 172.21.15.104:0/3990280899 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f245406da00 msgr2=0x7f2454141640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:09:02.896 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:02.890+0000 7f2458eaa640 1 --2- 172.21.15.104:0/3990280899 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f245406da00 0x7f2454141640 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7f24540bbec0 tx=0x7f244c01c930 comp rx=0 tx=0).stop 2024-03-28T14:09:02.896 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:02.890+0000 7f2458eaa640 1 -- 172.21.15.104:0/3990280899 shutdown_connections 2024-03-28T14:09:02.896 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:02.890+0000 7f2458eaa640 1 --2- 172.21.15.104:0/3990280899 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f245406da00 0x7f2454141640 unknown :-1 s=CLOSED pgs=453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:09:02.896 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:02.890+0000 7f2458eaa640 1 --2- 172.21.15.104:0/3990280899 >> v2:172.21.15.171:3300/0 conn(0x7f24540bb8c0 0x7f24540b6580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:09:02.896 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:02.890+0000 7f2458eaa640 1 -- 172.21.15.104:0/3990280899 >> 172.21.15.104:0/3990280899 conn(0x7f245402c800 msgr2=0x7f24540baf50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T14:09:02.896 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:02.890+0000 7f2458eaa640 1 -- 172.21.15.104:0/3990280899 shutdown_connections 2024-03-28T14:09:02.896 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:02.890+0000 7f2458eaa640 1 -- 172.21.15.104:0/3990280899 wait complete. 2024-03-28T14:09:02.896 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:02.890+0000 7f2458eaa640 1 Processor -- start 2024-03-28T14:09:02.897 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:02.890+0000 7f2458eaa640 1 -- start start 2024-03-28T14:09:02.897 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:02.890+0000 7f2458eaa640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f245406da00 0x7f245413c260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:09:02.897 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:02.890+0000 7f2458eaa640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f24540b5b50 0x7f24541e0470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:09:02.897 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:02.890+0000 7f2458eaa640 1 --2- >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f24540bb8c0 0x7f24541e09b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:09:02.897 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:02.890+0000 7f2458eaa640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f2454154c60 con 0x7f245406da00 2024-03-28T14:09:02.897 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:02.890+0000 7f2458eaa640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f2454154ae0 con 0x7f24540b5b50 2024-03-28T14:09:02.897 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:02.890+0000 7f2458eaa640 1 -- --> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] -- mon_getmap magic: 0 -- 0x7f2454154de0 con 0x7f24540bb8c0 2024-03-28T14:09:02.897 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:02.890+0000 7f245259c640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f245406da00 0x7f245413c260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:09:02.898 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:02.890+0000 7f245259c640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f245406da00 0x7f245413c260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:49894/0 (socket says 172.21.15.104:49894) 2024-03-28T14:09:02.898 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:02.890+0000 7f245259c640 1 -- 172.21.15.104:0/55669858 learned_addr learned my addr 172.21.15.104:0/55669858 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-03-28T14:09:02.898 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:02.890+0000 7f2452d9d640 1 --2- 172.21.15.104:0/55669858 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f24540bb8c0 0x7f24541e09b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:09:02.898 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:02.890+0000 7f245259c640 1 -- 172.21.15.104:0/55669858 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f24540bb8c0 msgr2=0x7f24541e09b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:09:02.898 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:02.890+0000 7f244bfff640 1 --2- 172.21.15.104:0/55669858 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f24540b5b50 0x7f24541e0470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:09:02.898 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:02.890+0000 7f245259c640 1 --2- 172.21.15.104:0/55669858 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f24540bb8c0 0x7f24541e09b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:09:02.898 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:02.890+0000 7f245259c640 1 -- 172.21.15.104:0/55669858 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f24540b5b50 msgr2=0x7f24541e0470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:09:02.898 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:02.890+0000 7f245259c640 1 --2- 172.21.15.104:0/55669858 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f24540b5b50 0x7f24541e0470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:09:02.899 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:02.890+0000 7f245259c640 1 -- 172.21.15.104:0/55669858 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2454137620 con 0x7f245406da00 2024-03-28T14:09:02.899 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:02.890+0000 7f245259c640 1 --2- 172.21.15.104:0/55669858 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f245406da00 0x7f245413c260 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7f244400e810 tx=0x7f244400ecd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:09:02.899 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:02.890+0000 7f244affd640 1 -- 172.21.15.104:0/55669858 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2444017070 con 0x7f245406da00 2024-03-28T14:09:02.899 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:02.890+0000 7f244affd640 1 -- 172.21.15.104:0/55669858 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f244400c8d0 con 0x7f245406da00 2024-03-28T14:09:02.899 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:02.890+0000 7f2458eaa640 1 -- 172.21.15.104:0/55669858 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f245413b5f0 con 0x7f245406da00 2024-03-28T14:09:02.899 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:02.890+0000 7f244affd640 1 -- 172.21.15.104:0/55669858 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f244400f040 con 0x7f245406da00 2024-03-28T14:09:02.899 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:02.890+0000 7f2458eaa640 1 -- 172.21.15.104:0/55669858 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2454145450 con 0x7f245406da00 2024-03-28T14:09:02.899 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:02.890+0000 7f2458eaa640 1 -- 172.21.15.104:0/55669858 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f24540687c0 con 0x7f245406da00 2024-03-28T14:09:02.901 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:02.890+0000 7f244affd640 1 -- 172.21.15.104:0/55669858 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) ==== 87803+0+0 (secure 0 0 0) 0x7f244401a070 con 0x7f245406da00 2024-03-28T14:09:02.901 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:02.894+0000 7f244affd640 1 --2- 172.21.15.104:0/55669858 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f2434067fc0 0x7f243406a480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:09:02.901 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:02.894+0000 7f244affd640 1 -- 172.21.15.104:0/55669858 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(33..33 src has 1..33) ==== 4943+0+0 (secure 0 0 0) 0x7f2444086450 con 0x7f245406da00 2024-03-28T14:09:02.901 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:02.894+0000 7f244affd640 1 --2- 172.21.15.104:0/55669858 >> [v2:172.21.15.104:6824/2352012626,v1:172.21.15.104:6825/2352012626] conn(0x7f243406de80 0x7f24340702a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:09:02.901 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:02.894+0000 7f244affd640 1 -- 172.21.15.104:0/55669858 --> [v2:172.21.15.104:6824/2352012626,v1:172.21.15.104:6825/2352012626] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f2434070950 con 0x7f243406de80 2024-03-28T14:09:02.902 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:02.894+0000 7f244affd640 1 -- 172.21.15.104:0/55669858 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7f24440868e0 con 0x7f245406da00 2024-03-28T14:09:02.902 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:02.894+0000 7f2452d9d640 1 --2- 172.21.15.104:0/55669858 >> [v2:172.21.15.104:6824/2352012626,v1:172.21.15.104:6825/2352012626] conn(0x7f243406de80 0x7f24340702a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:09:02.902 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:02.894+0000 7f244bfff640 1 --2- 172.21.15.104:0/55669858 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f2434067fc0 0x7f243406a480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:09:02.902 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:02.894+0000 7f2452d9d640 1 --2- 172.21.15.104:0/55669858 >> [v2:172.21.15.104:6824/2352012626,v1:172.21.15.104:6825/2352012626] conn(0x7f243406de80 0x7f24340702a0 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:09:02.902 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:02.894+0000 7f244bfff640 1 --2- 172.21.15.104:0/55669858 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f2434067fc0 0x7f243406a480 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f243c007fd0 tx=0x7f243c009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:09:02.902 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:02.894+0000 7f244affd640 1 -- 172.21.15.104:0/55669858 <== osd.2 v2:172.21.15.104:6824/2352012626 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f2434070950 con 0x7f243406de80 2024-03-28T14:09:02.910 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:02.906+0000 7f2458eaa640 1 -- 172.21.15.104:0/55669858 --> [v2:172.21.15.104:6824/2352012626,v1:172.21.15.104:6825/2352012626] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7f24540b6010 con 0x7f243406de80 2024-03-28T14:09:02.910 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:02.906+0000 7f244affd640 1 -- 172.21.15.104:0/55669858 <== osd.2 v2:172.21.15.104:6824/2352012626 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7f24540b6010 con 0x7f243406de80 2024-03-28T14:09:02.911 INFO:teuthology.orchestra.run.smithi104.stdout:{ 2024-03-28T14:09:02.911 INFO:teuthology.orchestra.run.smithi104.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-03-28T14:09:02.911 INFO:teuthology.orchestra.run.smithi104.stdout:} 2024-03-28T14:09:02.912 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:02.906+0000 7f2458eaa640 1 -- 172.21.15.104:0/55669858 >> [v2:172.21.15.104:6824/2352012626,v1:172.21.15.104:6825/2352012626] conn(0x7f243406de80 msgr2=0x7f24340702a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:09:02.912 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:02.906+0000 7f2458eaa640 1 --2- 172.21.15.104:0/55669858 >> [v2:172.21.15.104:6824/2352012626,v1:172.21.15.104:6825/2352012626] conn(0x7f243406de80 0x7f24340702a0 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:09:02.912 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:02.906+0000 7f2458eaa640 1 -- 172.21.15.104:0/55669858 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f2434067fc0 msgr2=0x7f243406a480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:09:02.912 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:02.906+0000 7f2458eaa640 1 --2- 172.21.15.104:0/55669858 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f2434067fc0 0x7f243406a480 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f243c007fd0 tx=0x7f243c009040 comp rx=0 tx=0).stop 2024-03-28T14:09:02.913 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:02.906+0000 7f2458eaa640 1 -- 172.21.15.104:0/55669858 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f245406da00 msgr2=0x7f245413c260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:09:02.913 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:02.906+0000 7f2458eaa640 1 --2- 172.21.15.104:0/55669858 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f245406da00 0x7f245413c260 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7f244400e810 tx=0x7f244400ecd0 comp rx=0 tx=0).stop 2024-03-28T14:09:02.913 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:02.906+0000 7f2452d9d640 1 -- 172.21.15.104:0/55669858 reap_dead start 2024-03-28T14:09:02.913 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:02.906+0000 7f2458eaa640 1 -- 172.21.15.104:0/55669858 shutdown_connections 2024-03-28T14:09:02.913 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:02.906+0000 7f2458eaa640 1 -- 172.21.15.104:0/55669858 >> 172.21.15.104:0/55669858 conn(0x7f245402c800 msgr2=0x7f24540b7a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T14:09:02.913 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:02.906+0000 7f2458eaa640 1 -- 172.21.15.104:0/55669858 shutdown_connections 2024-03-28T14:09:02.913 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:02.906+0000 7f2458eaa640 1 -- 172.21.15.104:0/55669858 wait complete. 2024-03-28T14:09:02.925 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 2 2024-03-28T14:09:03.037 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:03.034+0000 7fda311b5640 1 Processor -- start 2024-03-28T14:09:03.037 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:03.034+0000 7fda311b5640 1 -- start start 2024-03-28T14:09:03.038 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:03.034+0000 7fda311b5640 1 --2- >> v2:172.21.15.171:3300/0 conn(0x7fda2c0bb8c0 0x7fda2c0b5950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:09:03.038 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:03.034+0000 7fda311b5640 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fda2c0b74d0 0x7fda2c141640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:09:03.039 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:03.034+0000 7fda311b5640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fda2c0b65e0 con 0x7fda2c141b80 2024-03-28T14:09:03.039 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:03.034+0000 7fda311b5640 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 -- 0x7fda2c0bbca0 con 0x7fda2c0b74d0 2024-03-28T14:09:03.039 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:03.034+0000 7fda311b5640 1 -- --> v2:172.21.15.171:3300/0 -- mon_getmap magic: 0 -- 0x7fda2c0b6020 con 0x7fda2c0bb8c0 2024-03-28T14:09:03.039 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:03.034+0000 7fda2a59c640 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fda2c0b74d0 0x7fda2c141640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:09:03.039 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:03.034+0000 7fda2b59e640 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7fda2c141b80 0x7fda2c06dd90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:40560/0 (socket says 172.21.15.104:40560) 2024-03-28T14:09:03.039 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:03.034+0000 7fda2b59e640 1 -- 172.21.15.104:0/1794965070 learned_addr learned my addr 172.21.15.104:0/1794965070 (peer_addr_for_me v1:172.21.15.104:0/0) 2024-03-28T14:09:03.039 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:03.034+0000 7fda2ad9d640 1 --2- 172.21.15.104:0/1794965070 >> v2:172.21.15.171:3300/0 conn(0x7fda2c0bb8c0 0x7fda2c0b5950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:09:03.039 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:03.034+0000 7fda29d9b640 1 -- 172.21.15.104:0/1794965070 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1292175247 0 0) 0x7fda2c0b65e0 con 0x7fda2c141b80 2024-03-28T14:09:03.040 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:03.034+0000 7fda29d9b640 1 -- 172.21.15.104:0/1794965070 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fda14003530 con 0x7fda2c141b80 2024-03-28T14:09:03.040 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:03.034+0000 7fda2a59c640 1 -- 172.21.15.104:0/1794965070 >> v2:172.21.15.171:3300/0 conn(0x7fda2c0bb8c0 msgr2=0x7fda2c0b5950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T14:09:03.040 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:03.034+0000 7fda2a59c640 1 --2- 172.21.15.104:0/1794965070 >> v2:172.21.15.171:3300/0 conn(0x7fda2c0bb8c0 0x7fda2c0b5950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:09:03.040 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:03.034+0000 7fda2a59c640 1 -- 172.21.15.104:0/1794965070 >> v1:172.21.15.104:6789/0 conn(0x7fda2c141b80 legacy=0x7fda2c06dd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:09:03.040 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:03.034+0000 7fda2a59c640 1 -- 172.21.15.104:0/1794965070 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fda2c0b6300 con 0x7fda2c0b74d0 2024-03-28T14:09:03.041 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:03.038+0000 7fda2a59c640 1 --2- 172.21.15.104:0/1794965070 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fda2c0b74d0 0x7fda2c141640 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7fda2c0b8af0 tx=0x7fda1c01c480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6893b4d7d03e404e server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:09:03.042 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:03.038+0000 7fda29d9b640 1 -- 172.21.15.104:0/1794965070 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda1c0b1070 con 0x7fda2c0b74d0 2024-03-28T14:09:03.042 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:03.038+0000 7fda29d9b640 1 -- 172.21.15.104:0/1794965070 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fda1c0ac5f0 con 0x7fda2c0b74d0 2024-03-28T14:09:03.042 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:03.038+0000 7fda29d9b640 1 -- 172.21.15.104:0/1794965070 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda1c0ac9a0 con 0x7fda2c0b74d0 2024-03-28T14:09:03.042 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:03.038+0000 7fda311b5640 1 -- 172.21.15.104:0/1794965070 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fda2c0b74d0 msgr2=0x7fda2c141640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:09:03.042 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:03.038+0000 7fda311b5640 1 --2- 172.21.15.104:0/1794965070 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fda2c0b74d0 0x7fda2c141640 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7fda2c0b8af0 tx=0x7fda1c01c480 comp rx=0 tx=0).stop 2024-03-28T14:09:03.042 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:03.038+0000 7fda311b5640 1 -- 172.21.15.104:0/1794965070 shutdown_connections 2024-03-28T14:09:03.042 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:03.038+0000 7fda311b5640 1 --2- 172.21.15.104:0/1794965070 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fda2c0b74d0 0x7fda2c141640 unknown :-1 s=CLOSED pgs=456 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:09:03.043 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:03.038+0000 7fda311b5640 1 --2- 172.21.15.104:0/1794965070 >> v2:172.21.15.171:3300/0 conn(0x7fda2c0bb8c0 0x7fda2c0b5950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:09:03.043 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:03.038+0000 7fda311b5640 1 -- 172.21.15.104:0/1794965070 >> 172.21.15.104:0/1794965070 conn(0x7fda2c02c800 msgr2=0x7fda2c0baf50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T14:09:03.043 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:03.038+0000 7fda311b5640 1 -- 172.21.15.104:0/1794965070 shutdown_connections 2024-03-28T14:09:03.043 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:03.038+0000 7fda311b5640 1 -- 172.21.15.104:0/1794965070 wait complete. 2024-03-28T14:09:03.046 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:03.038+0000 7fda311b5640 1 Processor -- start 2024-03-28T14:09:03.046 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:03.038+0000 7fda311b5640 1 -- start start 2024-03-28T14:09:03.046 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:03.038+0000 7fda311b5640 1 --2- >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fda2c0b74d0 0x7fda2c142080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:09:03.046 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:03.038+0000 7fda311b5640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fda2c0bb8c0 0x7fda2c147d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:09:03.047 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:03.038+0000 7fda311b5640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fda2c141b80 0x7fda2c137460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:09:03.047 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:03.038+0000 7fda311b5640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7fda2c154c20 con 0x7fda2c141b80 2024-03-28T14:09:03.047 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:03.038+0000 7fda311b5640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7fda2c154aa0 con 0x7fda2c0bb8c0 2024-03-28T14:09:03.047 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:03.038+0000 7fda311b5640 1 -- --> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] -- mon_getmap magic: 0 -- 0x7fda2c154da0 con 0x7fda2c0b74d0 2024-03-28T14:09:03.047 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:03.038+0000 7fda2b59e640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fda2c141b80 0x7fda2c137460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:09:03.047 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:03.038+0000 7fda2b59e640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fda2c141b80 0x7fda2c137460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:49906/0 (socket says 172.21.15.104:49906) 2024-03-28T14:09:03.047 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:03.038+0000 7fda2b59e640 1 -- 172.21.15.104:0/3285019637 learned_addr learned my addr 172.21.15.104:0/3285019637 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-03-28T14:09:03.047 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:03.038+0000 7fda2ad9d640 1 --2- 172.21.15.104:0/3285019637 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fda2c0b74d0 0x7fda2c142080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:09:03.048 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:03.038+0000 7fda2a59c640 1 --2- 172.21.15.104:0/3285019637 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fda2c0bb8c0 0x7fda2c147d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:09:03.048 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:03.042+0000 7fda2b59e640 1 -- 172.21.15.104:0/3285019637 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fda2c0b74d0 msgr2=0x7fda2c142080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:09:03.048 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:03.042+0000 7fda2b59e640 1 --2- 172.21.15.104:0/3285019637 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fda2c0b74d0 0x7fda2c142080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:09:03.048 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:03.042+0000 7fda2b59e640 1 -- 172.21.15.104:0/3285019637 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fda2c0bb8c0 msgr2=0x7fda2c147d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:09:03.048 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:03.042+0000 7fda2b59e640 1 --2- 172.21.15.104:0/3285019637 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fda2c0bb8c0 0x7fda2c147d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:09:03.049 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:03.042+0000 7fda2b59e640 1 -- 172.21.15.104:0/3285019637 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fda2c1386b0 con 0x7fda2c141b80 2024-03-28T14:09:03.049 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:03.042+0000 7fda2a59c640 1 --2- 172.21.15.104:0/3285019637 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fda2c0bb8c0 0x7fda2c147d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T14:09:03.049 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:03.042+0000 7fda2b59e640 1 --2- 172.21.15.104:0/3285019637 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fda2c141b80 0x7fda2c137460 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7fda2400caa0 tx=0x7fda2400cf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:09:03.049 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:03.042+0000 7fda137fe640 1 -- 172.21.15.104:0/3285019637 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda24017070 con 0x7fda2c141b80 2024-03-28T14:09:03.049 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:03.042+0000 7fda137fe640 1 -- 172.21.15.104:0/3285019637 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fda2400ac00 con 0x7fda2c141b80 2024-03-28T14:09:03.049 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:03.042+0000 7fda311b5640 1 -- 172.21.15.104:0/3285019637 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fda2c138990 con 0x7fda2c141b80 2024-03-28T14:09:03.049 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:03.042+0000 7fda137fe640 1 -- 172.21.15.104:0/3285019637 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda24012400 con 0x7fda2c141b80 2024-03-28T14:09:03.049 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:03.042+0000 7fda311b5640 1 -- 172.21.15.104:0/3285019637 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fda2c1441c0 con 0x7fda2c141b80 2024-03-28T14:09:03.050 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:03.042+0000 7fda311b5640 1 -- 172.21.15.104:0/3285019637 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fda2c0687c0 con 0x7fda2c141b80 2024-03-28T14:09:03.050 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:03.042+0000 7fda137fe640 1 -- 172.21.15.104:0/3285019637 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) ==== 87803+0+0 (secure 0 0 0) 0x7fda2401a070 con 0x7fda2c141b80 2024-03-28T14:09:03.050 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:03.042+0000 7fda137fe640 1 --2- 172.21.15.104:0/3285019637 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7fda18067fc0 0x7fda1806a480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:09:03.050 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:03.042+0000 7fda137fe640 1 -- 172.21.15.104:0/3285019637 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(33..33 src has 1..33) ==== 4943+0+0 (secure 0 0 0) 0x7fda24085f90 con 0x7fda2c141b80 2024-03-28T14:09:03.051 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:03.042+0000 7fda2ad9d640 1 --2- 172.21.15.104:0/3285019637 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7fda18067fc0 0x7fda1806a480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:09:03.051 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:03.042+0000 7fda2ad9d640 1 --2- 172.21.15.104:0/3285019637 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7fda18067fc0 0x7fda1806a480 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fda2000b9d0 tx=0x7fda20008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:09:03.051 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:03.046+0000 7fda137fe640 1 -- 172.21.15.104:0/3285019637 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179117 (secure 0 0 0) 0x7fda24056ad0 con 0x7fda2c141b80 2024-03-28T14:09:03.208 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:03.206+0000 7fda311b5640 1 -- 172.21.15.104:0/3285019637 --> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "2", "target": ["mon-mgr", ""]}) -- 0x7fda2c14baf0 con 0x7fda18067fc0 2024-03-28T14:09:03.209 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:03.206+0000 7fda137fe640 1 -- 172.21.15.104:0/3285019637 <== mgr.4100 v2:172.21.15.171:6832/1522752412 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 2 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7fda2c14baf0 con 0x7fda18067fc0 2024-03-28T14:09:03.209 INFO:teuthology.orchestra.run.smithi104.stderr:instructed osd(s) 2 to deep-scrub 2024-03-28T14:09:03.212 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:03.206+0000 7fda311b5640 1 -- 172.21.15.104:0/3285019637 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7fda18067fc0 msgr2=0x7fda1806a480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:09:03.212 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:03.206+0000 7fda311b5640 1 --2- 172.21.15.104:0/3285019637 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7fda18067fc0 0x7fda1806a480 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fda2000b9d0 tx=0x7fda20008040 comp rx=0 tx=0).stop 2024-03-28T14:09:03.212 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:03.206+0000 7fda311b5640 1 -- 172.21.15.104:0/3285019637 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fda2c141b80 msgr2=0x7fda2c137460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:09:03.212 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:03.206+0000 7fda311b5640 1 --2- 172.21.15.104:0/3285019637 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fda2c141b80 0x7fda2c137460 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7fda2400caa0 tx=0x7fda2400cf60 comp rx=0 tx=0).stop 2024-03-28T14:09:03.212 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:03.206+0000 7fda311b5640 1 -- 172.21.15.104:0/3285019637 shutdown_connections 2024-03-28T14:09:03.212 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:03.206+0000 7fda311b5640 1 --2- 172.21.15.104:0/3285019637 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7fda18067fc0 0x7fda1806a480 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:09:03.212 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:03.206+0000 7fda311b5640 1 --2- 172.21.15.104:0/3285019637 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fda2c141b80 0x7fda2c137460 unknown :-1 s=CLOSED pgs=457 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:09:03.213 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:03.206+0000 7fda311b5640 1 --2- 172.21.15.104:0/3285019637 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fda2c0bb8c0 0x7fda2c147d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:09:03.213 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:03.206+0000 7fda311b5640 1 --2- 172.21.15.104:0/3285019637 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fda2c0b74d0 0x7fda2c142080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:09:03.213 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:03.206+0000 7fda311b5640 1 -- 172.21.15.104:0/3285019637 >> 172.21.15.104:0/3285019637 conn(0x7fda2c02c800 msgr2=0x7fda2c148ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T14:09:03.213 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:03.206+0000 7fda311b5640 1 -- 172.21.15.104:0/3285019637 shutdown_connections 2024-03-28T14:09:03.213 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:03.206+0000 7fda311b5640 1 -- 172.21.15.104:0/3285019637 wait complete. 2024-03-28T14:09:03.228 INFO:tasks.ceph:Scrubbing osd.3 2024-03-28T14:09:03.228 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 config set osd_debug_deep_scrub_sleep 0 2024-03-28T14:09:03.350 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:03.346+0000 7f9aee4d7640 1 Processor -- start 2024-03-28T14:09:03.351 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:03.346+0000 7f9aee4d7640 1 -- start start 2024-03-28T14:09:03.353 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:03.346+0000 7f9aee4d7640 1 --2- >> v2:172.21.15.171:3300/0 conn(0x7f9ae80af660 0x7f9ae8147bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:09:03.354 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:03.346+0000 7f9aee4d7640 1 --2- >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f9ae80b29e0 0x7f9ae814ef20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:09:03.354 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:03.346+0000 7f9aee4d7640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9ae80b4e80 con 0x7f9ae80b4aa0 2024-03-28T14:09:03.354 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:03.346+0000 7f9aee4d7640 1 -- --> v2:172.21.15.171:3300/0 -- mon_getmap magic: 0 -- 0x7f9ae80afaa0 con 0x7f9ae80af660 2024-03-28T14:09:03.354 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:03.346+0000 7f9aee4d7640 1 -- --> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] -- mon_getmap magic: 0 -- 0x7f9ae80b0890 con 0x7f9ae80b29e0 2024-03-28T14:09:03.354 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:03.346+0000 7f9ae7fff640 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f9ae80b4aa0 0x7f9ae80b22d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:40572/0 (socket says 172.21.15.104:40572) 2024-03-28T14:09:03.354 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:03.346+0000 7f9ae7fff640 1 -- 172.21.15.104:0/1747715939 learned_addr learned my addr 172.21.15.104:0/1747715939 (peer_addr_for_me v1:172.21.15.104:0/0) 2024-03-28T14:09:03.355 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:03.346+0000 7f9ae77fe640 1 --2- 172.21.15.104:0/1747715939 >> v2:172.21.15.171:3300/0 conn(0x7f9ae80af660 0x7f9ae8147bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:09:03.355 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:03.346+0000 7f9aeca74640 1 --2- 172.21.15.104:0/1747715939 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f9ae80b29e0 0x7f9ae814ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:09:03.355 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:03.346+0000 7f9ae6ffd640 1 -- 172.21.15.104:0/1747715939 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1394647822 0 0) 0x7f9ae80b4e80 con 0x7f9ae80b4aa0 2024-03-28T14:09:03.355 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:03.346+0000 7f9ae6ffd640 1 -- 172.21.15.104:0/1747715939 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9ad0003530 con 0x7f9ae80b4aa0 2024-03-28T14:09:03.355 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:03.346+0000 7f9ae6ffd640 1 -- 172.21.15.104:0/1747715939 <== mon.0 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2012653632 0 0) 0x7f9ad0003530 con 0x7f9ae80b4aa0 2024-03-28T14:09:03.356 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:03.346+0000 7f9ae6ffd640 1 -- 172.21.15.104:0/1747715939 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f9ae80b29e0 msgr2=0x7f9ae814ef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T14:09:03.356 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:03.346+0000 7f9ae6ffd640 1 --2- 172.21.15.104:0/1747715939 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f9ae80b29e0 0x7f9ae814ef20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:09:03.356 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:03.346+0000 7f9ae6ffd640 1 -- 172.21.15.104:0/1747715939 >> v2:172.21.15.171:3300/0 conn(0x7f9ae80af660 msgr2=0x7f9ae8147bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T14:09:03.356 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:03.346+0000 7f9ae6ffd640 1 --2- 172.21.15.104:0/1747715939 >> v2:172.21.15.171:3300/0 conn(0x7f9ae80af660 0x7f9ae8147bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:09:03.357 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:03.346+0000 7f9ae6ffd640 1 -- 172.21.15.104:0/1747715939 --> v1:172.21.15.104:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9ae80ae0a0 con 0x7f9ae80b4aa0 2024-03-28T14:09:03.357 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:03.346+0000 7f9ae6ffd640 1 -- 172.21.15.104:0/1747715939 <== mon.0 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3745580849 0 0) 0x7f9ad8001680 con 0x7f9ae80b4aa0 2024-03-28T14:09:03.357 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:03.346+0000 7f9ae6ffd640 1 -- 172.21.15.104:0/1747715939 >> v1:172.21.15.104:6789/0 conn(0x7f9ae80b4aa0 legacy=0x7f9ae80b22d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:09:03.357 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:03.346+0000 7f9ae6ffd640 1 --2- 172.21.15.104:0/1747715939 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9ad00039c0 0x7f9ad0003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:09:03.358 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:03.346+0000 7f9ae6ffd640 1 --2- 172.21.15.104:0/1747715939 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f9ae80b29e0 0x7f9ad0004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:09:03.358 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:03.350+0000 7f9ae6ffd640 1 --2- 172.21.15.104:0/1747715939 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f9ad0004d50 0x7f9ad00051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:09:03.358 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:03.350+0000 7f9ae6ffd640 1 -- 172.21.15.104:0/1747715939 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f9ae80afaa0 con 0x7f9ad00039c0 2024-03-28T14:09:03.358 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:03.350+0000 7f9ae6ffd640 1 -- 172.21.15.104:0/1747715939 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f9ae80b0890 con 0x7f9ad0004d50 2024-03-28T14:09:03.358 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:03.350+0000 7f9ae6ffd640 1 -- 172.21.15.104:0/1747715939 --> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] -- mon_getmap magic: 0 -- 0x7f9ad000ba00 con 0x7f9ae80b29e0 2024-03-28T14:09:03.358 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:03.350+0000 7f9ae7fff640 1 --2- 172.21.15.104:0/1747715939 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9ad00039c0 0x7f9ad0003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:09:03.359 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:03.350+0000 7f9ae77fe640 1 --2- 172.21.15.104:0/1747715939 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f9ae80b29e0 0x7f9ad0004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:09:03.359 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:03.350+0000 7f9aeca74640 1 --2- 172.21.15.104:0/1747715939 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f9ad0004d50 0x7f9ad00051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:09:03.359 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:03.350+0000 7f9ae7fff640 1 -- 172.21.15.104:0/1747715939 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f9ae80b29e0 msgr2=0x7f9ad0004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T14:09:03.359 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:03.350+0000 7f9ae7fff640 1 --2- 172.21.15.104:0/1747715939 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f9ae80b29e0 0x7f9ad0004810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:09:03.359 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:03.350+0000 7f9ae7fff640 1 -- 172.21.15.104:0/1747715939 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f9ad0004d50 msgr2=0x7f9ad00051d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T14:09:03.359 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:03.350+0000 7f9ae7fff640 1 --2- 172.21.15.104:0/1747715939 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f9ad0004d50 0x7f9ad00051d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:09:03.359 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:03.350+0000 7f9ae7fff640 1 -- 172.21.15.104:0/1747715939 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9ad0003530 con 0x7f9ad00039c0 2024-03-28T14:09:03.364 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:03.350+0000 7f9ae7fff640 1 --2- 172.21.15.104:0/1747715939 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9ad00039c0 0x7f9ad0003de0 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f9ad800c240 tx=0x7f9ad801ccc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f7cae942f60886ad server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:09:03.364 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:03.350+0000 7f9ae6ffd640 1 -- 172.21.15.104:0/1747715939 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ad8007d60 con 0x7f9ad00039c0 2024-03-28T14:09:03.364 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:03.350+0000 7f9ae6ffd640 1 -- 172.21.15.104:0/1747715939 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9ad80a9020 con 0x7f9ad00039c0 2024-03-28T14:09:03.364 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:03.350+0000 7f9ae6ffd640 1 -- 172.21.15.104:0/1747715939 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ad8005ae0 con 0x7f9ad00039c0 2024-03-28T14:09:03.365 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:03.350+0000 7f9aee4d7640 1 -- 172.21.15.104:0/1747715939 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9ad00039c0 msgr2=0x7f9ad0003de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:09:03.365 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:03.350+0000 7f9aee4d7640 1 --2- 172.21.15.104:0/1747715939 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9ad00039c0 0x7f9ad0003de0 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f9ad800c240 tx=0x7f9ad801ccc0 comp rx=0 tx=0).stop 2024-03-28T14:09:03.365 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:03.350+0000 7f9aeca74640 1 -- 172.21.15.104:0/1747715939 reap_dead start 2024-03-28T14:09:03.365 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:03.350+0000 7f9aee4d7640 1 -- 172.21.15.104:0/1747715939 shutdown_connections 2024-03-28T14:09:03.365 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:03.350+0000 7f9aee4d7640 1 -- 172.21.15.104:0/1747715939 >> 172.21.15.104:0/1747715939 conn(0x7f9ae80b8f00 msgr2=0x7f9ae80b9310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T14:09:03.365 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:03.350+0000 7f9aee4d7640 1 -- 172.21.15.104:0/1747715939 shutdown_connections 2024-03-28T14:09:03.365 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:03.350+0000 7f9aee4d7640 1 -- 172.21.15.104:0/1747715939 wait complete. 2024-03-28T14:09:03.365 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:03.354+0000 7f9aee4d7640 1 Processor -- start 2024-03-28T14:09:03.366 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:03.354+0000 7f9aee4d7640 1 -- start start 2024-03-28T14:09:03.366 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:03.354+0000 7f9aee4d7640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f9ad00039c0 0x7f9ae813da90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:09:03.366 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:03.354+0000 7f9aee4d7640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9ae813bd60 0x7f9ae8136270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:09:03.366 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:03.354+0000 7f9aee4d7640 1 --2- >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f9ae813c1e0 0x7f9ae81367b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:09:03.366 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:03.354+0000 7f9aee4d7640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f9ae806dc40 con 0x7f9ae813bd60 2024-03-28T14:09:03.366 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:03.354+0000 7f9aee4d7640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f9ae80ae9a0 con 0x7f9ad00039c0 2024-03-28T14:09:03.366 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:03.354+0000 7f9aee4d7640 1 -- --> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] -- mon_getmap magic: 0 -- 0x7f9ae813d030 con 0x7f9ae813c1e0 2024-03-28T14:09:03.367 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:03.354+0000 7f9ae77fe640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9ae813bd60 0x7f9ae8136270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:09:03.367 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:03.354+0000 7f9ae77fe640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9ae813bd60 0x7f9ae8136270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:49922/0 (socket says 172.21.15.104:49922) 2024-03-28T14:09:03.367 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:03.354+0000 7f9ae77fe640 1 -- 172.21.15.104:0/3516368789 learned_addr learned my addr 172.21.15.104:0/3516368789 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-03-28T14:09:03.367 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:03.354+0000 7f9aeca74640 1 --2- 172.21.15.104:0/3516368789 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f9ae813c1e0 0x7f9ae81367b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:09:03.367 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:03.354+0000 7f9ae7fff640 1 --2- 172.21.15.104:0/3516368789 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f9ad00039c0 0x7f9ae813da90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:09:03.367 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:03.354+0000 7f9ae77fe640 1 -- 172.21.15.104:0/3516368789 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f9ae813c1e0 msgr2=0x7f9ae81367b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:09:03.367 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:03.354+0000 7f9ae77fe640 1 --2- 172.21.15.104:0/3516368789 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f9ae813c1e0 0x7f9ae81367b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:09:03.367 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:03.354+0000 7f9ae77fe640 1 -- 172.21.15.104:0/3516368789 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f9ad00039c0 msgr2=0x7f9ae813da90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:09:03.368 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:03.354+0000 7f9ae77fe640 1 --2- 172.21.15.104:0/3516368789 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f9ad00039c0 0x7f9ae813da90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:09:03.368 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:03.354+0000 7f9ae77fe640 1 -- 172.21.15.104:0/3516368789 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9ae813f020 con 0x7f9ae813bd60 2024-03-28T14:09:03.368 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:03.354+0000 7f9ae77fe640 1 --2- 172.21.15.104:0/3516368789 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9ae813bd60 0x7f9ae8136270 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7f9adc0115e0 tx=0x7f9adc011aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:09:03.368 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:03.354+0000 7f9ae4ff9640 1 -- 172.21.15.104:0/3516368789 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9adc00f8f0 con 0x7f9ae813bd60 2024-03-28T14:09:03.368 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:03.354+0000 7f9ae4ff9640 1 -- 172.21.15.104:0/3516368789 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9adc00fa90 con 0x7f9ae813bd60 2024-03-28T14:09:03.368 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:03.354+0000 7f9ae4ff9640 1 -- 172.21.15.104:0/3516368789 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9adc00fe40 con 0x7f9ae813bd60 2024-03-28T14:09:03.368 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:03.354+0000 7f9aee4d7640 1 -- 172.21.15.104:0/3516368789 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9ae813d280 con 0x7f9ae813bd60 2024-03-28T14:09:03.368 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:03.354+0000 7f9aee4d7640 1 -- 172.21.15.104:0/3516368789 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9ae813e3d0 con 0x7f9ae813bd60 2024-03-28T14:09:03.368 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:03.354+0000 7f9aee4d7640 1 -- 172.21.15.104:0/3516368789 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f9ae80687c0 con 0x7f9ae813bd60 2024-03-28T14:09:03.369 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:03.354+0000 7f9ae4ff9640 1 -- 172.21.15.104:0/3516368789 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) ==== 87803+0+0 (secure 0 0 0) 0x7f9adc015460 con 0x7f9ae813bd60 2024-03-28T14:09:03.369 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:03.354+0000 7f9ae4ff9640 1 --2- 172.21.15.104:0/3516368789 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f9ad4067ff0 0x7f9ad406a4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:09:03.369 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:03.354+0000 7f9ae4ff9640 1 -- 172.21.15.104:0/3516368789 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(33..33 src has 1..33) ==== 4943+0+0 (secure 0 0 0) 0x7f9adc086b70 con 0x7f9ae813bd60 2024-03-28T14:09:03.369 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:03.354+0000 7f9ae4ff9640 1 --2- 172.21.15.104:0/3516368789 >> [v2:172.21.15.104:6809/3568347381,v1:172.21.15.104:6811/3568347381] conn(0x7f9ad406deb0 0x7f9ad40702d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:09:03.369 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:03.354+0000 7f9ae4ff9640 1 -- 172.21.15.104:0/3516368789 --> [v2:172.21.15.104:6809/3568347381,v1:172.21.15.104:6811/3568347381] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f9adc00ce90 con 0x7f9ad406deb0 2024-03-28T14:09:03.369 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:03.354+0000 7f9ae4ff9640 1 -- 172.21.15.104:0/3516368789 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7f9adc087000 con 0x7f9ae813bd60 2024-03-28T14:09:03.369 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:03.354+0000 7f9aeca74640 1 --2- 172.21.15.104:0/3516368789 >> [v2:172.21.15.104:6809/3568347381,v1:172.21.15.104:6811/3568347381] conn(0x7f9ad406deb0 0x7f9ad40702d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:09:03.369 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:03.358+0000 7f9ae7fff640 1 --2- 172.21.15.104:0/3516368789 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f9ad4067ff0 0x7f9ad406a4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:09:03.370 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:03.358+0000 7f9aeca74640 1 --2- 172.21.15.104:0/3516368789 >> [v2:172.21.15.104:6809/3568347381,v1:172.21.15.104:6811/3568347381] conn(0x7f9ad406deb0 0x7f9ad40702d0 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:09:03.370 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:03.358+0000 7f9ae7fff640 1 --2- 172.21.15.104:0/3516368789 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f9ad4067ff0 0x7f9ad406a4b0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f9ad80aa000 tx=0x7f9ad8001ac0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:09:03.370 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:03.358+0000 7f9ae4ff9640 1 -- 172.21.15.104:0/3516368789 <== osd.3 v2:172.21.15.104:6809/3568347381 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f9adc00ce90 con 0x7f9ad406deb0 2024-03-28T14:09:03.373 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:03.370+0000 7f9aee4d7640 1 -- 172.21.15.104:0/3516368789 --> [v2:172.21.15.104:6809/3568347381,v1:172.21.15.104:6811/3568347381] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7f9ae80738d0 con 0x7f9ad406deb0 2024-03-28T14:09:03.373 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:03.370+0000 7f9ae4ff9640 1 -- 172.21.15.104:0/3516368789 <== osd.3 v2:172.21.15.104:6809/3568347381 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7f9ae80738d0 con 0x7f9ad406deb0 2024-03-28T14:09:03.374 INFO:teuthology.orchestra.run.smithi104.stdout:{ 2024-03-28T14:09:03.374 INFO:teuthology.orchestra.run.smithi104.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-03-28T14:09:03.374 INFO:teuthology.orchestra.run.smithi104.stdout:} 2024-03-28T14:09:03.375 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:03.370+0000 7f9aee4d7640 1 -- 172.21.15.104:0/3516368789 >> [v2:172.21.15.104:6809/3568347381,v1:172.21.15.104:6811/3568347381] conn(0x7f9ad406deb0 msgr2=0x7f9ad40702d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:09:03.375 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:03.370+0000 7f9aee4d7640 1 --2- 172.21.15.104:0/3516368789 >> [v2:172.21.15.104:6809/3568347381,v1:172.21.15.104:6811/3568347381] conn(0x7f9ad406deb0 0x7f9ad40702d0 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:09:03.375 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:03.370+0000 7f9aee4d7640 1 -- 172.21.15.104:0/3516368789 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f9ad4067ff0 msgr2=0x7f9ad406a4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:09:03.375 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:03.370+0000 7f9aee4d7640 1 --2- 172.21.15.104:0/3516368789 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f9ad4067ff0 0x7f9ad406a4b0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f9ad80aa000 tx=0x7f9ad8001ac0 comp rx=0 tx=0).stop 2024-03-28T14:09:03.376 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:03.370+0000 7f9aee4d7640 1 -- 172.21.15.104:0/3516368789 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9ae813bd60 msgr2=0x7f9ae8136270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:09:03.376 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:03.370+0000 7f9aee4d7640 1 --2- 172.21.15.104:0/3516368789 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9ae813bd60 0x7f9ae8136270 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7f9adc0115e0 tx=0x7f9adc011aa0 comp rx=0 tx=0).stop 2024-03-28T14:09:03.376 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:03.370+0000 7f9aeca74640 1 -- 172.21.15.104:0/3516368789 reap_dead start 2024-03-28T14:09:03.376 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:03.370+0000 7f9aee4d7640 1 -- 172.21.15.104:0/3516368789 shutdown_connections 2024-03-28T14:09:03.376 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:03.370+0000 7f9aee4d7640 1 -- 172.21.15.104:0/3516368789 >> 172.21.15.104:0/3516368789 conn(0x7f9ae80b8f00 msgr2=0x7f9ae80b3420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T14:09:03.376 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:03.370+0000 7f9aee4d7640 1 -- 172.21.15.104:0/3516368789 shutdown_connections 2024-03-28T14:09:03.376 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:03.370+0000 7f9aee4d7640 1 -- 172.21.15.104:0/3516368789 wait complete. 2024-03-28T14:09:03.388 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 3 2024-03-28T14:09:03.493 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:03.490+0000 7f3271fdf640 1 Processor -- start 2024-03-28T14:09:03.493 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:03.490+0000 7f3271fdf640 1 -- start start 2024-03-28T14:09:03.496 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:03.490+0000 7f3271fdf640 1 --2- >> v2:172.21.15.171:3300/0 conn(0x7f326c0b7a40 0x7f326c14b840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:09:03.496 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:03.490+0000 7f3271fdf640 1 --2- >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f326c141500 0x7f326c141930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:09:03.497 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:03.490+0000 7f3271fdf640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f326c0b51e0 con 0x7f326c0b4e00 2024-03-28T14:09:03.497 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:03.490+0000 7f3271fdf640 1 -- --> v2:172.21.15.171:3300/0 -- mon_getmap magic: 0 -- 0x7f326c14ad60 con 0x7f326c0b7a40 2024-03-28T14:09:03.497 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:03.490+0000 7f3271fdf640 1 -- --> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] -- mon_getmap magic: 0 -- 0x7f326c14b040 con 0x7f326c141500 2024-03-28T14:09:03.497 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:03.490+0000 7f326b7fe640 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f326c0b4e00 0x7f326c0b5c60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:40588/0 (socket says 172.21.15.104:40588) 2024-03-28T14:09:03.497 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:03.490+0000 7f326b7fe640 1 -- 172.21.15.104:0/2056163754 learned_addr learned my addr 172.21.15.104:0/2056163754 (peer_addr_for_me v1:172.21.15.104:0/0) 2024-03-28T14:09:03.497 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:03.490+0000 7f326bfff640 1 --2- 172.21.15.104:0/2056163754 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f326c141500 0x7f326c141930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:09:03.497 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:03.490+0000 7f326affd640 1 --2- 172.21.15.104:0/2056163754 >> v2:172.21.15.171:3300/0 conn(0x7f326c0b7a40 0x7f326c14b840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:09:03.498 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:03.490+0000 7f326a7fc640 1 -- 172.21.15.104:0/2056163754 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2308929630 0 0) 0x7f326c0b51e0 con 0x7f326c0b4e00 2024-03-28T14:09:03.498 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:03.490+0000 7f326a7fc640 1 -- 172.21.15.104:0/2056163754 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3254003530 con 0x7f326c0b4e00 2024-03-28T14:09:03.498 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:03.490+0000 7f326a7fc640 1 -- 172.21.15.104:0/2056163754 <== mon.0 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3150920930 0 0) 0x7f3254003530 con 0x7f326c0b4e00 2024-03-28T14:09:03.498 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:03.490+0000 7f326a7fc640 1 -- 172.21.15.104:0/2056163754 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f326c141500 msgr2=0x7f326c141930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T14:09:03.498 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:03.490+0000 7f326a7fc640 1 --2- 172.21.15.104:0/2056163754 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f326c141500 0x7f326c141930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:09:03.498 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:03.490+0000 7f326a7fc640 1 -- 172.21.15.104:0/2056163754 >> v2:172.21.15.171:3300/0 conn(0x7f326c0b7a40 msgr2=0x7f326c14b840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T14:09:03.498 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:03.490+0000 7f326a7fc640 1 --2- 172.21.15.104:0/2056163754 >> v2:172.21.15.171:3300/0 conn(0x7f326c0b7a40 0x7f326c14b840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:09:03.498 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:03.490+0000 7f326a7fc640 1 -- 172.21.15.104:0/2056163754 --> v1:172.21.15.104:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f326c14b320 con 0x7f326c0b4e00 2024-03-28T14:09:03.499 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:03.490+0000 7f326a7fc640 1 -- 172.21.15.104:0/2056163754 <== mon.0 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3745580849 0 0) 0x7f325c001680 con 0x7f326c0b4e00 2024-03-28T14:09:03.499 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:03.490+0000 7f326affd640 1 --2- 172.21.15.104:0/2056163754 >> v2:172.21.15.171:3300/0 conn(0x7f326c0b7a40 0x7f326c14b840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T14:09:03.499 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:03.490+0000 7f326a7fc640 1 -- 172.21.15.104:0/2056163754 >> v1:172.21.15.104:6789/0 conn(0x7f326c0b4e00 legacy=0x7f326c0b5c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:09:03.499 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:03.490+0000 7f326a7fc640 1 --2- 172.21.15.104:0/2056163754 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f32540039c0 0x7f3254003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:09:03.499 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:03.490+0000 7f326a7fc640 1 --2- 172.21.15.104:0/2056163754 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3254004810 0x7f3254004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:09:03.499 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:03.490+0000 7f326a7fc640 1 --2- 172.21.15.104:0/2056163754 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f326c141500 0x7f32540051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:09:03.499 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:03.490+0000 7f326a7fc640 1 -- 172.21.15.104:0/2056163754 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f326c14ad60 con 0x7f3254004810 2024-03-28T14:09:03.500 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:03.490+0000 7f326a7fc640 1 -- 172.21.15.104:0/2056163754 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f326c14b040 con 0x7f32540039c0 2024-03-28T14:09:03.500 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:03.490+0000 7f326a7fc640 1 -- 172.21.15.104:0/2056163754 --> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] -- mon_getmap magic: 0 -- 0x7f325400ba80 con 0x7f326c141500 2024-03-28T14:09:03.500 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:03.490+0000 7f326affd640 1 --2- 172.21.15.104:0/2056163754 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3254004810 0x7f3254004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:09:03.500 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:03.490+0000 7f326b7fe640 1 --2- 172.21.15.104:0/2056163754 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f32540039c0 0x7f3254003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:09:03.500 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:03.490+0000 7f326bfff640 1 --2- 172.21.15.104:0/2056163754 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f326c141500 0x7f32540051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:09:03.500 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:03.490+0000 7f326affd640 1 -- 172.21.15.104:0/2056163754 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f326c141500 msgr2=0x7f32540051d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T14:09:03.500 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:03.490+0000 7f326affd640 1 --2- 172.21.15.104:0/2056163754 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f326c141500 0x7f32540051d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:09:03.500 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:03.490+0000 7f326affd640 1 -- 172.21.15.104:0/2056163754 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f32540039c0 msgr2=0x7f3254003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T14:09:03.500 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:03.490+0000 7f326affd640 1 --2- 172.21.15.104:0/2056163754 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f32540039c0 0x7f3254003de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:09:03.501 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:03.490+0000 7f326affd640 1 -- 172.21.15.104:0/2056163754 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3254003530 con 0x7f3254004810 2024-03-28T14:09:03.501 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:03.490+0000 7f326bfff640 1 --2- 172.21.15.104:0/2056163754 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f326c141500 0x7f32540051d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T14:09:03.505 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:03.494+0000 7f326affd640 1 --2- 172.21.15.104:0/2056163754 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3254004810 0x7f3254004c90 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7f32600098a0 tx=0x7f326001c530 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4fe65e1dfac1e73a server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:09:03.505 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:03.494+0000 7f326a7fc640 1 -- 172.21.15.104:0/2056163754 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f326000f040 con 0x7f3254004810 2024-03-28T14:09:03.505 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:03.494+0000 7f326a7fc640 1 -- 172.21.15.104:0/2056163754 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f32600b0070 con 0x7f3254004810 2024-03-28T14:09:03.505 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:03.494+0000 7f326a7fc640 1 -- 172.21.15.104:0/2056163754 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f32600a7070 con 0x7f3254004810 2024-03-28T14:09:03.506 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:03.494+0000 7f3271fdf640 1 -- 172.21.15.104:0/2056163754 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3254004810 msgr2=0x7f3254004c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:09:03.506 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:03.494+0000 7f3271fdf640 1 --2- 172.21.15.104:0/2056163754 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3254004810 0x7f3254004c90 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7f32600098a0 tx=0x7f326001c530 comp rx=0 tx=0).stop 2024-03-28T14:09:03.506 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:03.494+0000 7f326bfff640 1 -- 172.21.15.104:0/2056163754 reap_dead start 2024-03-28T14:09:03.506 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:03.494+0000 7f3271fdf640 1 -- 172.21.15.104:0/2056163754 shutdown_connections 2024-03-28T14:09:03.506 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:03.494+0000 7f3271fdf640 1 -- 172.21.15.104:0/2056163754 >> 172.21.15.104:0/2056163754 conn(0x7f326c0b9b40 msgr2=0x7f326c0b9f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T14:09:03.506 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:03.494+0000 7f3271fdf640 1 -- 172.21.15.104:0/2056163754 shutdown_connections 2024-03-28T14:09:03.506 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:03.494+0000 7f3271fdf640 1 -- 172.21.15.104:0/2056163754 wait complete. 2024-03-28T14:09:03.507 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:03.494+0000 7f3271fdf640 1 Processor -- start 2024-03-28T14:09:03.507 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:03.494+0000 7f3271fdf640 1 -- start start 2024-03-28T14:09:03.507 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:03.494+0000 7f3271fdf640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f3254004810 0x7f326c0bd3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:09:03.507 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:03.494+0000 7f3271fdf640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f326c14c9b0 0x7f326c1e00b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:09:03.507 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:03.494+0000 7f3271fdf640 1 --2- >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f326c143c40 0x7f326c1440c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:09:03.507 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:03.494+0000 7f3271fdf640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f326c152860 con 0x7f326c14c9b0 2024-03-28T14:09:03.508 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:03.494+0000 7f3271fdf640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f326c1529e0 con 0x7f3254004810 2024-03-28T14:09:03.508 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:03.494+0000 7f3271fdf640 1 -- --> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] -- mon_getmap magic: 0 -- 0x7f326c152b60 con 0x7f326c143c40 2024-03-28T14:09:03.508 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:03.494+0000 7f326affd640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f326c14c9b0 0x7f326c1e00b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:09:03.508 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:03.494+0000 7f326affd640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f326c14c9b0 0x7f326c1e00b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:49940/0 (socket says 172.21.15.104:49940) 2024-03-28T14:09:03.508 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:03.494+0000 7f326affd640 1 -- 172.21.15.104:0/1202812193 learned_addr learned my addr 172.21.15.104:0/1202812193 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-03-28T14:09:03.508 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:03.498+0000 7f326bfff640 1 --2- 172.21.15.104:0/1202812193 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f326c143c40 0x7f326c1440c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:09:03.508 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:03.498+0000 7f326b7fe640 1 --2- 172.21.15.104:0/1202812193 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f3254004810 0x7f326c0bd3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:09:03.508 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:03.498+0000 7f326affd640 1 -- 172.21.15.104:0/1202812193 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f326c143c40 msgr2=0x7f326c1440c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:09:03.509 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:03.498+0000 7f326affd640 1 --2- 172.21.15.104:0/1202812193 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f326c143c40 0x7f326c1440c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:09:03.509 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:03.498+0000 7f326affd640 1 -- 172.21.15.104:0/1202812193 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f3254004810 msgr2=0x7f326c0bd3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:09:03.509 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:03.498+0000 7f326affd640 1 --2- 172.21.15.104:0/1202812193 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f3254004810 0x7f326c0bd3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:09:03.509 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:03.498+0000 7f326affd640 1 -- 172.21.15.104:0/1202812193 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f326c14d100 con 0x7f326c14c9b0 2024-03-28T14:09:03.509 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:03.498+0000 7f326b7fe640 1 --2- 172.21.15.104:0/1202812193 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f3254004810 0x7f326c0bd3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T14:09:03.509 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:03.498+0000 7f326affd640 1 --2- 172.21.15.104:0/1202812193 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f326c14c9b0 0x7f326c1e00b0 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7f326001c360 tx=0x7f3254005840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:09:03.510 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:03.498+0000 7f3253fff640 1 -- 172.21.15.104:0/1202812193 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f32600c2070 con 0x7f326c14c9b0 2024-03-28T14:09:03.510 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:03.498+0000 7f3253fff640 1 -- 172.21.15.104:0/1202812193 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f32600ab530 con 0x7f326c14c9b0 2024-03-28T14:09:03.510 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:03.498+0000 7f3271fdf640 1 -- 172.21.15.104:0/1202812193 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f326c1e3620 con 0x7f326c14c9b0 2024-03-28T14:09:03.510 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:03.498+0000 7f3253fff640 1 -- 172.21.15.104:0/1202812193 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f32600ab940 con 0x7f326c14c9b0 2024-03-28T14:09:03.510 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:03.498+0000 7f3271fdf640 1 -- 172.21.15.104:0/1202812193 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f326c1e3be0 con 0x7f326c14c9b0 2024-03-28T14:09:03.510 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:03.498+0000 7f3271fdf640 1 -- 172.21.15.104:0/1202812193 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f326c02cc50 con 0x7f326c14c9b0 2024-03-28T14:09:03.511 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:03.498+0000 7f3253fff640 1 -- 172.21.15.104:0/1202812193 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) ==== 87803+0+0 (secure 0 0 0) 0x7f32600c5070 con 0x7f326c14c9b0 2024-03-28T14:09:03.511 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:03.498+0000 7f3253fff640 1 --2- 172.21.15.104:0/1202812193 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f3258067fc0 0x7f325806a480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:09:03.511 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:03.498+0000 7f3253fff640 1 -- 172.21.15.104:0/1202812193 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(33..33 src has 1..33) ==== 4943+0+0 (secure 0 0 0) 0x7f326011fa90 con 0x7f326c14c9b0 2024-03-28T14:09:03.511 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:03.498+0000 7f326b7fe640 1 --2- 172.21.15.104:0/1202812193 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f3258067fc0 0x7f325806a480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:09:03.511 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:03.498+0000 7f326b7fe640 1 --2- 172.21.15.104:0/1202812193 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f3258067fc0 0x7f325806a480 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f325c001060 tx=0x7f325c009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:09:03.514 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:03.502+0000 7f3253fff640 1 -- 172.21.15.104:0/1202812193 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179117 (secure 0 0 0) 0x7f3260016240 con 0x7f326c14c9b0 2024-03-28T14:09:03.654 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:03.650+0000 7f3271fdf640 1 -- 172.21.15.104:0/1202812193 --> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "3", "target": ["mon-mgr", ""]}) -- 0x7f326c0687c0 con 0x7f3258067fc0 2024-03-28T14:09:03.655 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:03.650+0000 7f3253fff640 1 -- 172.21.15.104:0/1202812193 <== mgr.4100 v2:172.21.15.171:6832/1522752412 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 3 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7f326c0687c0 con 0x7f3258067fc0 2024-03-28T14:09:03.655 INFO:teuthology.orchestra.run.smithi104.stderr:instructed osd(s) 3 to deep-scrub 2024-03-28T14:09:03.657 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:03.654+0000 7f3271fdf640 1 -- 172.21.15.104:0/1202812193 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f3258067fc0 msgr2=0x7f325806a480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:09:03.657 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:03.654+0000 7f3271fdf640 1 --2- 172.21.15.104:0/1202812193 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f3258067fc0 0x7f325806a480 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f325c001060 tx=0x7f325c009040 comp rx=0 tx=0).stop 2024-03-28T14:09:03.657 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:03.654+0000 7f3271fdf640 1 -- 172.21.15.104:0/1202812193 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f326c14c9b0 msgr2=0x7f326c1e00b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:09:03.657 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:03.654+0000 7f3271fdf640 1 --2- 172.21.15.104:0/1202812193 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f326c14c9b0 0x7f326c1e00b0 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7f326001c360 tx=0x7f3254005840 comp rx=0 tx=0).stop 2024-03-28T14:09:03.657 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:03.654+0000 7f3271fdf640 1 -- 172.21.15.104:0/1202812193 shutdown_connections 2024-03-28T14:09:03.658 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:03.654+0000 7f3271fdf640 1 --2- 172.21.15.104:0/1202812193 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f326c143c40 0x7f326c1440c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:09:03.658 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:03.654+0000 7f3271fdf640 1 --2- 172.21.15.104:0/1202812193 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f3258067fc0 0x7f325806a480 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:09:03.658 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:03.654+0000 7f3271fdf640 1 --2- 172.21.15.104:0/1202812193 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f326c14c9b0 0x7f326c1e00b0 unknown :-1 s=CLOSED pgs=463 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:09:03.658 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:03.654+0000 7f3271fdf640 1 --2- 172.21.15.104:0/1202812193 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f3254004810 0x7f326c0bd3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:09:03.658 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:03.654+0000 7f3271fdf640 1 -- 172.21.15.104:0/1202812193 >> 172.21.15.104:0/1202812193 conn(0x7f326c0b9b40 msgr2=0x7f326c146df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T14:09:03.658 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:03.654+0000 7f3271fdf640 1 -- 172.21.15.104:0/1202812193 shutdown_connections 2024-03-28T14:09:03.658 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:03.654+0000 7f3271fdf640 1 -- 172.21.15.104:0/1202812193 wait complete. 2024-03-28T14:09:03.672 INFO:tasks.ceph:Scrubbing osd.4 2024-03-28T14:09:03.673 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 config set osd_debug_deep_scrub_sleep 0 2024-03-28T14:09:03.802 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:03.798+0000 7fa274683640 1 Processor -- start 2024-03-28T14:09:03.803 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:03.798+0000 7fa274683640 1 -- start start 2024-03-28T14:09:03.805 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:03.798+0000 7fa274683640 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fa26c0bb8c0 0x7fa26c0b5950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:09:03.805 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:03.798+0000 7fa274683640 1 --2- >> v2:172.21.15.171:3300/0 conn(0x7fa26c0b6790 0x7fa26c13f470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:09:03.805 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:03.798+0000 7fa274683640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa26c0bbca0 con 0x7fa26c13f9b0 2024-03-28T14:09:03.805 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:03.798+0000 7fa274683640 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 -- 0x7fa26c0b7430 con 0x7fa26c0bb8c0 2024-03-28T14:09:03.805 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:03.798+0000 7fa274683640 1 -- --> v2:172.21.15.171:3300/0 -- mon_getmap magic: 0 -- 0x7fa26c0b6de0 con 0x7fa26c0b6790 2024-03-28T14:09:03.805 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:03.798+0000 7fa27241f640 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fa26c0bb8c0 0x7fa26c0b5950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:09:03.805 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:03.798+0000 7fa27241f640 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fa26c0bb8c0 0x7fa26c0b5950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:49956/0 (socket says 172.21.15.104:49956) 2024-03-28T14:09:03.806 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:03.798+0000 7fa272c20640 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7fa26c13f9b0 0x7fa26c0b1710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:40604/0 (socket says 172.21.15.104:40604) 2024-03-28T14:09:03.806 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:03.798+0000 7fa27241f640 1 -- 172.21.15.104:0/4056494667 learned_addr learned my addr 172.21.15.104:0/4056494667 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-03-28T14:09:03.806 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:03.802+0000 7fa271c1e640 1 --2- 172.21.15.104:0/4056494667 >> v2:172.21.15.171:3300/0 conn(0x7fa26c0b6790 0x7fa26c13f470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:09:03.806 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:03.802+0000 7fa27241f640 1 -- 172.21.15.104:0/4056494667 >> v2:172.21.15.171:3300/0 conn(0x7fa26c0b6790 msgr2=0x7fa26c13f470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T14:09:03.806 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:03.802+0000 7fa27241f640 1 --2- 172.21.15.104:0/4056494667 >> v2:172.21.15.171:3300/0 conn(0x7fa26c0b6790 0x7fa26c13f470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:09:03.807 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:03.802+0000 7fa27241f640 1 -- 172.21.15.104:0/4056494667 >> v1:172.21.15.104:6789/0 conn(0x7fa26c13f9b0 legacy=0x7fa26c0b1710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:09:03.807 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:03.802+0000 7fa27241f640 1 -- 172.21.15.104:0/4056494667 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa26c0b70c0 con 0x7fa26c0bb8c0 2024-03-28T14:09:03.807 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:03.802+0000 7fa27141d640 1 -- 172.21.15.104:0/4056494667 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3536075657 0 0) 0x7fa26c0bbca0 con 0x7fa26c13f9b0 2024-03-28T14:09:03.807 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:03.802+0000 7fa271c1e640 1 --2- 172.21.15.104:0/4056494667 >> v2:172.21.15.171:3300/0 conn(0x7fa26c0b6790 0x7fa26c13f470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T14:09:03.812 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:03.802+0000 7fa27241f640 1 --2- 172.21.15.104:0/4056494667 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa26c0bb8c0 0x7fa26c0b5950 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7fa26c0b2b60 tx=0x7fa26801c400 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7f1a7c2fcb36407f server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:09:03.812 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:03.802+0000 7fa27141d640 1 -- 172.21.15.104:0/4056494667 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2680b0070 con 0x7fa26c0bb8c0 2024-03-28T14:09:03.812 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:03.802+0000 7fa27141d640 1 -- 172.21.15.104:0/4056494667 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa2680ab600 con 0x7fa26c0bb8c0 2024-03-28T14:09:03.812 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:03.802+0000 7fa27141d640 1 -- 172.21.15.104:0/4056494667 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2680ab9d0 con 0x7fa26c0bb8c0 2024-03-28T14:09:03.813 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:03.802+0000 7fa274683640 1 -- 172.21.15.104:0/4056494667 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa26c0bb8c0 msgr2=0x7fa26c0b5950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:09:03.813 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:03.802+0000 7fa274683640 1 --2- 172.21.15.104:0/4056494667 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa26c0bb8c0 0x7fa26c0b5950 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7fa26c0b2b60 tx=0x7fa26801c400 comp rx=0 tx=0).stop 2024-03-28T14:09:03.813 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:03.802+0000 7fa274683640 1 -- 172.21.15.104:0/4056494667 shutdown_connections 2024-03-28T14:09:03.813 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:03.802+0000 7fa274683640 1 --2- 172.21.15.104:0/4056494667 >> v2:172.21.15.171:3300/0 conn(0x7fa26c0b6790 0x7fa26c13f470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:09:03.813 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:03.802+0000 7fa274683640 1 --2- 172.21.15.104:0/4056494667 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa26c0bb8c0 0x7fa26c0b5950 unknown :-1 s=CLOSED pgs=465 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:09:03.814 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:03.802+0000 7fa274683640 1 -- 172.21.15.104:0/4056494667 >> 172.21.15.104:0/4056494667 conn(0x7fa26c02c800 msgr2=0x7fa26c0baf50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T14:09:03.814 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:03.802+0000 7fa274683640 1 -- 172.21.15.104:0/4056494667 shutdown_connections 2024-03-28T14:09:03.814 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:03.802+0000 7fa274683640 1 -- 172.21.15.104:0/4056494667 wait complete. 2024-03-28T14:09:03.814 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:03.806+0000 7fa274683640 1 Processor -- start 2024-03-28T14:09:03.814 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:03.806+0000 7fa274683640 1 -- start start 2024-03-28T14:09:03.814 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:03.806+0000 7fa274683640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fa26c0b6790 0x7fa26c13df10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:09:03.815 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:03.806+0000 7fa274683640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa26c0bb8c0 0x7fa26c1e1300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:09:03.815 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:03.806+0000 7fa274683640 1 --2- >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fa26c13f9b0 0x7fa26c1320e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:09:03.815 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:03.806+0000 7fa274683640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7fa26c152a10 con 0x7fa26c0bb8c0 2024-03-28T14:09:03.815 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:03.806+0000 7fa274683640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7fa26c152890 con 0x7fa26c0b6790 2024-03-28T14:09:03.815 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:03.806+0000 7fa274683640 1 -- --> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] -- mon_getmap magic: 0 -- 0x7fa26c152b90 con 0x7fa26c13f9b0 2024-03-28T14:09:03.815 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:03.806+0000 7fa271c1e640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa26c0bb8c0 0x7fa26c1e1300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:09:03.815 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:03.806+0000 7fa271c1e640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa26c0bb8c0 0x7fa26c1e1300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:49970/0 (socket says 172.21.15.104:49970) 2024-03-28T14:09:03.815 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:03.806+0000 7fa271c1e640 1 -- 172.21.15.104:0/1613369828 learned_addr learned my addr 172.21.15.104:0/1613369828 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-03-28T14:09:03.815 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:03.806+0000 7fa27241f640 1 --2- 172.21.15.104:0/1613369828 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fa26c0b6790 0x7fa26c13df10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:09:03.816 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:03.806+0000 7fa272c20640 1 --2- 172.21.15.104:0/1613369828 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fa26c13f9b0 0x7fa26c1320e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:09:03.816 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:03.806+0000 7fa271c1e640 1 -- 172.21.15.104:0/1613369828 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fa26c13f9b0 msgr2=0x7fa26c1320e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:09:03.816 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:03.806+0000 7fa271c1e640 1 --2- 172.21.15.104:0/1613369828 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fa26c13f9b0 0x7fa26c1320e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:09:03.816 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:03.806+0000 7fa271c1e640 1 -- 172.21.15.104:0/1613369828 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fa26c0b6790 msgr2=0x7fa26c13df10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:09:03.816 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:03.806+0000 7fa271c1e640 1 --2- 172.21.15.104:0/1613369828 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fa26c0b6790 0x7fa26c13df10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:09:03.816 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:03.806+0000 7fa271c1e640 1 -- 172.21.15.104:0/1613369828 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa26c13e740 con 0x7fa26c0bb8c0 2024-03-28T14:09:03.816 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:03.806+0000 7fa271c1e640 1 --2- 172.21.15.104:0/1613369828 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa26c0bb8c0 0x7fa26c1e1300 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7fa25c00c7f0 tx=0x7fa25c00ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:09:03.816 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:03.806+0000 7fa262ffd640 1 -- 172.21.15.104:0/1613369828 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa25c007a30 con 0x7fa26c0bb8c0 2024-03-28T14:09:03.817 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:03.806+0000 7fa262ffd640 1 -- 172.21.15.104:0/1613369828 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa25c007bd0 con 0x7fa26c0bb8c0 2024-03-28T14:09:03.817 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:03.806+0000 7fa274683640 1 -- 172.21.15.104:0/1613369828 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa26c0ba150 con 0x7fa26c0bb8c0 2024-03-28T14:09:03.817 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:03.806+0000 7fa262ffd640 1 -- 172.21.15.104:0/1613369828 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa25c0113f0 con 0x7fa26c0bb8c0 2024-03-28T14:09:03.817 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:03.806+0000 7fa274683640 1 -- 172.21.15.104:0/1613369828 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa26c139120 con 0x7fa26c0bb8c0 2024-03-28T14:09:03.817 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:03.806+0000 7fa274683640 1 -- 172.21.15.104:0/1613369828 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fa26c1407b0 con 0x7fa26c0bb8c0 2024-03-28T14:09:03.817 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:03.806+0000 7fa262ffd640 1 -- 172.21.15.104:0/1613369828 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) ==== 87803+0+0 (secure 0 0 0) 0x7fa25c011590 con 0x7fa26c0bb8c0 2024-03-28T14:09:03.817 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:03.806+0000 7fa262ffd640 1 --2- 172.21.15.104:0/1613369828 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7fa250067fc0 0x7fa25006a480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:09:03.817 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:03.806+0000 7fa262ffd640 1 -- 172.21.15.104:0/1613369828 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(33..33 src has 1..33) ==== 4943+0+0 (secure 0 0 0) 0x7fa25c086920 con 0x7fa26c0bb8c0 2024-03-28T14:09:03.817 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:03.806+0000 7fa262ffd640 1 --2- 172.21.15.104:0/1613369828 >> [v2:172.21.15.171:6800/2107001077,v1:172.21.15.171:6801/2107001077] conn(0x7fa25006de80 0x7fa2500702a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:09:03.818 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:03.806+0000 7fa262ffd640 1 -- 172.21.15.104:0/1613369828 --> [v2:172.21.15.171:6800/2107001077,v1:172.21.15.171:6801/2107001077] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fa25c018ec0 con 0x7fa25006de80 2024-03-28T14:09:03.818 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:03.806+0000 7fa262ffd640 1 -- 172.21.15.104:0/1613369828 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7fa25c086db0 con 0x7fa26c0bb8c0 2024-03-28T14:09:03.818 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:03.806+0000 7fa27241f640 1 --2- 172.21.15.104:0/1613369828 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7fa250067fc0 0x7fa25006a480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:09:03.818 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:03.810+0000 7fa272c20640 1 --2- 172.21.15.104:0/1613369828 >> [v2:172.21.15.171:6800/2107001077,v1:172.21.15.171:6801/2107001077] conn(0x7fa25006de80 0x7fa2500702a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:09:03.818 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:03.810+0000 7fa27241f640 1 --2- 172.21.15.104:0/1613369828 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7fa250067fc0 0x7fa25006a480 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fa2680ad000 tx=0x7fa2680a6000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:09:03.819 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:03.810+0000 7fa272c20640 1 --2- 172.21.15.104:0/1613369828 >> [v2:172.21.15.171:6800/2107001077,v1:172.21.15.171:6801/2107001077] conn(0x7fa25006de80 0x7fa2500702a0 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:09:03.819 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:03.810+0000 7fa262ffd640 1 -- 172.21.15.104:0/1613369828 <== osd.4 v2:172.21.15.171:6800/2107001077 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fa25c018ec0 con 0x7fa25006de80 2024-03-28T14:09:03.827 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:03.822+0000 7fa274683640 1 -- 172.21.15.104:0/1613369828 --> [v2:172.21.15.171:6800/2107001077,v1:172.21.15.171:6801/2107001077] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7fa26c0687c0 con 0x7fa25006de80 2024-03-28T14:09:03.828 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:03.826+0000 7fa262ffd640 1 -- 172.21.15.104:0/1613369828 <== osd.4 v2:172.21.15.171:6800/2107001077 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7fa26c0687c0 con 0x7fa25006de80 2024-03-28T14:09:03.828 INFO:teuthology.orchestra.run.smithi104.stdout:{ 2024-03-28T14:09:03.829 INFO:teuthology.orchestra.run.smithi104.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-03-28T14:09:03.829 INFO:teuthology.orchestra.run.smithi104.stdout:} 2024-03-28T14:09:03.830 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:03.826+0000 7fa274683640 1 -- 172.21.15.104:0/1613369828 >> [v2:172.21.15.171:6800/2107001077,v1:172.21.15.171:6801/2107001077] conn(0x7fa25006de80 msgr2=0x7fa2500702a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:09:03.830 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:03.826+0000 7fa274683640 1 --2- 172.21.15.104:0/1613369828 >> [v2:172.21.15.171:6800/2107001077,v1:172.21.15.171:6801/2107001077] conn(0x7fa25006de80 0x7fa2500702a0 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:09:03.830 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:03.826+0000 7fa274683640 1 -- 172.21.15.104:0/1613369828 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7fa250067fc0 msgr2=0x7fa25006a480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:09:03.830 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:03.826+0000 7fa274683640 1 --2- 172.21.15.104:0/1613369828 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7fa250067fc0 0x7fa25006a480 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fa2680ad000 tx=0x7fa2680a6000 comp rx=0 tx=0).stop 2024-03-28T14:09:03.830 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:03.826+0000 7fa274683640 1 -- 172.21.15.104:0/1613369828 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa26c0bb8c0 msgr2=0x7fa26c1e1300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:09:03.830 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:03.826+0000 7fa274683640 1 --2- 172.21.15.104:0/1613369828 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa26c0bb8c0 0x7fa26c1e1300 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7fa25c00c7f0 tx=0x7fa25c00ccb0 comp rx=0 tx=0).stop 2024-03-28T14:09:03.831 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:03.826+0000 7fa272c20640 1 -- 172.21.15.104:0/1613369828 reap_dead start 2024-03-28T14:09:03.831 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:03.826+0000 7fa274683640 1 -- 172.21.15.104:0/1613369828 shutdown_connections 2024-03-28T14:09:03.831 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:03.826+0000 7fa274683640 1 -- 172.21.15.104:0/1613369828 >> 172.21.15.104:0/1613369828 conn(0x7fa26c02c800 msgr2=0x7fa26c072220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T14:09:03.831 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:03.826+0000 7fa274683640 1 -- 172.21.15.104:0/1613369828 shutdown_connections 2024-03-28T14:09:03.831 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:03.826+0000 7fa274683640 1 -- 172.21.15.104:0/1613369828 wait complete. 2024-03-28T14:09:03.844 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 4 2024-03-28T14:09:03.949 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:03.946+0000 7fa0364b3640 1 Processor -- start 2024-03-28T14:09:03.950 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:03.946+0000 7fa0364b3640 1 -- start start 2024-03-28T14:09:03.952 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:03.946+0000 7fa0364b3640 1 --2- >> v2:172.21.15.171:3300/0 conn(0x7fa03006d7e0 0x7fa0300af680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:09:03.953 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:03.946+0000 7fa0364b3640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa0300b4e80 con 0x7fa0300af070 2024-03-28T14:09:03.953 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:03.946+0000 7fa0364b3640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa03006dbc0 con 0x7fa0300afbc0 2024-03-28T14:09:03.953 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:03.946+0000 7fa0364b3640 1 -- --> v2:172.21.15.171:3300/0 -- mon_getmap magic: 0 -- 0x7fa0300b1210 con 0x7fa03006d7e0 2024-03-28T14:09:03.953 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:03.946+0000 7fa034cb0640 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7fa0300af070 0x7fa030147e20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:40614/0 (socket says 172.21.15.104:40614) 2024-03-28T14:09:03.953 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:03.946+0000 7fa034cb0640 1 -- 172.21.15.104:0/1656975970 learned_addr learned my addr 172.21.15.104:0/1656975970 (peer_addr_for_me v1:172.21.15.104:0/0) 2024-03-28T14:09:03.953 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:03.946+0000 7fa027fff640 1 -- 172.21.15.104:0/1656975970 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 163572910 0 0) 0x7fa0300b4e80 con 0x7fa0300af070 2024-03-28T14:09:03.953 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:03.946+0000 7fa027fff640 1 -- 172.21.15.104:0/1656975970 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa01c003530 con 0x7fa0300af070 2024-03-28T14:09:03.954 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:03.946+0000 7fa0354b1640 1 --2- 172.21.15.104:0/1656975970 >> v2:172.21.15.171:3300/0 conn(0x7fa03006d7e0 0x7fa0300af680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:09:03.954 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:03.946+0000 7fa027fff640 1 -- 172.21.15.104:0/1656975970 <== mon.0 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2051501523 0 0) 0x7fa01c003530 con 0x7fa0300af070 2024-03-28T14:09:03.954 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:03.946+0000 7fa027fff640 1 -- 172.21.15.104:0/1656975970 >> v2:172.21.15.171:3300/0 conn(0x7fa03006d7e0 msgr2=0x7fa0300af680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T14:09:03.954 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:03.946+0000 7fa027fff640 1 --2- 172.21.15.104:0/1656975970 >> v2:172.21.15.171:3300/0 conn(0x7fa03006d7e0 0x7fa0300af680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:09:03.954 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:03.946+0000 7fa027fff640 1 -- 172.21.15.104:0/1656975970 >> v1:172.21.15.171:6789/0 conn(0x7fa0300afbc0 legacy=0x7fa03014ed40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:09:03.954 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:03.946+0000 7fa027fff640 1 -- 172.21.15.104:0/1656975970 --> v1:172.21.15.104:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa0300b14f0 con 0x7fa0300af070 2024-03-28T14:09:03.954 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:03.946+0000 7fa027fff640 1 -- 172.21.15.104:0/1656975970 <== mon.0 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3745580849 0 0) 0x7fa02c001680 con 0x7fa0300af070 2024-03-28T14:09:03.955 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:03.946+0000 7fa027fff640 1 -- 172.21.15.104:0/1656975970 >> v1:172.21.15.104:6789/0 conn(0x7fa0300af070 legacy=0x7fa030147e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:09:03.955 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:03.946+0000 7fa027fff640 1 --2- 172.21.15.104:0/1656975970 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa01c003a10 0x7fa01c003e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:09:03.955 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:03.946+0000 7fa027fff640 1 --2- 172.21.15.104:0/1656975970 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fa01c004810 0x7fa01c004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:09:03.955 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:03.946+0000 7fa027fff640 1 --2- 172.21.15.104:0/1656975970 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fa01c005220 0x7fa01c00b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:09:03.955 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:03.946+0000 7fa027fff640 1 -- 172.21.15.104:0/1656975970 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7fa0300b1210 con 0x7fa01c003a10 2024-03-28T14:09:03.955 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:03.946+0000 7fa027fff640 1 -- 172.21.15.104:0/1656975970 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7fa01c00c170 con 0x7fa01c005220 2024-03-28T14:09:03.955 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:03.946+0000 7fa027fff640 1 -- 172.21.15.104:0/1656975970 --> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] -- mon_getmap magic: 0 -- 0x7fa01c00c370 con 0x7fa01c004810 2024-03-28T14:09:03.955 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:03.946+0000 7fa0354b1640 1 --2- 172.21.15.104:0/1656975970 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa01c003a10 0x7fa01c003e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:09:03.956 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:03.946+0000 7fa034cb0640 1 --2- 172.21.15.104:0/1656975970 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fa01c004810 0x7fa01c004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:09:03.956 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:03.946+0000 7fa035cb2640 1 --2- 172.21.15.104:0/1656975970 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fa01c005220 0x7fa01c00b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:09:03.956 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:03.946+0000 7fa0354b1640 1 -- 172.21.15.104:0/1656975970 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fa01c004810 msgr2=0x7fa01c004c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T14:09:03.956 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:03.946+0000 7fa0354b1640 1 --2- 172.21.15.104:0/1656975970 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fa01c004810 0x7fa01c004c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:09:03.956 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:03.946+0000 7fa0354b1640 1 -- 172.21.15.104:0/1656975970 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fa01c005220 msgr2=0x7fa01c00b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T14:09:03.956 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:03.946+0000 7fa0354b1640 1 --2- 172.21.15.104:0/1656975970 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fa01c005220 0x7fa01c00b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:09:03.956 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:03.946+0000 7fa0354b1640 1 -- 172.21.15.104:0/1656975970 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa01c003530 con 0x7fa01c003a10 2024-03-28T14:09:03.957 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:03.946+0000 7fa035cb2640 1 --2- 172.21.15.104:0/1656975970 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fa01c005220 0x7fa01c00b9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T14:09:03.957 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:03.946+0000 7fa035cb2640 1 -- 172.21.15.104:0/1656975970 reap_dead start 2024-03-28T14:09:03.963 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:03.950+0000 7fa0354b1640 1 --2- 172.21.15.104:0/1656975970 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa01c003a10 0x7fa01c003e30 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7fa02000ba30 tx=0x7fa02001c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fb9ee4430ac0189a server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:09:03.963 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:03.950+0000 7fa027fff640 1 -- 172.21.15.104:0/1656975970 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0200b3070 con 0x7fa01c003a10 2024-03-28T14:09:03.963 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:03.950+0000 7fa027fff640 1 -- 172.21.15.104:0/1656975970 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa0200ae630 con 0x7fa01c003a10 2024-03-28T14:09:03.963 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:03.950+0000 7fa027fff640 1 -- 172.21.15.104:0/1656975970 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0200aa050 con 0x7fa01c003a10 2024-03-28T14:09:03.963 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:03.950+0000 7fa0364b3640 1 -- 172.21.15.104:0/1656975970 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa01c003a10 msgr2=0x7fa01c003e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:09:03.964 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:03.950+0000 7fa0364b3640 1 --2- 172.21.15.104:0/1656975970 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa01c003a10 0x7fa01c003e30 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7fa02000ba30 tx=0x7fa02001c6a0 comp rx=0 tx=0).stop 2024-03-28T14:09:03.964 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:03.950+0000 7fa0364b3640 1 -- 172.21.15.104:0/1656975970 shutdown_connections 2024-03-28T14:09:03.964 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:03.950+0000 7fa0364b3640 1 --2- 172.21.15.104:0/1656975970 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa01c003a10 0x7fa01c003e30 unknown :-1 s=CLOSED pgs=468 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:09:03.964 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:03.950+0000 7fa0364b3640 1 -- 172.21.15.104:0/1656975970 >> 172.21.15.104:0/1656975970 conn(0x7fa0300b57a0 msgr2=0x7fa0300b5bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T14:09:03.964 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:03.950+0000 7fa0364b3640 1 -- 172.21.15.104:0/1656975970 shutdown_connections 2024-03-28T14:09:03.965 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:03.950+0000 7fa0364b3640 1 -- 172.21.15.104:0/1656975970 wait complete. 2024-03-28T14:09:03.965 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:03.954+0000 7fa0364b3640 1 Processor -- start 2024-03-28T14:09:03.965 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:03.954+0000 7fa0364b3640 1 -- start start 2024-03-28T14:09:03.965 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:03.954+0000 7fa0364b3640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fa01c003a10 0x7fa0301e3570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:09:03.966 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:03.954+0000 7fa0364b3640 1 --2- >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fa0300700f0 0x7fa030070570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:09:03.966 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:03.954+0000 7fa0364b3640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa03009e230 0x7fa03009e6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:09:03.967 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:03.954+0000 7fa0364b3640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7fa030148630 con 0x7fa03009e230 2024-03-28T14:09:03.967 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:03.954+0000 7fa0364b3640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7fa0300b0550 con 0x7fa01c003a10 2024-03-28T14:09:03.967 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:03.954+0000 7fa0364b3640 1 -- --> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] -- mon_getmap magic: 0 -- 0x7fa0301487b0 con 0x7fa0300700f0 2024-03-28T14:09:03.967 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:03.954+0000 7fa035cb2640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa03009e230 0x7fa03009e6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:09:03.968 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:03.954+0000 7fa035cb2640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa03009e230 0x7fa03009e6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:49992/0 (socket says 172.21.15.104:49992) 2024-03-28T14:09:03.968 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:03.954+0000 7fa035cb2640 1 -- 172.21.15.104:0/4039051277 learned_addr learned my addr 172.21.15.104:0/4039051277 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-03-28T14:09:03.968 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:03.954+0000 7fa0354b1640 1 --2- 172.21.15.104:0/4039051277 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fa01c003a10 0x7fa0301e3570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:09:03.968 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:03.954+0000 7fa034cb0640 1 --2- 172.21.15.104:0/4039051277 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fa0300700f0 0x7fa030070570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:09:03.969 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:03.954+0000 7fa035cb2640 1 -- 172.21.15.104:0/4039051277 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fa0300700f0 msgr2=0x7fa030070570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:09:03.969 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:03.954+0000 7fa035cb2640 1 --2- 172.21.15.104:0/4039051277 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fa0300700f0 0x7fa030070570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:09:03.969 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:03.954+0000 7fa035cb2640 1 -- 172.21.15.104:0/4039051277 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fa01c003a10 msgr2=0x7fa0301e3570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:09:03.970 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:03.954+0000 7fa035cb2640 1 --2- 172.21.15.104:0/4039051277 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fa01c003a10 0x7fa0301e3570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:09:03.970 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:03.954+0000 7fa035cb2640 1 -- 172.21.15.104:0/4039051277 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa0301e4120 con 0x7fa03009e230 2024-03-28T14:09:03.970 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:03.954+0000 7fa034cb0640 1 --2- 172.21.15.104:0/4039051277 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fa0300700f0 0x7fa030070570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T14:09:03.971 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:03.954+0000 7fa0354b1640 1 --2- 172.21.15.104:0/4039051277 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fa01c003a10 0x7fa0301e3570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T14:09:03.971 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:03.954+0000 7fa035cb2640 1 --2- 172.21.15.104:0/4039051277 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa03009e230 0x7fa03009e6b0 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7fa03006e060 tx=0x7fa02800d9d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:09:03.971 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:03.954+0000 7fa025ffb640 1 -- 172.21.15.104:0/4039051277 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa028018070 con 0x7fa03009e230 2024-03-28T14:09:03.971 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:03.954+0000 7fa025ffb640 1 -- 172.21.15.104:0/4039051277 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa028003db0 con 0x7fa03009e230 2024-03-28T14:09:03.971 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:03.954+0000 7fa0364b3640 1 -- 172.21.15.104:0/4039051277 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa030070ab0 con 0x7fa03009e230 2024-03-28T14:09:03.971 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:03.954+0000 7fa025ffb640 1 -- 172.21.15.104:0/4039051277 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0280052b0 con 0x7fa03009e230 2024-03-28T14:09:03.972 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:03.954+0000 7fa0364b3640 1 -- 172.21.15.104:0/4039051277 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa03009ca20 con 0x7fa03009e230 2024-03-28T14:09:03.972 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:03.954+0000 7fa0364b3640 1 -- 172.21.15.104:0/4039051277 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa030068640 con 0x7fa03009e230 2024-03-28T14:09:03.972 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:03.954+0000 7fa025ffb640 1 -- 172.21.15.104:0/4039051277 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) ==== 87803+0+0 (secure 0 0 0) 0x7fa02801b070 con 0x7fa03009e230 2024-03-28T14:09:03.972 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:03.954+0000 7fa025ffb640 1 --2- 172.21.15.104:0/4039051277 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7fa014067fc0 0x7fa01406a480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:09:03.972 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:03.954+0000 7fa025ffb640 1 -- 172.21.15.104:0/4039051277 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(33..33 src has 1..33) ==== 4943+0+0 (secure 0 0 0) 0x7fa028085800 con 0x7fa03009e230 2024-03-28T14:09:03.972 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:03.954+0000 7fa0354b1640 1 --2- 172.21.15.104:0/4039051277 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7fa014067fc0 0x7fa01406a480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:09:03.973 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:03.958+0000 7fa0354b1640 1 --2- 172.21.15.104:0/4039051277 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7fa014067fc0 0x7fa01406a480 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fa0200b0000 tx=0x7fa0200046d0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:09:03.973 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:03.958+0000 7fa025ffb640 1 -- 172.21.15.104:0/4039051277 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179117 (secure 0 0 0) 0x7fa030070ab0 con 0x7fa03009e230 2024-03-28T14:09:04.112 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:04.110+0000 7fa0364b3640 1 -- 172.21.15.104:0/4039051277 --> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "4", "target": ["mon-mgr", ""]}) -- 0x7fa0300b6f90 con 0x7fa014067fc0 2024-03-28T14:09:04.113 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:04.110+0000 7fa025ffb640 1 -- 172.21.15.104:0/4039051277 <== mgr.4100 v2:172.21.15.171:6832/1522752412 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 4 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7fa0300b6f90 con 0x7fa014067fc0 2024-03-28T14:09:04.113 INFO:teuthology.orchestra.run.smithi104.stderr:instructed osd(s) 4 to deep-scrub 2024-03-28T14:09:04.115 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:04.110+0000 7fa0364b3640 1 -- 172.21.15.104:0/4039051277 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7fa014067fc0 msgr2=0x7fa01406a480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:09:04.115 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:04.110+0000 7fa0364b3640 1 --2- 172.21.15.104:0/4039051277 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7fa014067fc0 0x7fa01406a480 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fa0200b0000 tx=0x7fa0200046d0 comp rx=0 tx=0).stop 2024-03-28T14:09:04.116 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:04.110+0000 7fa0364b3640 1 -- 172.21.15.104:0/4039051277 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa03009e230 msgr2=0x7fa03009e6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:09:04.116 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:04.110+0000 7fa0364b3640 1 --2- 172.21.15.104:0/4039051277 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa03009e230 0x7fa03009e6b0 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7fa03006e060 tx=0x7fa02800d9d0 comp rx=0 tx=0).stop 2024-03-28T14:09:04.116 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:04.110+0000 7fa0364b3640 1 -- 172.21.15.104:0/4039051277 shutdown_connections 2024-03-28T14:09:04.116 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:04.110+0000 7fa0364b3640 1 --2- 172.21.15.104:0/4039051277 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7fa014067fc0 0x7fa01406a480 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:09:04.116 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:04.110+0000 7fa0364b3640 1 --2- 172.21.15.104:0/4039051277 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa03009e230 0x7fa03009e6b0 unknown :-1 s=CLOSED pgs=469 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:09:04.116 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:04.110+0000 7fa0364b3640 1 --2- 172.21.15.104:0/4039051277 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fa0300700f0 0x7fa030070570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:09:04.116 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:04.110+0000 7fa0364b3640 1 --2- 172.21.15.104:0/4039051277 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fa01c003a10 0x7fa0301e3570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:09:04.117 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:04.110+0000 7fa0364b3640 1 -- 172.21.15.104:0/4039051277 >> 172.21.15.104:0/4039051277 conn(0x7fa0300b57a0 msgr2=0x7fa030067330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T14:09:04.117 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:04.110+0000 7fa0364b3640 1 -- 172.21.15.104:0/4039051277 shutdown_connections 2024-03-28T14:09:04.117 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:04.110+0000 7fa0364b3640 1 -- 172.21.15.104:0/4039051277 wait complete. 2024-03-28T14:09:04.131 INFO:tasks.ceph:Scrubbing osd.5 2024-03-28T14:09:04.131 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 config set osd_debug_deep_scrub_sleep 0 2024-03-28T14:09:04.258 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:04.254+0000 7fe68d324640 1 Processor -- start 2024-03-28T14:09:04.258 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:04.254+0000 7fe68d324640 1 -- start start 2024-03-28T14:09:04.259 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:04.254+0000 7fe68d324640 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fe68806c8f0 0x7fe6880b2740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:09:04.259 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:04.254+0000 7fe68d324640 1 --2- >> v2:172.21.15.171:3300/0 conn(0x7fe6880b3730 0x7fe688148c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:09:04.260 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:04.254+0000 7fe68d324640 1 --2- >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fe6881491b0 0x7fe688149620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:09:04.260 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:04.254+0000 7fe68d324640 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 -- 0x7fe6880b0b30 con 0x7fe68806c8f0 2024-03-28T14:09:04.260 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:04.254+0000 7fe68d324640 1 -- --> v2:172.21.15.171:3300/0 -- mon_getmap magic: 0 -- 0x7fe6880b22b0 con 0x7fe6880b3730 2024-03-28T14:09:04.260 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:04.254+0000 7fe68d324640 1 -- --> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] -- mon_getmap magic: 0 -- 0x7fe6880b3b60 con 0x7fe6881491b0 2024-03-28T14:09:04.260 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:04.254+0000 7fe686ffd640 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fe68806c8f0 0x7fe6880b2740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:09:04.260 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:04.254+0000 7fe686ffd640 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fe68806c8f0 0x7fe6880b2740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:50004/0 (socket says 172.21.15.104:50004) 2024-03-28T14:09:04.260 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:04.254+0000 7fe686ffd640 1 -- 172.21.15.104:0/4239921849 learned_addr learned my addr 172.21.15.104:0/4239921849 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-03-28T14:09:04.260 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:04.254+0000 7fe6877fe640 1 --2- 172.21.15.104:0/4239921849 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fe6881491b0 0x7fe688149620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:09:04.260 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:04.254+0000 7fe6867fc640 1 --2- 172.21.15.104:0/4239921849 >> v2:172.21.15.171:3300/0 conn(0x7fe6880b3730 0x7fe688148c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:09:04.261 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:04.254+0000 7fe686ffd640 1 -- 172.21.15.104:0/4239921849 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fe6881491b0 msgr2=0x7fe688149620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T14:09:04.261 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:04.254+0000 7fe686ffd640 1 --2- 172.21.15.104:0/4239921849 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fe6881491b0 0x7fe688149620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:09:04.261 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:04.254+0000 7fe686ffd640 1 -- 172.21.15.104:0/4239921849 >> v2:172.21.15.171:3300/0 conn(0x7fe6880b3730 msgr2=0x7fe688148c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T14:09:04.261 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:04.254+0000 7fe686ffd640 1 --2- 172.21.15.104:0/4239921849 >> v2:172.21.15.171:3300/0 conn(0x7fe6880b3730 0x7fe688148c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:09:04.261 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:04.254+0000 7fe686ffd640 1 -- 172.21.15.104:0/4239921849 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe68806ccd0 con 0x7fe68806c8f0 2024-03-28T14:09:04.262 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:04.258+0000 7fe686ffd640 1 --2- 172.21.15.104:0/4239921849 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe68806c8f0 0x7fe6880b2740 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7fe688072d50 tx=0x7fe67001c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=29ec83f825e6bbfd server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:09:04.262 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:04.258+0000 7fe685ffb640 1 -- 172.21.15.104:0/4239921849 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6700ab650 con 0x7fe68806c8f0 2024-03-28T14:09:04.263 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:04.258+0000 7fe685ffb640 1 -- 172.21.15.104:0/4239921849 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe6700b0070 con 0x7fe68806c8f0 2024-03-28T14:09:04.264 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:04.258+0000 7fe685ffb640 1 -- 172.21.15.104:0/4239921849 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6700aba10 con 0x7fe68806c8f0 2024-03-28T14:09:04.264 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:04.258+0000 7fe68d324640 1 -- 172.21.15.104:0/4239921849 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe68806c8f0 msgr2=0x7fe6880b2740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:09:04.264 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:04.258+0000 7fe68d324640 1 --2- 172.21.15.104:0/4239921849 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe68806c8f0 0x7fe6880b2740 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7fe688072d50 tx=0x7fe67001c490 comp rx=0 tx=0).stop 2024-03-28T14:09:04.264 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:04.258+0000 7fe68d324640 1 -- 172.21.15.104:0/4239921849 shutdown_connections 2024-03-28T14:09:04.264 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:04.258+0000 7fe68d324640 1 --2- 172.21.15.104:0/4239921849 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fe6881491b0 0x7fe688149620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:09:04.264 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:04.258+0000 7fe68d324640 1 --2- 172.21.15.104:0/4239921849 >> v2:172.21.15.171:3300/0 conn(0x7fe6880b3730 0x7fe688148c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:09:04.264 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:04.258+0000 7fe68d324640 1 --2- 172.21.15.104:0/4239921849 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe68806c8f0 0x7fe6880b2740 unknown :-1 s=CLOSED pgs=470 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:09:04.264 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:04.258+0000 7fe68d324640 1 -- 172.21.15.104:0/4239921849 >> 172.21.15.104:0/4239921849 conn(0x7fe6880b8f00 msgr2=0x7fe6880b9310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T14:09:04.264 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:04.258+0000 7fe68d324640 1 -- 172.21.15.104:0/4239921849 shutdown_connections 2024-03-28T14:09:04.265 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:04.258+0000 7fe68d324640 1 -- 172.21.15.104:0/4239921849 wait complete. 2024-03-28T14:09:04.265 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:04.258+0000 7fe68d324640 1 Processor -- start 2024-03-28T14:09:04.265 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:04.258+0000 7fe68d324640 1 -- start start 2024-03-28T14:09:04.266 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:04.258+0000 7fe68d324640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fe68806c8f0 0x7fe6881dfbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:09:04.266 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:04.258+0000 7fe68d324640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe6880b3730 0x7fe68814ed90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:09:04.266 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:04.258+0000 7fe68d324640 1 --2- >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fe6881491b0 0x7fe68814f2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:09:04.266 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:04.262+0000 7fe68d324640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7fe6880b1660 con 0x7fe6880b3730 2024-03-28T14:09:04.283 INFO:teuthology.orchestra.run.smithi104.stdout:{ 2024-03-28T14:09:04.284 INFO:teuthology.orchestra.run.smithi104.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-03-28T14:09:04.284 INFO:teuthology.orchestra.run.smithi104.stdout:} 2024-03-28T14:09:04.307 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:04.262+0000 7fe68d324640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7fe6880b14e0 con 0x7fe68806c8f0 2024-03-28T14:09:04.307 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:04.262+0000 7fe68d324640 1 -- --> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] -- mon_getmap magic: 0 -- 0x7fe68806dc70 con 0x7fe6881491b0 2024-03-28T14:09:04.307 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:04.262+0000 7fe6867fc640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe6880b3730 0x7fe68814ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:09:04.307 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:04.262+0000 7fe6867fc640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe6880b3730 0x7fe68814ed90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:50016/0 (socket says 172.21.15.104:50016) 2024-03-28T14:09:04.308 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:04.262+0000 7fe6867fc640 1 -- 172.21.15.104:0/1086989157 learned_addr learned my addr 172.21.15.104:0/1086989157 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-03-28T14:09:04.308 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:04.262+0000 7fe686ffd640 1 --2- 172.21.15.104:0/1086989157 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fe68806c8f0 0x7fe6881dfbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:09:04.308 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:04.262+0000 7fe6877fe640 1 --2- 172.21.15.104:0/1086989157 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fe6881491b0 0x7fe68814f2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:09:04.308 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:04.262+0000 7fe6867fc640 1 -- 172.21.15.104:0/1086989157 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fe6881491b0 msgr2=0x7fe68814f2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:09:04.308 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:04.262+0000 7fe6867fc640 1 --2- 172.21.15.104:0/1086989157 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fe6881491b0 0x7fe68814f2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:09:04.309 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:04.262+0000 7fe6867fc640 1 -- 172.21.15.104:0/1086989157 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fe68806c8f0 msgr2=0x7fe6881dfbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:09:04.309 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:04.262+0000 7fe6867fc640 1 --2- 172.21.15.104:0/1086989157 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fe68806c8f0 0x7fe6881dfbc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:09:04.309 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:04.262+0000 7fe6867fc640 1 -- 172.21.15.104:0/1086989157 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe6880af0d0 con 0x7fe6880b3730 2024-03-28T14:09:04.309 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:04.262+0000 7fe6867fc640 1 --2- 172.21.15.104:0/1086989157 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe6880b3730 0x7fe68814ed90 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7fe67800d5f0 tx=0x7fe67800dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:09:04.309 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:04.262+0000 7fe67f7fe640 1 -- 172.21.15.104:0/1086989157 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6780125f0 con 0x7fe6880b3730 2024-03-28T14:09:04.309 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:04.262+0000 7fe67f7fe640 1 -- 172.21.15.104:0/1086989157 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe678019070 con 0x7fe6880b3730 2024-03-28T14:09:04.309 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:04.262+0000 7fe67f7fe640 1 -- 172.21.15.104:0/1086989157 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6780129b0 con 0x7fe6880b3730 2024-03-28T14:09:04.309 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:04.262+0000 7fe68d324640 1 -- 172.21.15.104:0/1086989157 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe6881e0500 con 0x7fe6880b3730 2024-03-28T14:09:04.310 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:04.262+0000 7fe68d324640 1 -- 172.21.15.104:0/1086989157 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe6880aed50 con 0x7fe6880b3730 2024-03-28T14:09:04.310 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:04.262+0000 7fe68d324640 1 -- 172.21.15.104:0/1086989157 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fe6880687c0 con 0x7fe6880b3730 2024-03-28T14:09:04.310 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:04.262+0000 7fe67f7fe640 1 -- 172.21.15.104:0/1086989157 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) ==== 87803+0+0 (secure 0 0 0) 0x7fe678012b50 con 0x7fe6880b3730 2024-03-28T14:09:04.310 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:04.262+0000 7fe67f7fe640 1 --2- 172.21.15.104:0/1086989157 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7fe668067fc0 0x7fe66806a480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:09:04.310 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:04.262+0000 7fe67f7fe640 1 -- 172.21.15.104:0/1086989157 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(33..33 src has 1..33) ==== 4943+0+0 (secure 0 0 0) 0x7fe6780870f0 con 0x7fe6880b3730 2024-03-28T14:09:04.310 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:04.262+0000 7fe67f7fe640 1 --2- 172.21.15.104:0/1086989157 >> [v2:172.21.15.171:6808/2934826925,v1:172.21.15.171:6810/2934826925] conn(0x7fe66806de80 0x7fe6680702a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:09:04.311 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:04.262+0000 7fe67f7fe640 1 -- 172.21.15.104:0/1086989157 --> [v2:172.21.15.171:6808/2934826925,v1:172.21.15.171:6810/2934826925] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fe678001d40 con 0x7fe66806de80 2024-03-28T14:09:04.311 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:04.262+0000 7fe67f7fe640 1 -- 172.21.15.104:0/1086989157 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7fe678087580 con 0x7fe6880b3730 2024-03-28T14:09:04.311 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:04.262+0000 7fe686ffd640 1 --2- 172.21.15.104:0/1086989157 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7fe668067fc0 0x7fe66806a480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:09:04.311 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:04.262+0000 7fe6877fe640 1 --2- 172.21.15.104:0/1086989157 >> [v2:172.21.15.171:6808/2934826925,v1:172.21.15.171:6810/2934826925] conn(0x7fe66806de80 0x7fe6680702a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:09:04.311 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:04.262+0000 7fe686ffd640 1 --2- 172.21.15.104:0/1086989157 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7fe668067fc0 0x7fe66806a480 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7fe6700af000 tx=0x7fe670008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:09:04.311 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:04.262+0000 7fe6877fe640 1 --2- 172.21.15.104:0/1086989157 >> [v2:172.21.15.171:6808/2934826925,v1:172.21.15.171:6810/2934826925] conn(0x7fe66806de80 0x7fe6680702a0 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:09:04.311 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:04.266+0000 7fe67f7fe640 1 -- 172.21.15.104:0/1086989157 <== osd.5 v2:172.21.15.171:6808/2934826925 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fe678001d40 con 0x7fe66806de80 2024-03-28T14:09:04.311 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:04.278+0000 7fe68d324640 1 -- 172.21.15.104:0/1086989157 --> [v2:172.21.15.171:6808/2934826925,v1:172.21.15.171:6810/2934826925] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7fe6881e0100 con 0x7fe66806de80 2024-03-28T14:09:04.311 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:04.278+0000 7fe67f7fe640 1 -- 172.21.15.104:0/1086989157 <== osd.5 v2:172.21.15.171:6808/2934826925 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7fe6881e0100 con 0x7fe66806de80 2024-03-28T14:09:04.312 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:04.278+0000 7fe68d324640 1 -- 172.21.15.104:0/1086989157 >> [v2:172.21.15.171:6808/2934826925,v1:172.21.15.171:6810/2934826925] conn(0x7fe66806de80 msgr2=0x7fe6680702a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:09:04.312 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:04.278+0000 7fe68d324640 1 --2- 172.21.15.104:0/1086989157 >> [v2:172.21.15.171:6808/2934826925,v1:172.21.15.171:6810/2934826925] conn(0x7fe66806de80 0x7fe6680702a0 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:09:04.312 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:04.278+0000 7fe68d324640 1 -- 172.21.15.104:0/1086989157 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7fe668067fc0 msgr2=0x7fe66806a480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:09:04.312 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:04.278+0000 7fe68d324640 1 --2- 172.21.15.104:0/1086989157 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7fe668067fc0 0x7fe66806a480 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7fe6700af000 tx=0x7fe670008040 comp rx=0 tx=0).stop 2024-03-28T14:09:04.312 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:04.282+0000 7fe68d324640 1 -- 172.21.15.104:0/1086989157 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe6880b3730 msgr2=0x7fe68814ed90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:09:04.312 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:04.282+0000 7fe68d324640 1 --2- 172.21.15.104:0/1086989157 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe6880b3730 0x7fe68814ed90 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7fe67800d5f0 tx=0x7fe67800dab0 comp rx=0 tx=0).stop 2024-03-28T14:09:04.312 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:04.282+0000 7fe6877fe640 1 -- 172.21.15.104:0/1086989157 reap_dead start 2024-03-28T14:09:04.312 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:04.282+0000 7fe68d324640 1 -- 172.21.15.104:0/1086989157 shutdown_connections 2024-03-28T14:09:04.313 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:04.282+0000 7fe68d324640 1 -- 172.21.15.104:0/1086989157 >> 172.21.15.104:0/1086989157 conn(0x7fe6880b8f00 msgr2=0x7fe688067330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T14:09:04.313 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:04.282+0000 7fe68d324640 1 -- 172.21.15.104:0/1086989157 shutdown_connections 2024-03-28T14:09:04.313 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:04.282+0000 7fe68d324640 1 -- 172.21.15.104:0/1086989157 wait complete. 2024-03-28T14:09:04.322 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 5 2024-03-28T14:09:04.441 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:04.438+0000 7f2184293640 1 Processor -- start 2024-03-28T14:09:04.441 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:04.438+0000 7f2184293640 1 -- start start 2024-03-28T14:09:04.443 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:04.438+0000 7f2184293640 1 --2- >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f217c06c8f0 0x7f217c0ae0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:09:04.444 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:04.438+0000 7f2184293640 1 --2- >> v2:172.21.15.171:3300/0 conn(0x7f217c14b340 0x7f217c14b7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:09:04.444 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:04.438+0000 7f2184293640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f217c06ccd0 con 0x7f217c0b3730 2024-03-28T14:09:04.444 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:04.438+0000 7f2184293640 1 -- --> v2:172.21.15.171:3300/0 -- mon_getmap magic: 0 -- 0x7f217c0b0230 con 0x7f217c14b340 2024-03-28T14:09:04.444 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:04.438+0000 7f2184293640 1 -- --> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] -- mon_getmap magic: 0 -- 0x7f217c0b2e80 con 0x7f217c06c8f0 2024-03-28T14:09:04.444 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:04.438+0000 7f2182830640 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f217c0b3730 0x7f217c0b3b40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:40630/0 (socket says 172.21.15.104:40630) 2024-03-28T14:09:04.444 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:04.438+0000 7f2182830640 1 -- 172.21.15.104:0/1263312878 learned_addr learned my addr 172.21.15.104:0/1263312878 (peer_addr_for_me v1:172.21.15.104:0/0) 2024-03-28T14:09:04.444 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:04.438+0000 7f218202f640 1 --2- 172.21.15.104:0/1263312878 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f217c06c8f0 0x7f217c0ae0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:09:04.445 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:04.438+0000 7f218182e640 1 --2- 172.21.15.104:0/1263312878 >> v2:172.21.15.171:3300/0 conn(0x7f217c14b340 0x7f217c14b7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:09:04.445 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:04.438+0000 7f218102d640 1 -- 172.21.15.104:0/1263312878 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4238464616 0 0) 0x7f217c06ccd0 con 0x7f217c0b3730 2024-03-28T14:09:04.445 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:04.438+0000 7f218102d640 1 -- 172.21.15.104:0/1263312878 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2168003530 con 0x7f217c0b3730 2024-03-28T14:09:04.445 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:04.438+0000 7f218102d640 1 -- 172.21.15.104:0/1263312878 <== mon.0 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1397936515 0 0) 0x7f2168003530 con 0x7f217c0b3730 2024-03-28T14:09:04.445 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:04.438+0000 7f218102d640 1 -- 172.21.15.104:0/1263312878 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f217c06c8f0 msgr2=0x7f217c0ae0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T14:09:04.445 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:04.438+0000 7f218102d640 1 --2- 172.21.15.104:0/1263312878 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f217c06c8f0 0x7f217c0ae0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:09:04.445 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:04.438+0000 7f218102d640 1 -- 172.21.15.104:0/1263312878 >> v2:172.21.15.171:3300/0 conn(0x7f217c14b340 msgr2=0x7f217c14b7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T14:09:04.445 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:04.438+0000 7f218102d640 1 --2- 172.21.15.104:0/1263312878 >> v2:172.21.15.171:3300/0 conn(0x7f217c14b340 0x7f217c14b7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:09:04.445 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:04.438+0000 7f218102d640 1 -- 172.21.15.104:0/1263312878 --> v1:172.21.15.104:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f217c0b5460 con 0x7f217c0b3730 2024-03-28T14:09:04.446 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:04.438+0000 7f218102d640 1 -- 172.21.15.104:0/1263312878 <== mon.0 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3745580849 0 0) 0x7f2174001a00 con 0x7f217c0b3730 2024-03-28T14:09:04.446 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:04.438+0000 7f218102d640 1 -- 172.21.15.104:0/1263312878 >> v1:172.21.15.104:6789/0 conn(0x7f217c0b3730 legacy=0x7f217c0b3b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:09:04.446 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:04.438+0000 7f218102d640 1 --2- 172.21.15.104:0/1263312878 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f21680039c0 0x7f2168003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:09:04.446 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:04.438+0000 7f218102d640 1 --2- 172.21.15.104:0/1263312878 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2168004810 0x7f2168004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:09:04.446 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:04.438+0000 7f218102d640 1 --2- 172.21.15.104:0/1263312878 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f217c06c8f0 0x7f21680051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:09:04.446 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:04.438+0000 7f218102d640 1 -- 172.21.15.104:0/1263312878 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f217c0b2e80 con 0x7f2168004810 2024-03-28T14:09:04.447 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:04.438+0000 7f218102d640 1 -- 172.21.15.104:0/1263312878 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f216800b9d0 con 0x7f21680039c0 2024-03-28T14:09:04.447 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:04.438+0000 7f218102d640 1 -- 172.21.15.104:0/1263312878 --> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] -- mon_getmap magic: 0 -- 0x7f216800bb50 con 0x7f217c06c8f0 2024-03-28T14:09:04.447 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:04.438+0000 7f218182e640 1 --2- 172.21.15.104:0/1263312878 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2168004810 0x7f2168004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:09:04.447 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:04.438+0000 7f218202f640 1 --2- 172.21.15.104:0/1263312878 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f21680039c0 0x7f2168003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:09:04.447 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:04.438+0000 7f2182830640 1 --2- 172.21.15.104:0/1263312878 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f217c06c8f0 0x7f21680051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:09:04.447 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:04.438+0000 7f218182e640 1 -- 172.21.15.104:0/1263312878 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f217c06c8f0 msgr2=0x7f21680051d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T14:09:04.447 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:04.438+0000 7f218182e640 1 --2- 172.21.15.104:0/1263312878 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f217c06c8f0 0x7f21680051d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:09:04.447 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:04.438+0000 7f218182e640 1 -- 172.21.15.104:0/1263312878 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f21680039c0 msgr2=0x7f2168003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T14:09:04.448 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:04.438+0000 7f218182e640 1 --2- 172.21.15.104:0/1263312878 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f21680039c0 0x7f2168003de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:09:04.448 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:04.438+0000 7f218182e640 1 -- 172.21.15.104:0/1263312878 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2168003530 con 0x7f2168004810 2024-03-28T14:09:04.451 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:04.442+0000 7f218182e640 1 --2- 172.21.15.104:0/1263312878 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2168004810 0x7f2168004c90 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7f2178009ae0 tx=0x7f217801cab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f2faad460324b39b server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:09:04.451 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:04.442+0000 7f218102d640 1 -- 172.21.15.104:0/1263312878 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f21780b2070 con 0x7f2168004810 2024-03-28T14:09:04.452 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:04.442+0000 7f218102d640 1 -- 172.21.15.104:0/1263312878 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f21780a55c0 con 0x7f2168004810 2024-03-28T14:09:04.452 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:04.442+0000 7f218102d640 1 -- 172.21.15.104:0/1263312878 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f21780096b0 con 0x7f2168004810 2024-03-28T14:09:04.452 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:04.442+0000 7f2184293640 1 -- 172.21.15.104:0/1263312878 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2168004810 msgr2=0x7f2168004c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:09:04.452 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:04.442+0000 7f2184293640 1 --2- 172.21.15.104:0/1263312878 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2168004810 0x7f2168004c90 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7f2178009ae0 tx=0x7f217801cab0 comp rx=0 tx=0).stop 2024-03-28T14:09:04.452 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:04.442+0000 7f2182830640 1 -- 172.21.15.104:0/1263312878 reap_dead start 2024-03-28T14:09:04.452 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:04.442+0000 7f2184293640 1 -- 172.21.15.104:0/1263312878 shutdown_connections 2024-03-28T14:09:04.452 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:04.442+0000 7f2184293640 1 -- 172.21.15.104:0/1263312878 >> 172.21.15.104:0/1263312878 conn(0x7f217c0b8f00 msgr2=0x7f217c0b9310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T14:09:04.452 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:04.442+0000 7f2184293640 1 -- 172.21.15.104:0/1263312878 shutdown_connections 2024-03-28T14:09:04.453 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:04.442+0000 7f2184293640 1 -- 172.21.15.104:0/1263312878 wait complete. 2024-03-28T14:09:04.453 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:04.442+0000 7f2184293640 1 Processor -- start 2024-03-28T14:09:04.453 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:04.442+0000 7f2184293640 1 -- start start 2024-03-28T14:09:04.453 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:04.446+0000 7f2184293640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f2168004810 0x7f217c0716d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:09:04.453 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:04.446+0000 7f2184293640 1 --2- >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f217c1e05b0 0x7f217c1466b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:09:04.453 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:04.446+0000 7f2184293640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f217c146bf0 0x7f217c1dcfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:09:04.453 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:04.446+0000 7f2184293640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f217c0afdd0 con 0x7f217c146bf0 2024-03-28T14:09:04.453 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:04.446+0000 7f2184293640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f217c0af940 con 0x7f2168004810 2024-03-28T14:09:04.454 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:04.446+0000 7f2184293640 1 -- --> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] -- mon_getmap magic: 0 -- 0x7f217c0aff50 con 0x7f217c1e05b0 2024-03-28T14:09:04.454 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:04.446+0000 7f2182830640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f217c146bf0 0x7f217c1dcfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:09:04.454 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:04.446+0000 7f2182830640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f217c146bf0 0x7f217c1dcfc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:50034/0 (socket says 172.21.15.104:50034) 2024-03-28T14:09:04.454 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:04.446+0000 7f2182830640 1 -- 172.21.15.104:0/3657409594 learned_addr learned my addr 172.21.15.104:0/3657409594 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-03-28T14:09:04.454 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:04.446+0000 7f218202f640 1 --2- 172.21.15.104:0/3657409594 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f2168004810 0x7f217c0716d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:09:04.454 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:04.446+0000 7f218182e640 1 --2- 172.21.15.104:0/3657409594 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f217c1e05b0 0x7f217c1466b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:09:04.454 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:04.446+0000 7f2182830640 1 -- 172.21.15.104:0/3657409594 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f217c1e05b0 msgr2=0x7f217c1466b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:09:04.454 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:04.446+0000 7f2182830640 1 --2- 172.21.15.104:0/3657409594 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f217c1e05b0 0x7f217c1466b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:09:04.454 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:04.446+0000 7f2182830640 1 -- 172.21.15.104:0/3657409594 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f2168004810 msgr2=0x7f217c0716d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:09:04.455 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:04.446+0000 7f2182830640 1 --2- 172.21.15.104:0/3657409594 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f2168004810 0x7f217c0716d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:09:04.455 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:04.446+0000 7f2182830640 1 -- 172.21.15.104:0/3657409594 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f217c1e0a40 con 0x7f217c146bf0 2024-03-28T14:09:04.455 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:04.446+0000 7f218202f640 1 --2- 172.21.15.104:0/3657409594 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f2168004810 0x7f217c0716d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T14:09:04.455 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:04.446+0000 7f2182830640 1 --2- 172.21.15.104:0/3657409594 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f217c146bf0 0x7f217c1dcfc0 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f217c06d8d0 tx=0x7f2174007bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:09:04.455 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:04.446+0000 7f2172ffd640 1 -- 172.21.15.104:0/3657409594 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f217401c070 con 0x7f217c146bf0 2024-03-28T14:09:04.455 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:04.446+0000 7f2172ffd640 1 -- 172.21.15.104:0/3657409594 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2174011840 con 0x7f217c146bf0 2024-03-28T14:09:04.455 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:04.446+0000 7f2184293640 1 -- 172.21.15.104:0/3657409594 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f217c1e0d20 con 0x7f217c146bf0 2024-03-28T14:09:04.455 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:04.446+0000 7f2172ffd640 1 -- 172.21.15.104:0/3657409594 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2174010040 con 0x7f217c146bf0 2024-03-28T14:09:04.456 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:04.446+0000 7f2184293640 1 -- 172.21.15.104:0/3657409594 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f217c072370 con 0x7f217c146bf0 2024-03-28T14:09:04.456 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:04.446+0000 7f2184293640 1 -- 172.21.15.104:0/3657409594 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f217c1d74c0 con 0x7f217c146bf0 2024-03-28T14:09:04.456 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:04.446+0000 7f2172ffd640 1 -- 172.21.15.104:0/3657409594 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) ==== 87803+0+0 (secure 0 0 0) 0x7f217401f070 con 0x7f217c146bf0 2024-03-28T14:09:04.456 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:04.446+0000 7f2172ffd640 1 --2- 172.21.15.104:0/3657409594 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f2160067fc0 0x7f216006a480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:09:04.456 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:04.446+0000 7f2172ffd640 1 -- 172.21.15.104:0/3657409594 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(33..33 src has 1..33) ==== 4943+0+0 (secure 0 0 0) 0x7f2174089030 con 0x7f217c146bf0 2024-03-28T14:09:04.456 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:04.446+0000 7f218202f640 1 --2- 172.21.15.104:0/3657409594 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f2160067fc0 0x7f216006a480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:09:04.457 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:04.450+0000 7f218202f640 1 --2- 172.21.15.104:0/3657409594 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f2160067fc0 0x7f216006a480 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f217c09bf30 tx=0x7f216c008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:09:04.457 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:04.450+0000 7f2172ffd640 1 -- 172.21.15.104:0/3657409594 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179117 (secure 0 0 0) 0x7f217c1d74c0 con 0x7f217c146bf0 2024-03-28T14:09:04.617 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:04.614+0000 7f2184293640 1 -- 172.21.15.104:0/3657409594 --> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "5", "target": ["mon-mgr", ""]}) -- 0x7f217c1e1520 con 0x7f2160067fc0 2024-03-28T14:09:04.618 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:04.614+0000 7f2172ffd640 1 -- 172.21.15.104:0/3657409594 <== mgr.4100 v2:172.21.15.171:6832/1522752412 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 5 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7f216c013930 con 0x7f2160067fc0 2024-03-28T14:09:04.618 INFO:teuthology.orchestra.run.smithi104.stderr:instructed osd(s) 5 to deep-scrub 2024-03-28T14:09:04.620 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:04.618+0000 7f2184293640 1 -- 172.21.15.104:0/3657409594 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f2160067fc0 msgr2=0x7f216006a480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:09:04.620 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:04.618+0000 7f2184293640 1 --2- 172.21.15.104:0/3657409594 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f2160067fc0 0x7f216006a480 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f217c09bf30 tx=0x7f216c008040 comp rx=0 tx=0).stop 2024-03-28T14:09:04.621 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:04.618+0000 7f2184293640 1 -- 172.21.15.104:0/3657409594 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f217c146bf0 msgr2=0x7f217c1dcfc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:09:04.621 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:04.618+0000 7f2184293640 1 --2- 172.21.15.104:0/3657409594 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f217c146bf0 0x7f217c1dcfc0 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f217c06d8d0 tx=0x7f2174007bb0 comp rx=0 tx=0).stop 2024-03-28T14:09:04.621 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:04.618+0000 7f2184293640 1 -- 172.21.15.104:0/3657409594 shutdown_connections 2024-03-28T14:09:04.621 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:04.618+0000 7f2184293640 1 --2- 172.21.15.104:0/3657409594 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f2160067fc0 0x7f216006a480 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:09:04.621 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:04.618+0000 7f2184293640 1 --2- 172.21.15.104:0/3657409594 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f217c146bf0 0x7f217c1dcfc0 unknown :-1 s=CLOSED pgs=474 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:09:04.621 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:04.618+0000 7f2184293640 1 --2- 172.21.15.104:0/3657409594 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f217c1e05b0 0x7f217c1466b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:09:04.621 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:04.618+0000 7f2184293640 1 --2- 172.21.15.104:0/3657409594 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f2168004810 0x7f217c0716d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:09:04.621 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:04.618+0000 7f2184293640 1 -- 172.21.15.104:0/3657409594 >> 172.21.15.104:0/3657409594 conn(0x7f217c0b8f00 msgr2=0x7f217c0b9310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T14:09:04.622 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:04.618+0000 7f2184293640 1 -- 172.21.15.104:0/3657409594 shutdown_connections 2024-03-28T14:09:04.622 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:04.618+0000 7f2184293640 1 -- 172.21.15.104:0/3657409594 wait complete. 2024-03-28T14:09:04.637 INFO:tasks.ceph:Scrubbing osd.6 2024-03-28T14:09:04.637 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 config set osd_debug_deep_scrub_sleep 0 2024-03-28T14:09:04.767 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:04.762+0000 7f87c388a640 1 Processor -- start 2024-03-28T14:09:04.768 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:04.762+0000 7f87c388a640 1 -- start start 2024-03-28T14:09:04.770 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:04.766+0000 7f87c388a640 1 --2- >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f87bc0b3730 0x7f87bc1511c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:09:04.770 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:04.766+0000 7f87c388a640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f87bc149a90 con 0x7f87bc0b2740 2024-03-28T14:09:04.771 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:04.766+0000 7f87c388a640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f87bc06ccd0 con 0x7f87bc06c8f0 2024-03-28T14:09:04.771 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:04.766+0000 7f87c388a640 1 -- --> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] -- mon_getmap magic: 0 -- 0x7f87bc0b2b70 con 0x7f87bc0b3730 2024-03-28T14:09:04.771 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:04.766+0000 7f87c0e25640 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f87bc0b2740 0x7f87bc149380 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:40638/0 (socket says 172.21.15.104:40638) 2024-03-28T14:09:04.771 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:04.766+0000 7f87c0e25640 1 -- 172.21.15.104:0/1068644068 learned_addr learned my addr 172.21.15.104:0/1068644068 (peer_addr_for_me v1:172.21.15.104:0/0) 2024-03-28T14:09:04.771 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:04.766+0000 7f87affff640 1 -- 172.21.15.104:0/1068644068 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1178444542 0 0) 0x7f87bc149a90 con 0x7f87bc0b2740 2024-03-28T14:09:04.771 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:04.766+0000 7f87c1e27640 1 --2- 172.21.15.104:0/1068644068 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f87bc0b3730 0x7f87bc1511c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:09:04.771 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:04.766+0000 7f87affff640 1 -- 172.21.15.104:0/1068644068 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f87a8003530 con 0x7f87bc0b2740 2024-03-28T14:09:04.771 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:04.766+0000 7f87affff640 1 -- 172.21.15.104:0/1068644068 <== mon.0 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 898328216 0 0) 0x7f87a8003530 con 0x7f87bc0b2740 2024-03-28T14:09:04.772 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:04.766+0000 7f87affff640 1 -- 172.21.15.104:0/1068644068 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f87bc0b3730 msgr2=0x7f87bc1511c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T14:09:04.772 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:04.766+0000 7f87affff640 1 --2- 172.21.15.104:0/1068644068 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f87bc0b3730 0x7f87bc1511c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:09:04.772 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:04.766+0000 7f87affff640 1 -- 172.21.15.104:0/1068644068 >> v1:172.21.15.171:6789/0 conn(0x7f87bc06c8f0 legacy=0x7f87bc148c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:09:04.772 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:04.766+0000 7f87affff640 1 -- 172.21.15.104:0/1068644068 --> v1:172.21.15.104:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f87bc0b1980 con 0x7f87bc0b2740 2024-03-28T14:09:04.772 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:04.766+0000 7f87affff640 1 -- 172.21.15.104:0/1068644068 <== mon.0 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3745580849 0 0) 0x7f87b8001680 con 0x7f87bc0b2740 2024-03-28T14:09:04.772 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:04.766+0000 7f87affff640 1 -- 172.21.15.104:0/1068644068 >> v1:172.21.15.104:6789/0 conn(0x7f87bc0b2740 legacy=0x7f87bc149380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:09:04.772 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:04.766+0000 7f87affff640 1 --2- 172.21.15.104:0/1068644068 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f87a80039f0 0x7f87a8003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:09:04.773 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:04.766+0000 7f87affff640 1 --2- 172.21.15.104:0/1068644068 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f87bc0b3730 0x7f87a8004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:09:04.773 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:04.766+0000 7f87affff640 1 --2- 172.21.15.104:0/1068644068 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f87a8004d50 0x7f87a80051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:09:04.773 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:04.766+0000 7f87affff640 1 -- 172.21.15.104:0/1068644068 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f87bc0b2b70 con 0x7f87a8004d50 2024-03-28T14:09:04.773 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:04.766+0000 7f87affff640 1 -- 172.21.15.104:0/1068644068 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f87a800bb00 con 0x7f87a80039f0 2024-03-28T14:09:04.773 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:04.766+0000 7f87affff640 1 -- 172.21.15.104:0/1068644068 --> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] -- mon_getmap magic: 0 -- 0x7f87a800bd00 con 0x7f87bc0b3730 2024-03-28T14:09:04.773 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:04.766+0000 7f87c1e27640 1 --2- 172.21.15.104:0/1068644068 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f87a8004d50 0x7f87a80051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:09:04.773 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:04.766+0000 7f87c1626640 1 --2- 172.21.15.104:0/1068644068 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f87a80039f0 0x7f87a8003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:09:04.773 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:04.766+0000 7f87c0e25640 1 --2- 172.21.15.104:0/1068644068 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f87bc0b3730 0x7f87a8004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:09:04.773 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:04.766+0000 7f87c1e27640 1 -- 172.21.15.104:0/1068644068 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f87bc0b3730 msgr2=0x7f87a8004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T14:09:04.774 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:04.766+0000 7f87c1e27640 1 --2- 172.21.15.104:0/1068644068 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f87bc0b3730 0x7f87a8004810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:09:04.774 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:04.766+0000 7f87c1e27640 1 -- 172.21.15.104:0/1068644068 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f87a80039f0 msgr2=0x7f87a8003e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T14:09:04.774 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:04.766+0000 7f87c1e27640 1 --2- 172.21.15.104:0/1068644068 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f87a80039f0 0x7f87a8003e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:09:04.774 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:04.766+0000 7f87c1e27640 1 -- 172.21.15.104:0/1068644068 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f87a8003530 con 0x7f87a8004d50 2024-03-28T14:09:04.774 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:04.766+0000 7f87c1626640 1 --2- 172.21.15.104:0/1068644068 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f87a80039f0 0x7f87a8003e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T14:09:04.774 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:04.766+0000 7f87c0e25640 1 --2- 172.21.15.104:0/1068644068 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f87bc0b3730 0x7f87a8004810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T14:09:04.778 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:04.766+0000 7f87c1e27640 1 --2- 172.21.15.104:0/1068644068 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f87a8004d50 0x7f87a80051d0 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7f87b4009fa0 tx=0x7f87b401cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4a9daff4c2a04d4f server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:09:04.778 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:04.766+0000 7f87affff640 1 -- 172.21.15.104:0/1068644068 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87b40b2070 con 0x7f87a8004d50 2024-03-28T14:09:04.778 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:04.766+0000 7f87affff640 1 -- 172.21.15.104:0/1068644068 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f87b40a5d10 con 0x7f87a8004d50 2024-03-28T14:09:04.778 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:04.766+0000 7f87affff640 1 -- 172.21.15.104:0/1068644068 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87b4009bf0 con 0x7f87a8004d50 2024-03-28T14:09:04.779 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:04.770+0000 7f87c388a640 1 -- 172.21.15.104:0/1068644068 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f87a8004d50 msgr2=0x7f87a80051d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:09:04.779 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:04.770+0000 7f87c388a640 1 --2- 172.21.15.104:0/1068644068 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f87a8004d50 0x7f87a80051d0 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7f87b4009fa0 tx=0x7f87b401cc40 comp rx=0 tx=0).stop 2024-03-28T14:09:04.779 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:04.770+0000 7f87c1e27640 1 -- 172.21.15.104:0/1068644068 reap_dead start 2024-03-28T14:09:04.779 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:04.770+0000 7f87c388a640 1 -- 172.21.15.104:0/1068644068 shutdown_connections 2024-03-28T14:09:04.779 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:04.770+0000 7f87c388a640 1 -- 172.21.15.104:0/1068644068 >> 172.21.15.104:0/1068644068 conn(0x7f87bc0b8f00 msgr2=0x7f87bc0b9310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T14:09:04.779 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:04.770+0000 7f87c388a640 1 -- 172.21.15.104:0/1068644068 shutdown_connections 2024-03-28T14:09:04.779 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:04.770+0000 7f87c388a640 1 -- 172.21.15.104:0/1068644068 wait complete. 2024-03-28T14:09:04.779 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:04.770+0000 7f87c388a640 1 Processor -- start 2024-03-28T14:09:04.780 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:04.770+0000 7f87c388a640 1 -- start start 2024-03-28T14:09:04.780 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:04.770+0000 7f87c388a640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f87a8004d50 0x7f87bc142020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:09:04.780 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:04.770+0000 7f87c388a640 1 --2- >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f87bc1416b0 0x7f87bc1de4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:09:04.780 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:04.770+0000 7f87c388a640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f87bc1de9f0 0x7f87bc09cad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:09:04.780 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:04.770+0000 7f87c388a640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f87bc0b0b30 con 0x7f87bc1de9f0 2024-03-28T14:09:04.780 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:04.770+0000 7f87c388a640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f87bc0b3ba0 con 0x7f87a8004d50 2024-03-28T14:09:04.780 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:04.770+0000 7f87c388a640 1 -- --> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] -- mon_getmap magic: 0 -- 0x7f87bc0b22b0 con 0x7f87bc1416b0 2024-03-28T14:09:04.780 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:04.770+0000 7f87c1e27640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f87bc1de9f0 0x7f87bc09cad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:09:04.780 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:04.770+0000 7f87c1e27640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f87bc1de9f0 0x7f87bc09cad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:50048/0 (socket says 172.21.15.104:50048) 2024-03-28T14:09:04.781 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:04.770+0000 7f87c1e27640 1 -- 172.21.15.104:0/934665941 learned_addr learned my addr 172.21.15.104:0/934665941 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-03-28T14:09:04.781 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:04.770+0000 7f87c0e25640 1 --2- 172.21.15.104:0/934665941 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f87bc1416b0 0x7f87bc1de4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:09:04.781 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:04.770+0000 7f87c1626640 1 --2- 172.21.15.104:0/934665941 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f87a8004d50 0x7f87bc142020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:09:04.781 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:04.770+0000 7f87c1e27640 1 -- 172.21.15.104:0/934665941 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f87bc1416b0 msgr2=0x7f87bc1de4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:09:04.781 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:04.770+0000 7f87c1e27640 1 --2- 172.21.15.104:0/934665941 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f87bc1416b0 0x7f87bc1de4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:09:04.781 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:04.770+0000 7f87c1e27640 1 -- 172.21.15.104:0/934665941 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f87a8004d50 msgr2=0x7f87bc142020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:09:04.781 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:04.770+0000 7f87c1e27640 1 --2- 172.21.15.104:0/934665941 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f87a8004d50 0x7f87bc142020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:09:04.781 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:04.770+0000 7f87c1e27640 1 -- 172.21.15.104:0/934665941 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f87bc142850 con 0x7f87bc1de9f0 2024-03-28T14:09:04.782 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:04.770+0000 7f87c1e27640 1 --2- 172.21.15.104:0/934665941 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f87bc1de9f0 0x7f87bc09cad0 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7f87b40b0040 tx=0x7f87b4009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:09:04.782 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:04.770+0000 7f87adffb640 1 -- 172.21.15.104:0/934665941 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87b40b2040 con 0x7f87bc1de9f0 2024-03-28T14:09:04.782 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:04.770+0000 7f87adffb640 1 -- 172.21.15.104:0/934665941 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f87b40a5b10 con 0x7f87bc1de9f0 2024-03-28T14:09:04.782 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:04.770+0000 7f87c388a640 1 -- 172.21.15.104:0/934665941 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f87bc142b30 con 0x7f87bc1de9f0 2024-03-28T14:09:04.782 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:04.770+0000 7f87adffb640 1 -- 172.21.15.104:0/934665941 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87b8001d50 con 0x7f87bc1de9f0 2024-03-28T14:09:04.782 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:04.770+0000 7f87c388a640 1 -- 172.21.15.104:0/934665941 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f87bc143e00 con 0x7f87bc1de9f0 2024-03-28T14:09:04.782 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:04.770+0000 7f87c388a640 1 -- 172.21.15.104:0/934665941 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f87bc0687c0 con 0x7f87bc1de9f0 2024-03-28T14:09:04.782 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:04.774+0000 7f87adffb640 1 -- 172.21.15.104:0/934665941 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) ==== 87803+0+0 (secure 0 0 0) 0x7f87b40c0030 con 0x7f87bc1de9f0 2024-03-28T14:09:04.783 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:04.774+0000 7f87adffb640 1 --2- 172.21.15.104:0/934665941 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f87a0067ff0 0x7f87a006a4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:09:04.783 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:04.774+0000 7f87adffb640 1 -- 172.21.15.104:0/934665941 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(33..33 src has 1..33) ==== 4943+0+0 (secure 0 0 0) 0x7f87b4121220 con 0x7f87bc1de9f0 2024-03-28T14:09:04.783 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:04.774+0000 7f87adffb640 1 --2- 172.21.15.104:0/934665941 >> [v2:172.21.15.171:6824/436742919,v1:172.21.15.171:6825/436742919] conn(0x7f87a006deb0 0x7f87a00702d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:09:04.783 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:04.774+0000 7f87adffb640 1 -- 172.21.15.104:0/934665941 --> [v2:172.21.15.171:6824/436742919,v1:172.21.15.171:6825/436742919] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f87a0070980 con 0x7f87a006deb0 2024-03-28T14:09:04.783 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:04.774+0000 7f87adffb640 1 -- 172.21.15.104:0/934665941 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7f87b41216b0 con 0x7f87bc1de9f0 2024-03-28T14:09:04.783 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:04.774+0000 7f87c1626640 1 --2- 172.21.15.104:0/934665941 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f87a0067ff0 0x7f87a006a4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:09:04.783 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:04.774+0000 7f87c0e25640 1 --2- 172.21.15.104:0/934665941 >> [v2:172.21.15.171:6824/436742919,v1:172.21.15.171:6825/436742919] conn(0x7f87a006deb0 0x7f87a00702d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:09:04.783 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:04.774+0000 7f87c1626640 1 --2- 172.21.15.104:0/934665941 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f87a0067ff0 0x7f87a006a4b0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f87b0001760 tx=0x7f87b0009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:09:04.784 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:04.774+0000 7f87c0e25640 1 --2- 172.21.15.104:0/934665941 >> [v2:172.21.15.171:6824/436742919,v1:172.21.15.171:6825/436742919] conn(0x7f87a006deb0 0x7f87a00702d0 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:09:04.784 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:04.774+0000 7f87adffb640 1 -- 172.21.15.104:0/934665941 <== osd.6 v2:172.21.15.171:6824/436742919 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f87a0070980 con 0x7f87a006deb0 2024-03-28T14:09:04.793 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:04.790+0000 7f87c388a640 1 -- 172.21.15.104:0/934665941 --> [v2:172.21.15.171:6824/436742919,v1:172.21.15.171:6825/436742919] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7f87bc072910 con 0x7f87a006deb0 2024-03-28T14:09:04.793 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:04.790+0000 7f87adffb640 1 -- 172.21.15.104:0/934665941 <== osd.6 v2:172.21.15.171:6824/436742919 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7f87bc072910 con 0x7f87a006deb0 2024-03-28T14:09:04.794 INFO:teuthology.orchestra.run.smithi104.stdout:{ 2024-03-28T14:09:04.795 INFO:teuthology.orchestra.run.smithi104.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-03-28T14:09:04.795 INFO:teuthology.orchestra.run.smithi104.stdout:} 2024-03-28T14:09:04.796 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:04.790+0000 7f87c388a640 1 -- 172.21.15.104:0/934665941 >> [v2:172.21.15.171:6824/436742919,v1:172.21.15.171:6825/436742919] conn(0x7f87a006deb0 msgr2=0x7f87a00702d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:09:04.796 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:04.790+0000 7f87c388a640 1 --2- 172.21.15.104:0/934665941 >> [v2:172.21.15.171:6824/436742919,v1:172.21.15.171:6825/436742919] conn(0x7f87a006deb0 0x7f87a00702d0 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:09:04.796 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:04.790+0000 7f87c388a640 1 -- 172.21.15.104:0/934665941 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f87a0067ff0 msgr2=0x7f87a006a4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:09:04.796 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:04.790+0000 7f87c388a640 1 --2- 172.21.15.104:0/934665941 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f87a0067ff0 0x7f87a006a4b0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f87b0001760 tx=0x7f87b0009040 comp rx=0 tx=0).stop 2024-03-28T14:09:04.796 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:04.790+0000 7f87c388a640 1 -- 172.21.15.104:0/934665941 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f87bc1de9f0 msgr2=0x7f87bc09cad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:09:04.796 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:04.790+0000 7f87c388a640 1 --2- 172.21.15.104:0/934665941 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f87bc1de9f0 0x7f87bc09cad0 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7f87b40b0040 tx=0x7f87b4009750 comp rx=0 tx=0).stop 2024-03-28T14:09:04.796 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:04.790+0000 7f87c1e27640 1 -- 172.21.15.104:0/934665941 reap_dead start 2024-03-28T14:09:04.797 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:04.790+0000 7f87c388a640 1 -- 172.21.15.104:0/934665941 shutdown_connections 2024-03-28T14:09:04.797 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:04.790+0000 7f87c388a640 1 -- 172.21.15.104:0/934665941 >> 172.21.15.104:0/934665941 conn(0x7f87bc0b8f00 msgr2=0x7f87bc0b4bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T14:09:04.797 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:04.790+0000 7f87c388a640 1 -- 172.21.15.104:0/934665941 shutdown_connections 2024-03-28T14:09:04.797 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:04.790+0000 7f87c388a640 1 -- 172.21.15.104:0/934665941 wait complete. 2024-03-28T14:09:04.810 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 6 2024-03-28T14:09:04.917 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:04.914+0000 7f879d20f640 1 Processor -- start 2024-03-28T14:09:04.917 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:04.914+0000 7f879d20f640 1 -- start start 2024-03-28T14:09:04.919 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:04.914+0000 7f879d20f640 1 --2- >> v2:172.21.15.171:3300/0 conn(0x7f879806b830 0x7f879809aa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:09:04.919 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:04.914+0000 7f879d20f640 1 --2- >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f879813f4c0 0x7f879813f8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:09:04.919 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:04.914+0000 7f879d20f640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f879809c450 con 0x7f87980b4fa0 2024-03-28T14:09:04.919 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:04.914+0000 7f879d20f640 1 -- --> v2:172.21.15.171:3300/0 -- mon_getmap magic: 0 -- 0x7f879806bc10 con 0x7f879806b830 2024-03-28T14:09:04.919 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:04.914+0000 7f879d20f640 1 -- --> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] -- mon_getmap magic: 0 -- 0x7f8798072ea0 con 0x7f879813f4c0 2024-03-28T14:09:04.919 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:04.914+0000 7f8796d9d640 1 --2- >> v2:172.21.15.171:3300/0 conn(0x7f879806b830 0x7f879809aa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:09:04.920 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:04.914+0000 7f879759e640 1 --2- >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f879813f4c0 0x7f879813f8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:09:04.920 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:04.914+0000 7f879659c640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7f87980b4fa0 0x7f879809c930 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.104:37332/0 (socket says 172.21.15.104:37332) 2024-03-28T14:09:04.920 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:04.914+0000 7f879659c640 1 -- 172.21.15.104:0/3852366098 learned_addr learned my addr 172.21.15.104:0/3852366098 (peer_addr_for_me v1:172.21.15.104:0/0) 2024-03-28T14:09:04.920 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:04.914+0000 7f879759e640 1 -- 172.21.15.104:0/3852366098 >> v2:172.21.15.171:3300/0 conn(0x7f879806b830 msgr2=0x7f879809aa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T14:09:04.920 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:04.914+0000 7f879759e640 1 --2- 172.21.15.104:0/3852366098 >> v2:172.21.15.171:3300/0 conn(0x7f879806b830 0x7f879809aa10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:09:04.920 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:04.914+0000 7f879759e640 1 -- 172.21.15.104:0/3852366098 >> v1:172.21.15.171:6789/0 conn(0x7f87980b4fa0 legacy=0x7f879809c930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:09:04.920 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:04.914+0000 7f879759e640 1 -- 172.21.15.104:0/3852366098 --> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8798073180 con 0x7f879813f4c0 2024-03-28T14:09:04.920 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:04.914+0000 7f8796d9d640 1 --2- 172.21.15.104:0/3852366098 >> v2:172.21.15.171:3300/0 conn(0x7f879806b830 0x7f879809aa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T14:09:04.921 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:04.918+0000 7f879759e640 1 --2- 172.21.15.104:0/3852366098 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f879813f4c0 0x7f879813f8f0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f87880097c0 tx=0x7f878801c410 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9148e82a4de4430 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:09:04.921 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:04.918+0000 7f8795d9b640 1 -- 172.21.15.104:0/3852366098 <== mon.2 v2:172.21.15.171:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87880b0070 con 0x7f879813f4c0 2024-03-28T14:09:04.921 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:04.918+0000 7f8795d9b640 1 -- 172.21.15.104:0/3852366098 <== mon.2 v2:172.21.15.171:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f87880ab600 con 0x7f879813f4c0 2024-03-28T14:09:04.921 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:04.918+0000 7f8795d9b640 1 -- 172.21.15.104:0/3852366098 <== mon.2 v2:172.21.15.171:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87880ab9d0 con 0x7f879813f4c0 2024-03-28T14:09:04.922 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:04.918+0000 7f879d20f640 1 -- 172.21.15.104:0/3852366098 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f879813f4c0 msgr2=0x7f879813f8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:09:04.922 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:04.918+0000 7f879d20f640 1 --2- 172.21.15.104:0/3852366098 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f879813f4c0 0x7f879813f8f0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f87880097c0 tx=0x7f878801c410 comp rx=0 tx=0).stop 2024-03-28T14:09:04.923 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:04.918+0000 7f879d20f640 1 -- 172.21.15.104:0/3852366098 shutdown_connections 2024-03-28T14:09:04.923 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:04.918+0000 7f879d20f640 1 --2- 172.21.15.104:0/3852366098 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f879813f4c0 0x7f879813f8f0 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:09:04.923 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:04.918+0000 7f879d20f640 1 --2- 172.21.15.104:0/3852366098 >> v2:172.21.15.171:3300/0 conn(0x7f879806b830 0x7f879809aa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:09:04.923 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:04.918+0000 7f879d20f640 1 -- 172.21.15.104:0/3852366098 >> 172.21.15.104:0/3852366098 conn(0x7f87980ba390 msgr2=0x7f87980ba7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T14:09:04.923 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:04.918+0000 7f879d20f640 1 -- 172.21.15.104:0/3852366098 shutdown_connections 2024-03-28T14:09:04.923 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:04.918+0000 7f879d20f640 1 -- 172.21.15.104:0/3852366098 wait complete. 2024-03-28T14:09:04.925 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:04.918+0000 7f879d20f640 1 Processor -- start 2024-03-28T14:09:04.925 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:04.918+0000 7f879d20f640 1 -- start start 2024-03-28T14:09:04.926 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:04.922+0000 7f879d20f640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f879806b830 0x7f8798072090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:09:04.926 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:04.922+0000 7f879d20f640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f87980b4fa0 0x7f87980b12f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:09:04.926 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:04.922+0000 7f879d20f640 1 --2- >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f879813f4c0 0x7f879813a0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:09:04.926 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:04.922+0000 7f879d20f640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f8798152a80 con 0x7f87980b4fa0 2024-03-28T14:09:04.926 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:04.922+0000 7f879d20f640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f8798152900 con 0x7f879806b830 2024-03-28T14:09:04.926 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:04.922+0000 7f879d20f640 1 -- --> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] -- mon_getmap magic: 0 -- 0x7f8798152c00 con 0x7f879813f4c0 2024-03-28T14:09:04.926 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:04.922+0000 7f879659c640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f87980b4fa0 0x7f87980b12f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:09:04.926 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:04.922+0000 7f879659c640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f87980b4fa0 0x7f87980b12f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:50050/0 (socket says 172.21.15.104:50050) 2024-03-28T14:09:04.927 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:04.922+0000 7f879659c640 1 -- 172.21.15.104:0/3229854535 learned_addr learned my addr 172.21.15.104:0/3229854535 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-03-28T14:09:04.927 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:04.922+0000 7f8796d9d640 1 --2- 172.21.15.104:0/3229854535 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f879806b830 0x7f8798072090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:09:04.927 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:04.922+0000 7f879759e640 1 --2- 172.21.15.104:0/3229854535 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f879813f4c0 0x7f879813a0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:09:04.927 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:04.922+0000 7f879659c640 1 -- 172.21.15.104:0/3229854535 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f879813f4c0 msgr2=0x7f879813a0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:09:04.927 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:04.922+0000 7f879659c640 1 --2- 172.21.15.104:0/3229854535 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f879813f4c0 0x7f879813a0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:09:04.927 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:04.922+0000 7f879659c640 1 -- 172.21.15.104:0/3229854535 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f879806b830 msgr2=0x7f8798072090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:09:04.927 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:04.922+0000 7f879659c640 1 --2- 172.21.15.104:0/3229854535 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f879806b830 0x7f8798072090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:09:04.927 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:04.922+0000 7f879659c640 1 -- 172.21.15.104:0/3229854535 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f879809c9c0 con 0x7f87980b4fa0 2024-03-28T14:09:04.927 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:04.922+0000 7f879659c640 1 --2- 172.21.15.104:0/3229854535 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f87980b4fa0 0x7f87980b12f0 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7f878c00c420 tx=0x7f878c00c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:09:04.928 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:04.922+0000 7f877f7fe640 1 -- 172.21.15.104:0/3229854535 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f878c017070 con 0x7f87980b4fa0 2024-03-28T14:09:04.928 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:04.922+0000 7f877f7fe640 1 -- 172.21.15.104:0/3229854535 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f878c005cf0 con 0x7f87980b4fa0 2024-03-28T14:09:04.928 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:04.922+0000 7f877f7fe640 1 -- 172.21.15.104:0/3229854535 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f878c0072f0 con 0x7f87980b4fa0 2024-03-28T14:09:04.928 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:04.922+0000 7f879d20f640 1 -- 172.21.15.104:0/3229854535 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f879809dd00 con 0x7f87980b4fa0 2024-03-28T14:09:04.928 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:04.922+0000 7f879d20f640 1 -- 172.21.15.104:0/3229854535 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f87980b2ca0 con 0x7f87980b4fa0 2024-03-28T14:09:04.929 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:04.922+0000 7f879d20f640 1 -- 172.21.15.104:0/3229854535 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f87980687c0 con 0x7f87980b4fa0 2024-03-28T14:09:04.929 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:04.922+0000 7f877f7fe640 1 -- 172.21.15.104:0/3229854535 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) ==== 87803+0+0 (secure 0 0 0) 0x7f878c01a070 con 0x7f87980b4fa0 2024-03-28T14:09:04.929 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:04.922+0000 7f877f7fe640 1 --2- 172.21.15.104:0/3229854535 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f8784067fc0 0x7f878406a480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:09:04.929 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:04.922+0000 7f877f7fe640 1 -- 172.21.15.104:0/3229854535 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(33..33 src has 1..33) ==== 4943+0+0 (secure 0 0 0) 0x7f878c085580 con 0x7f87980b4fa0 2024-03-28T14:09:04.930 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:04.922+0000 7f8796d9d640 1 --2- 172.21.15.104:0/3229854535 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f8784067fc0 0x7f878406a480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:09:04.930 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:04.926+0000 7f8796d9d640 1 --2- 172.21.15.104:0/3229854535 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f8784067fc0 0x7f878406a480 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f8790009bd0 tx=0x7f879000b040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:09:04.931 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:04.926+0000 7f877f7fe640 1 -- 172.21.15.104:0/3229854535 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179117 (secure 0 0 0) 0x7f878c0560c0 con 0x7f87980b4fa0 2024-03-28T14:09:05.082 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:05.078+0000 7f879d20f640 1 -- 172.21.15.104:0/3229854535 --> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "6", "target": ["mon-mgr", ""]}) -- 0x7f8798144e10 con 0x7f8784067fc0 2024-03-28T14:09:05.082 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:05.078+0000 7f877f7fe640 1 -- 172.21.15.104:0/3229854535 <== mgr.4100 v2:172.21.15.171:6832/1522752412 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 6 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7f8798144e10 con 0x7f8784067fc0 2024-03-28T14:09:05.083 INFO:teuthology.orchestra.run.smithi104.stderr:instructed osd(s) 6 to deep-scrub 2024-03-28T14:09:05.085 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:05.082+0000 7f879d20f640 1 -- 172.21.15.104:0/3229854535 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f8784067fc0 msgr2=0x7f878406a480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:09:05.085 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:05.082+0000 7f879d20f640 1 --2- 172.21.15.104:0/3229854535 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f8784067fc0 0x7f878406a480 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f8790009bd0 tx=0x7f879000b040 comp rx=0 tx=0).stop 2024-03-28T14:09:05.085 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:05.082+0000 7f879d20f640 1 -- 172.21.15.104:0/3229854535 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f87980b4fa0 msgr2=0x7f87980b12f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:09:05.085 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:05.082+0000 7f879d20f640 1 --2- 172.21.15.104:0/3229854535 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f87980b4fa0 0x7f87980b12f0 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7f878c00c420 tx=0x7f878c00c8e0 comp rx=0 tx=0).stop 2024-03-28T14:09:05.085 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:05.082+0000 7f879d20f640 1 -- 172.21.15.104:0/3229854535 shutdown_connections 2024-03-28T14:09:05.085 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:05.082+0000 7f879d20f640 1 --2- 172.21.15.104:0/3229854535 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f879813f4c0 0x7f879813a0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:09:05.086 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:05.082+0000 7f879d20f640 1 --2- 172.21.15.104:0/3229854535 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f8784067fc0 0x7f878406a480 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:09:05.086 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:05.082+0000 7f879d20f640 1 --2- 172.21.15.104:0/3229854535 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f87980b4fa0 0x7f87980b12f0 unknown :-1 s=CLOSED pgs=478 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:09:05.086 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:05.082+0000 7f879d20f640 1 --2- 172.21.15.104:0/3229854535 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f879806b830 0x7f8798072090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:09:05.086 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:05.082+0000 7f879d20f640 1 -- 172.21.15.104:0/3229854535 >> 172.21.15.104:0/3229854535 conn(0x7f87980ba390 msgr2=0x7f87980b6a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T14:09:05.086 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:05.082+0000 7f879d20f640 1 -- 172.21.15.104:0/3229854535 shutdown_connections 2024-03-28T14:09:05.086 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:05.082+0000 7f879d20f640 1 -- 172.21.15.104:0/3229854535 wait complete. 2024-03-28T14:09:05.100 INFO:tasks.ceph:Scrubbing osd.7 2024-03-28T14:09:05.100 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 config set osd_debug_deep_scrub_sleep 0 2024-03-28T14:09:05.226 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:05.222+0000 7f3db4b31640 1 Processor -- start 2024-03-28T14:09:05.226 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:05.222+0000 7f3db4b31640 1 -- start start 2024-03-28T14:09:05.227 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:05.222+0000 7f3db4b31640 1 --2- >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f3db006e570 0x7f3db006f0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:09:05.227 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:05.222+0000 7f3db4b31640 1 --2- >> v2:172.21.15.171:3300/0 conn(0x7f3db0147cd0 0x7f3db0148160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:09:05.228 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:05.222+0000 7f3db4b31640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3db00b9c20 con 0x7f3db006e950 2024-03-28T14:09:05.228 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:05.222+0000 7f3db4b31640 1 -- --> v2:172.21.15.171:3300/0 -- mon_getmap magic: 0 -- 0x7f3db006bce0 con 0x7f3db0147cd0 2024-03-28T14:09:05.228 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:05.222+0000 7f3db4b31640 1 -- --> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] -- mon_getmap magic: 0 -- 0x7f3db0073410 con 0x7f3db006e570 2024-03-28T14:09:05.228 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:05.222+0000 7f3dae59c640 1 --2- >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f3db006e570 0x7f3db006f0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:09:05.228 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:05.222+0000 7f3daed9d640 1 --2- >> v2:172.21.15.171:3300/0 conn(0x7f3db0147cd0 0x7f3db0148160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:09:05.228 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:05.222+0000 7f3dae59c640 1 --2- >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f3db006e570 0x7f3db006f0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.171:3301/0 says I am v2:172.21.15.104:32812/0 (socket says 172.21.15.104:32812) 2024-03-28T14:09:05.228 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:05.222+0000 7f3daed9d640 1 --2- >> v2:172.21.15.171:3300/0 conn(0x7f3db0147cd0 0x7f3db0148160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.171:3300/0 says I am v2:172.21.15.104:51778/0 (socket says 172.21.15.104:51778) 2024-03-28T14:09:05.228 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:05.222+0000 7f3dae59c640 1 -- 172.21.15.104:0/4175096624 learned_addr learned my addr 172.21.15.104:0/4175096624 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-03-28T14:09:05.229 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:05.226+0000 7f3dae59c640 1 -- 172.21.15.104:0/4175096624 >> v2:172.21.15.171:3300/0 conn(0x7f3db0147cd0 msgr2=0x7f3db0148160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T14:09:05.229 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:05.226+0000 7f3dae59c640 1 --2- 172.21.15.104:0/4175096624 >> v2:172.21.15.171:3300/0 conn(0x7f3db0147cd0 0x7f3db0148160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:09:05.229 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:05.226+0000 7f3dae59c640 1 -- 172.21.15.104:0/4175096624 >> v1:172.21.15.171:6789/0 conn(0x7f3db006e950 legacy=0x7f3db006b5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:09:05.230 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:05.226+0000 7f3dae59c640 1 -- 172.21.15.104:0/4175096624 --> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3db006d940 con 0x7f3db006e570 2024-03-28T14:09:05.230 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:05.226+0000 7f3daed9d640 1 --2- 172.21.15.104:0/4175096624 >> v2:172.21.15.171:3300/0 conn(0x7f3db0147cd0 0x7f3db0148160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T14:09:05.230 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:05.226+0000 7f3dae59c640 1 --2- 172.21.15.104:0/4175096624 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f3db006e570 0x7f3db006f0a0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f3da0009870 tx=0x7f3da001c4d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7afbc54ce2bf35a7 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:09:05.231 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:05.226+0000 7f3dadd9b640 1 -- 172.21.15.104:0/4175096624 <== mon.2 v2:172.21.15.171:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3da00b1070 con 0x7f3db006e570 2024-03-28T14:09:05.231 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:05.226+0000 7f3dadd9b640 1 -- 172.21.15.104:0/4175096624 <== mon.2 v2:172.21.15.171:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3da00ac620 con 0x7f3db006e570 2024-03-28T14:09:05.231 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:05.226+0000 7f3dadd9b640 1 -- 172.21.15.104:0/4175096624 <== mon.2 v2:172.21.15.171:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3da00ac9f0 con 0x7f3db006e570 2024-03-28T14:09:05.232 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:05.226+0000 7f3db4b31640 1 -- 172.21.15.104:0/4175096624 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f3db006e570 msgr2=0x7f3db006f0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:09:05.232 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:05.226+0000 7f3db4b31640 1 --2- 172.21.15.104:0/4175096624 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f3db006e570 0x7f3db006f0a0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f3da0009870 tx=0x7f3da001c4d0 comp rx=0 tx=0).stop 2024-03-28T14:09:05.232 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:05.226+0000 7f3db4b31640 1 -- 172.21.15.104:0/4175096624 shutdown_connections 2024-03-28T14:09:05.232 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:05.226+0000 7f3db4b31640 1 --2- 172.21.15.104:0/4175096624 >> v2:172.21.15.171:3300/0 conn(0x7f3db0147cd0 0x7f3db0148160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:09:05.232 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:05.226+0000 7f3db4b31640 1 --2- 172.21.15.104:0/4175096624 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f3db006e570 0x7f3db006f0a0 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:09:05.232 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:05.226+0000 7f3db4b31640 1 -- 172.21.15.104:0/4175096624 >> 172.21.15.104:0/4175096624 conn(0x7f3db00b7d90 msgr2=0x7f3db00b81a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T14:09:05.232 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:05.226+0000 7f3db4b31640 1 -- 172.21.15.104:0/4175096624 shutdown_connections 2024-03-28T14:09:05.232 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:05.226+0000 7f3db4b31640 1 -- 172.21.15.104:0/4175096624 wait complete. 2024-03-28T14:09:05.235 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:05.230+0000 7f3db4b31640 1 Processor -- start 2024-03-28T14:09:05.235 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:05.230+0000 7f3db4b31640 1 -- start start 2024-03-28T14:09:05.235 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:05.230+0000 7f3db4b31640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f3db006e570 0x7f3db01411d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:09:05.235 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:05.230+0000 7f3db4b31640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3db0147cd0 0x7f3db0141d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:09:05.235 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:05.230+0000 7f3db4b31640 1 --2- >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f3db0142240 0x7f3db01e3840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:09:05.235 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:05.230+0000 7f3db4b31640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f3db0154ce0 con 0x7f3db0147cd0 2024-03-28T14:09:05.235 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:05.230+0000 7f3db4b31640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f3db006eea0 con 0x7f3db006e570 2024-03-28T14:09:05.235 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:05.230+0000 7f3db4b31640 1 -- --> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] -- mon_getmap magic: 0 -- 0x7f3db0154e60 con 0x7f3db0142240 2024-03-28T14:09:05.236 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:05.230+0000 7f3da5d9b640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3db0147cd0 0x7f3db0141d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:09:05.236 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:05.230+0000 7f3da5d9b640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3db0147cd0 0x7f3db0141d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:50058/0 (socket says 172.21.15.104:50058) 2024-03-28T14:09:05.236 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:05.230+0000 7f3da5d9b640 1 -- 172.21.15.104:0/990541773 learned_addr learned my addr 172.21.15.104:0/990541773 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-03-28T14:09:05.236 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:05.230+0000 7f3dae59c640 1 --2- 172.21.15.104:0/990541773 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f3db006e570 0x7f3db01411d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:09:05.236 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:05.230+0000 7f3daed9d640 1 --2- 172.21.15.104:0/990541773 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f3db0142240 0x7f3db01e3840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:09:05.236 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:05.230+0000 7f3da5d9b640 1 -- 172.21.15.104:0/990541773 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f3db0142240 msgr2=0x7f3db01e3840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:09:05.236 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:05.230+0000 7f3da5d9b640 1 --2- 172.21.15.104:0/990541773 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f3db0142240 0x7f3db01e3840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:09:05.236 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:05.230+0000 7f3da5d9b640 1 -- 172.21.15.104:0/990541773 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f3db006e570 msgr2=0x7f3db01411d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:09:05.236 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:05.230+0000 7f3da5d9b640 1 --2- 172.21.15.104:0/990541773 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f3db006e570 0x7f3db01411d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:09:05.237 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:05.230+0000 7f3da5d9b640 1 -- 172.21.15.104:0/990541773 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3db0143900 con 0x7f3db0147cd0 2024-03-28T14:09:05.237 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:05.230+0000 7f3daed9d640 1 --2- 172.21.15.104:0/990541773 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f3db0142240 0x7f3db01e3840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T14:09:05.237 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:05.230+0000 7f3dae59c640 1 --2- 172.21.15.104:0/990541773 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f3db006e570 0x7f3db01411d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T14:09:05.237 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:05.230+0000 7f3da5d9b640 1 --2- 172.21.15.104:0/990541773 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3db0147cd0 0x7f3db0141d00 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7f3d9800c420 tx=0x7f3d9800c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:09:05.237 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:05.230+0000 7f3da77fe640 1 -- 172.21.15.104:0/990541773 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d980052d0 con 0x7f3db0147cd0 2024-03-28T14:09:05.237 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:05.230+0000 7f3da77fe640 1 -- 172.21.15.104:0/990541773 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3d98005470 con 0x7f3db0147cd0 2024-03-28T14:09:05.237 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:05.230+0000 7f3db4b31640 1 -- 172.21.15.104:0/990541773 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3db0141710 con 0x7f3db0147cd0 2024-03-28T14:09:05.237 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:05.230+0000 7f3da77fe640 1 -- 172.21.15.104:0/990541773 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d98007260 con 0x7f3db0147cd0 2024-03-28T14:09:05.238 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:05.230+0000 7f3db4b31640 1 -- 172.21.15.104:0/990541773 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3db01419c0 con 0x7f3db0147cd0 2024-03-28T14:09:05.238 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:05.230+0000 7f3db4b31640 1 -- 172.21.15.104:0/990541773 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f3db00687c0 con 0x7f3db0147cd0 2024-03-28T14:09:05.239 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:05.234+0000 7f3da77fe640 1 -- 172.21.15.104:0/990541773 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) ==== 87803+0+0 (secure 0 0 0) 0x7f3d98007400 con 0x7f3db0147cd0 2024-03-28T14:09:05.239 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:05.234+0000 7f3da77fe640 1 --2- 172.21.15.104:0/990541773 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f3d90067ff0 0x7f3d9006a4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:09:05.239 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:05.234+0000 7f3da77fe640 1 -- 172.21.15.104:0/990541773 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(33..33 src has 1..33) ==== 4943+0+0 (secure 0 0 0) 0x7f3d98086240 con 0x7f3db0147cd0 2024-03-28T14:09:05.239 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:05.234+0000 7f3da77fe640 1 --2- 172.21.15.104:0/990541773 >> [v2:172.21.15.171:6809/2055631175,v1:172.21.15.171:6811/2055631175] conn(0x7f3d9006deb0 0x7f3d900702d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:09:05.240 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:05.234+0000 7f3da77fe640 1 -- 172.21.15.104:0/990541773 --> [v2:172.21.15.171:6809/2055631175,v1:172.21.15.171:6811/2055631175] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f3d98015ec0 con 0x7f3d9006deb0 2024-03-28T14:09:05.240 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:05.234+0000 7f3da77fe640 1 -- 172.21.15.104:0/990541773 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7f3d980866d0 con 0x7f3db0147cd0 2024-03-28T14:09:05.240 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:05.234+0000 7f3dae59c640 1 --2- 172.21.15.104:0/990541773 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f3d90067ff0 0x7f3d9006a4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:09:05.240 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:05.234+0000 7f3daed9d640 1 --2- 172.21.15.104:0/990541773 >> [v2:172.21.15.171:6809/2055631175,v1:172.21.15.171:6811/2055631175] conn(0x7f3d9006deb0 0x7f3d900702d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:09:05.240 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:05.234+0000 7f3dae59c640 1 --2- 172.21.15.104:0/990541773 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f3d90067ff0 0x7f3d9006a4b0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f3da0009870 tx=0x7f3da0008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:09:05.240 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:05.234+0000 7f3daed9d640 1 --2- 172.21.15.104:0/990541773 >> [v2:172.21.15.171:6809/2055631175,v1:172.21.15.171:6811/2055631175] conn(0x7f3d9006deb0 0x7f3d900702d0 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:09:05.240 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:05.234+0000 7f3da77fe640 1 -- 172.21.15.104:0/990541773 <== osd.7 v2:172.21.15.171:6809/2055631175 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f3d98015ec0 con 0x7f3d9006deb0 2024-03-28T14:09:05.252 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:05.250+0000 7f3db4b31640 1 -- 172.21.15.104:0/990541773 --> [v2:172.21.15.171:6809/2055631175,v1:172.21.15.171:6811/2055631175] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7f3db0142950 con 0x7f3d9006deb0 2024-03-28T14:09:05.253 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:05.250+0000 7f3da77fe640 1 -- 172.21.15.104:0/990541773 <== osd.7 v2:172.21.15.171:6809/2055631175 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7f3db0142950 con 0x7f3d9006deb0 2024-03-28T14:09:05.253 INFO:teuthology.orchestra.run.smithi104.stdout:{ 2024-03-28T14:09:05.253 INFO:teuthology.orchestra.run.smithi104.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-03-28T14:09:05.254 INFO:teuthology.orchestra.run.smithi104.stdout:} 2024-03-28T14:09:05.254 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:05.250+0000 7f3db4b31640 1 -- 172.21.15.104:0/990541773 >> [v2:172.21.15.171:6809/2055631175,v1:172.21.15.171:6811/2055631175] conn(0x7f3d9006deb0 msgr2=0x7f3d900702d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:09:05.255 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:05.250+0000 7f3db4b31640 1 --2- 172.21.15.104:0/990541773 >> [v2:172.21.15.171:6809/2055631175,v1:172.21.15.171:6811/2055631175] conn(0x7f3d9006deb0 0x7f3d900702d0 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:09:05.255 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:05.250+0000 7f3db4b31640 1 -- 172.21.15.104:0/990541773 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f3d90067ff0 msgr2=0x7f3d9006a4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:09:05.255 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:05.250+0000 7f3db4b31640 1 --2- 172.21.15.104:0/990541773 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f3d90067ff0 0x7f3d9006a4b0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f3da0009870 tx=0x7f3da0008040 comp rx=0 tx=0).stop 2024-03-28T14:09:05.255 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:05.250+0000 7f3db4b31640 1 -- 172.21.15.104:0/990541773 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3db0147cd0 msgr2=0x7f3db0141d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:09:05.255 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:05.250+0000 7f3db4b31640 1 --2- 172.21.15.104:0/990541773 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3db0147cd0 0x7f3db0141d00 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7f3d9800c420 tx=0x7f3d9800c8e0 comp rx=0 tx=0).stop 2024-03-28T14:09:05.255 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:05.250+0000 7f3daed9d640 1 -- 172.21.15.104:0/990541773 reap_dead start 2024-03-28T14:09:05.255 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:05.250+0000 7f3db4b31640 1 -- 172.21.15.104:0/990541773 shutdown_connections 2024-03-28T14:09:05.255 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:05.250+0000 7f3db4b31640 1 -- 172.21.15.104:0/990541773 >> 172.21.15.104:0/990541773 conn(0x7f3db00b7d90 msgr2=0x7f3db0070c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T14:09:05.256 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:05.250+0000 7f3db4b31640 1 -- 172.21.15.104:0/990541773 shutdown_connections 2024-03-28T14:09:05.256 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:05.250+0000 7f3db4b31640 1 -- 172.21.15.104:0/990541773 wait complete. 2024-03-28T14:09:05.269 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 7 2024-03-28T14:09:05.382 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:05.378+0000 7fb176f5e640 1 Processor -- start 2024-03-28T14:09:05.383 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:05.378+0000 7fb176f5e640 1 -- start start 2024-03-28T14:09:05.384 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:05.378+0000 7fb176f5e640 1 --2- >> v2:172.21.15.171:3300/0 conn(0x7fb1700b4aa0 0x7fb1700af660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:09:05.384 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:05.378+0000 7fb176f5e640 1 --2- >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fb1700b29e0 0x7fb17014ef20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:09:05.384 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:05.378+0000 7fb176f5e640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb170148030 con 0x7fb170147bf0 2024-03-28T14:09:05.384 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:05.378+0000 7fb176f5e640 1 -- --> v2:172.21.15.171:3300/0 -- mon_getmap magic: 0 -- 0x7fb1700b4e80 con 0x7fb1700b4aa0 2024-03-28T14:09:05.384 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:05.378+0000 7fb176f5e640 1 -- --> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] -- mon_getmap magic: 0 -- 0x7fb1700b0890 con 0x7fb1700b29e0 2024-03-28T14:09:05.384 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:05.378+0000 7fb1754fb640 1 --2- >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fb1700b29e0 0x7fb17014ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:09:05.384 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:05.378+0000 7fb174cfa640 1 --2- >> v2:172.21.15.171:3300/0 conn(0x7fb1700b4aa0 0x7fb1700af660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:09:05.384 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:05.378+0000 7fb1754fb640 1 --2- >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fb1700b29e0 0x7fb17014ef20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.171:3301/0 says I am v2:172.21.15.104:32824/0 (socket says 172.21.15.104:32824) 2024-03-28T14:09:05.385 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:05.378+0000 7fb1754fb640 1 -- 172.21.15.104:0/300156791 learned_addr learned my addr 172.21.15.104:0/300156791 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-03-28T14:09:05.385 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:05.382+0000 7fb174cfa640 1 -- 172.21.15.104:0/300156791 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fb1700b29e0 msgr2=0x7fb17014ef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T14:09:05.385 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:05.382+0000 7fb174cfa640 1 --2- 172.21.15.104:0/300156791 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fb1700b29e0 0x7fb17014ef20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:09:05.385 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:05.382+0000 7fb174cfa640 1 -- 172.21.15.104:0/300156791 >> v1:172.21.15.171:6789/0 conn(0x7fb170147bf0 legacy=0x7fb1700b22d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:09:05.386 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:05.382+0000 7fb174cfa640 1 -- 172.21.15.104:0/300156791 --> v2:172.21.15.171:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb1700ae0a0 con 0x7fb1700b4aa0 2024-03-28T14:09:05.386 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:05.382+0000 7fb1754fb640 1 --2- 172.21.15.104:0/300156791 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fb1700b29e0 0x7fb17014ef20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T14:09:05.386 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:05.382+0000 7fb174cfa640 1 --2- 172.21.15.104:0/300156791 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fb1700b4aa0 0x7fb1700af660 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fb16c0097c0 tx=0x7fb16c01c410 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3a61dcb9743d794e server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:09:05.387 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:05.382+0000 7fb1677fe640 1 -- 172.21.15.104:0/300156791 <== mon.1 v2:172.21.15.171:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb16c0b0070 con 0x7fb1700b4aa0 2024-03-28T14:09:05.387 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:05.382+0000 7fb1677fe640 1 -- 172.21.15.104:0/300156791 <== mon.1 v2:172.21.15.171:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb16c0ab600 con 0x7fb1700b4aa0 2024-03-28T14:09:05.387 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:05.382+0000 7fb1677fe640 1 -- 172.21.15.104:0/300156791 <== mon.1 v2:172.21.15.171:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb16c0ab9d0 con 0x7fb1700b4aa0 2024-03-28T14:09:05.388 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:05.382+0000 7fb176f5e640 1 -- 172.21.15.104:0/300156791 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fb1700b4aa0 msgr2=0x7fb1700af660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:09:05.388 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:05.382+0000 7fb176f5e640 1 --2- 172.21.15.104:0/300156791 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fb1700b4aa0 0x7fb1700af660 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fb16c0097c0 tx=0x7fb16c01c410 comp rx=0 tx=0).stop 2024-03-28T14:09:05.388 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:05.382+0000 7fb176f5e640 1 -- 172.21.15.104:0/300156791 shutdown_connections 2024-03-28T14:09:05.388 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:05.382+0000 7fb176f5e640 1 --2- 172.21.15.104:0/300156791 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fb1700b29e0 0x7fb17014ef20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:09:05.389 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:05.382+0000 7fb176f5e640 1 --2- 172.21.15.104:0/300156791 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fb1700b4aa0 0x7fb1700af660 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:09:05.389 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:05.382+0000 7fb176f5e640 1 -- 172.21.15.104:0/300156791 >> 172.21.15.104:0/300156791 conn(0x7fb1700b8f00 msgr2=0x7fb1700b9310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T14:09:05.389 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:05.382+0000 7fb176f5e640 1 -- 172.21.15.104:0/300156791 shutdown_connections 2024-03-28T14:09:05.389 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:05.382+0000 7fb176f5e640 1 -- 172.21.15.104:0/300156791 wait complete. 2024-03-28T14:09:05.391 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:05.386+0000 7fb176f5e640 1 Processor -- start 2024-03-28T14:09:05.391 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:05.386+0000 7fb176f5e640 1 -- start start 2024-03-28T14:09:05.391 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:05.386+0000 7fb176f5e640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb1700b29e0 0x7fb170147410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:09:05.391 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:05.386+0000 7fb176f5e640 1 --2- >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fb1700b4aa0 0x7fb170143510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:09:05.391 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:05.386+0000 7fb176f5e640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fb170147bf0 0x7fb17009c350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:09:05.391 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:05.386+0000 7fb176f5e640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7fb17006dc40 con 0x7fb1700b29e0 2024-03-28T14:09:05.392 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:05.386+0000 7fb176f5e640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7fb1700ae9a0 con 0x7fb170147bf0 2024-03-28T14:09:05.392 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:05.386+0000 7fb176f5e640 1 -- --> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] -- mon_getmap magic: 0 -- 0x7fb17009d700 con 0x7fb1700b4aa0 2024-03-28T14:09:05.392 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:05.386+0000 7fb174cfa640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb1700b29e0 0x7fb170147410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:09:05.392 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:05.386+0000 7fb174cfa640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb1700b29e0 0x7fb170147410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:50070/0 (socket says 172.21.15.104:50070) 2024-03-28T14:09:05.392 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:05.386+0000 7fb174cfa640 1 -- 172.21.15.104:0/571310934 learned_addr learned my addr 172.21.15.104:0/571310934 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-03-28T14:09:05.392 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:05.386+0000 7fb167fff640 1 --2- 172.21.15.104:0/571310934 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fb1700b4aa0 0x7fb170143510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:09:05.392 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:05.386+0000 7fb1754fb640 1 --2- 172.21.15.104:0/571310934 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fb170147bf0 0x7fb17009c350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:09:05.393 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:05.386+0000 7fb174cfa640 1 -- 172.21.15.104:0/571310934 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fb1700b4aa0 msgr2=0x7fb170143510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:09:05.393 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:05.386+0000 7fb174cfa640 1 --2- 172.21.15.104:0/571310934 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fb1700b4aa0 0x7fb170143510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:09:05.393 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:05.386+0000 7fb174cfa640 1 -- 172.21.15.104:0/571310934 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fb170147bf0 msgr2=0x7fb17009c350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:09:05.393 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:05.386+0000 7fb174cfa640 1 --2- 172.21.15.104:0/571310934 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fb170147bf0 0x7fb17009c350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:09:05.393 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:05.386+0000 7fb174cfa640 1 -- 172.21.15.104:0/571310934 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb17009d9a0 con 0x7fb1700b29e0 2024-03-28T14:09:05.393 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:05.386+0000 7fb174cfa640 1 --2- 172.21.15.104:0/571310934 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb1700b29e0 0x7fb170147410 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7fb16c0097c0 tx=0x7fb16c0191b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:09:05.393 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:05.386+0000 7fb1657fa640 1 -- 172.21.15.104:0/571310934 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb16c0b0040 con 0x7fb1700b29e0 2024-03-28T14:09:05.394 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:05.386+0000 7fb1657fa640 1 -- 172.21.15.104:0/571310934 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb16c0b9070 con 0x7fb1700b29e0 2024-03-28T14:09:05.394 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:05.386+0000 7fb176f5e640 1 -- 172.21.15.104:0/571310934 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb17009cd20 con 0x7fb1700b29e0 2024-03-28T14:09:05.394 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:05.386+0000 7fb1657fa640 1 -- 172.21.15.104:0/571310934 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb16c0b4650 con 0x7fb1700b29e0 2024-03-28T14:09:05.394 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:05.386+0000 7fb176f5e640 1 -- 172.21.15.104:0/571310934 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb17009aa80 con 0x7fb1700b29e0 2024-03-28T14:09:05.395 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:05.386+0000 7fb176f5e640 1 -- 172.21.15.104:0/571310934 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb1700687c0 con 0x7fb1700b29e0 2024-03-28T14:09:05.395 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:05.390+0000 7fb1657fa640 1 -- 172.21.15.104:0/571310934 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) ==== 87803+0+0 (secure 0 0 0) 0x7fb16c0b47f0 con 0x7fb1700b29e0 2024-03-28T14:09:05.395 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:05.390+0000 7fb1657fa640 1 --2- 172.21.15.104:0/571310934 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7fb15c067fc0 0x7fb15c06a480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:09:05.395 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:05.390+0000 7fb1657fa640 1 -- 172.21.15.104:0/571310934 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(33..33 src has 1..33) ==== 4943+0+0 (secure 0 0 0) 0x7fb16c11ff50 con 0x7fb1700b29e0 2024-03-28T14:09:05.396 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:05.390+0000 7fb167fff640 1 --2- 172.21.15.104:0/571310934 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7fb15c067fc0 0x7fb15c06a480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:09:05.396 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:05.390+0000 7fb167fff640 1 --2- 172.21.15.104:0/571310934 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7fb15c067fc0 0x7fb15c06a480 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fb1701445c0 tx=0x7fb160009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:09:05.397 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:05.394+0000 7fb1657fa640 1 -- 172.21.15.104:0/571310934 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179117 (secure 0 0 0) 0x7fb16c016090 con 0x7fb1700b29e0 2024-03-28T14:09:05.555 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:05.550+0000 7fb176f5e640 1 -- 172.21.15.104:0/571310934 --> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "7", "target": ["mon-mgr", ""]}) -- 0x7fb1700ae780 con 0x7fb15c067fc0 2024-03-28T14:09:05.555 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:05.550+0000 7fb1657fa640 1 -- 172.21.15.104:0/571310934 <== mgr.4100 v2:172.21.15.171:6832/1522752412 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 7 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7fb1700ae780 con 0x7fb15c067fc0 2024-03-28T14:09:05.556 INFO:teuthology.orchestra.run.smithi104.stderr:instructed osd(s) 7 to deep-scrub 2024-03-28T14:09:05.558 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:05.554+0000 7fb176f5e640 1 -- 172.21.15.104:0/571310934 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7fb15c067fc0 msgr2=0x7fb15c06a480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:09:05.558 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:05.554+0000 7fb176f5e640 1 --2- 172.21.15.104:0/571310934 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7fb15c067fc0 0x7fb15c06a480 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fb1701445c0 tx=0x7fb160009040 comp rx=0 tx=0).stop 2024-03-28T14:09:05.558 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:05.554+0000 7fb176f5e640 1 -- 172.21.15.104:0/571310934 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb1700b29e0 msgr2=0x7fb170147410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:09:05.558 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:05.554+0000 7fb176f5e640 1 --2- 172.21.15.104:0/571310934 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb1700b29e0 0x7fb170147410 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7fb16c0097c0 tx=0x7fb16c0191b0 comp rx=0 tx=0).stop 2024-03-28T14:09:05.558 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:05.554+0000 7fb176f5e640 1 -- 172.21.15.104:0/571310934 shutdown_connections 2024-03-28T14:09:05.558 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:05.554+0000 7fb176f5e640 1 --2- 172.21.15.104:0/571310934 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fb170147bf0 0x7fb17009c350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:09:05.558 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:05.554+0000 7fb176f5e640 1 --2- 172.21.15.104:0/571310934 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7fb1700b4aa0 0x7fb170143510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:09:05.559 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:05.554+0000 7fb176f5e640 1 --2- 172.21.15.104:0/571310934 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7fb15c067fc0 0x7fb15c06a480 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:09:05.559 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:05.554+0000 7fb176f5e640 1 --2- 172.21.15.104:0/571310934 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb1700b29e0 0x7fb170147410 unknown :-1 s=CLOSED pgs=480 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:09:05.559 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:05.554+0000 7fb176f5e640 1 -- 172.21.15.104:0/571310934 >> 172.21.15.104:0/571310934 conn(0x7fb1700b8f00 msgr2=0x7fb1700b3490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T14:09:05.559 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:05.554+0000 7fb176f5e640 1 -- 172.21.15.104:0/571310934 shutdown_connections 2024-03-28T14:09:05.559 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:05.554+0000 7fb176f5e640 1 -- 172.21.15.104:0/571310934 wait complete. 2024-03-28T14:09:05.573 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-28T14:09:05.688 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:05.686+0000 7efe9c461640 1 Processor -- start 2024-03-28T14:09:05.689 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:05.686+0000 7efe9c461640 1 -- start start 2024-03-28T14:09:05.690 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:05.686+0000 7efe9c461640 1 --2- >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7efe940b6790 0x7efe9413f470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:09:05.690 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:05.686+0000 7efe9c461640 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7efe9413f9b0 0x7efe940b1710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:09:05.690 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:05.686+0000 7efe9c461640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efe940bbca0 con 0x7efe940bb8c0 2024-03-28T14:09:05.690 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:05.686+0000 7efe9c461640 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 -- 0x7efe940b7430 con 0x7efe9413f9b0 2024-03-28T14:09:05.690 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:05.686+0000 7efe9c461640 1 -- --> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] -- mon_getmap magic: 0 -- 0x7efe940b6de0 con 0x7efe940b6790 2024-03-28T14:09:05.690 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:05.686+0000 7efe9a1fd640 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7efe940bb8c0 0x7efe940b5950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:40640/0 (socket says 172.21.15.104:40640) 2024-03-28T14:09:05.691 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:05.686+0000 7efe9a1fd640 1 -- 172.21.15.104:0/1597578226 learned_addr learned my addr 172.21.15.104:0/1597578226 (peer_addr_for_me v1:172.21.15.104:0/0) 2024-03-28T14:09:05.691 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:05.686+0000 7efe9a9fe640 1 --2- 172.21.15.104:0/1597578226 >> v2:172.21.15.104:3300/0 conn(0x7efe9413f9b0 0x7efe940b1710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:09:05.691 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:05.686+0000 7efe999fc640 1 --2- 172.21.15.104:0/1597578226 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7efe940b6790 0x7efe9413f470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:09:05.691 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:05.686+0000 7efe9a9fe640 1 -- 172.21.15.104:0/1597578226 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7efe940b6790 msgr2=0x7efe9413f470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T14:09:05.691 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:05.686+0000 7efe9a9fe640 1 --2- 172.21.15.104:0/1597578226 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7efe940b6790 0x7efe9413f470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:09:05.691 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:05.686+0000 7efe9a9fe640 1 -- 172.21.15.104:0/1597578226 >> v1:172.21.15.104:6789/0 conn(0x7efe940bb8c0 legacy=0x7efe940b5950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:09:05.691 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:05.686+0000 7efe9a9fe640 1 -- 172.21.15.104:0/1597578226 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efe940b70c0 con 0x7efe9413f9b0 2024-03-28T14:09:05.693 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:05.686+0000 7efe9a9fe640 1 --2- 172.21.15.104:0/1597578226 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7efe9413f9b0 0x7efe940b1710 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7efe8c009d90 tx=0x7efe8c01c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=78d5ae10c5d8c81b server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:09:05.693 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:05.690+0000 7efe991fb640 1 -- 172.21.15.104:0/1597578226 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe8c0b2070 con 0x7efe9413f9b0 2024-03-28T14:09:05.693 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:05.690+0000 7efe991fb640 1 -- 172.21.15.104:0/1597578226 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efe8c0a57a0 con 0x7efe9413f9b0 2024-03-28T14:09:05.693 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:05.690+0000 7efe991fb640 1 -- 172.21.15.104:0/1597578226 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe8c0a5b70 con 0x7efe9413f9b0 2024-03-28T14:09:05.693 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:05.690+0000 7efe9c461640 1 -- 172.21.15.104:0/1597578226 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7efe9413f9b0 msgr2=0x7efe940b1710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:09:05.693 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:05.690+0000 7efe9c461640 1 --2- 172.21.15.104:0/1597578226 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7efe9413f9b0 0x7efe940b1710 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7efe8c009d90 tx=0x7efe8c01c8e0 comp rx=0 tx=0).stop 2024-03-28T14:09:05.693 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:05.690+0000 7efe9c461640 1 -- 172.21.15.104:0/1597578226 shutdown_connections 2024-03-28T14:09:05.694 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:05.690+0000 7efe9c461640 1 --2- 172.21.15.104:0/1597578226 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7efe9413f9b0 0x7efe940b1710 unknown :-1 s=CLOSED pgs=482 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:09:05.694 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:05.690+0000 7efe9c461640 1 --2- 172.21.15.104:0/1597578226 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7efe940b6790 0x7efe9413f470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:09:05.694 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:05.690+0000 7efe9c461640 1 -- 172.21.15.104:0/1597578226 >> 172.21.15.104:0/1597578226 conn(0x7efe9402c800 msgr2=0x7efe940baf50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T14:09:05.694 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:05.690+0000 7efe9c461640 1 -- 172.21.15.104:0/1597578226 shutdown_connections 2024-03-28T14:09:05.694 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:05.690+0000 7efe9c461640 1 -- 172.21.15.104:0/1597578226 wait complete. 2024-03-28T14:09:05.696 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:05.690+0000 7efe9c461640 1 Processor -- start 2024-03-28T14:09:05.696 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:05.690+0000 7efe9c461640 1 -- start start 2024-03-28T14:09:05.697 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:05.690+0000 7efe9c461640 1 --2- >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7efe940b6790 0x7efe940ab350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:09:05.697 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:05.690+0000 7efe9c461640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7efe940bb8c0 0x7efe9413b200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:09:05.697 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:05.690+0000 7efe9c461640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7efe9413f9b0 0x7efe9413b740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:09:05.697 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:05.690+0000 7efe9c461640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7efe94152a10 con 0x7efe940bb8c0 2024-03-28T14:09:05.697 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:05.690+0000 7efe9c461640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7efe94152890 con 0x7efe9413f9b0 2024-03-28T14:09:05.697 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:05.690+0000 7efe9c461640 1 -- --> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] -- mon_getmap magic: 0 -- 0x7efe94152b90 con 0x7efe940b6790 2024-03-28T14:09:05.697 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:05.690+0000 7efe999fc640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7efe940bb8c0 0x7efe9413b200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:09:05.697 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:05.690+0000 7efe999fc640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7efe940bb8c0 0x7efe9413b200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:50082/0 (socket says 172.21.15.104:50082) 2024-03-28T14:09:05.698 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:05.690+0000 7efe999fc640 1 -- 172.21.15.104:0/1883001854 learned_addr learned my addr 172.21.15.104:0/1883001854 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-03-28T14:09:05.698 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:05.690+0000 7efe9a1fd640 1 --2- 172.21.15.104:0/1883001854 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7efe940b6790 0x7efe940ab350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:09:05.698 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:05.690+0000 7efe999fc640 1 -- 172.21.15.104:0/1883001854 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7efe940b6790 msgr2=0x7efe940ab350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:09:05.698 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:05.690+0000 7efe999fc640 1 --2- 172.21.15.104:0/1883001854 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7efe940b6790 0x7efe940ab350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:09:05.698 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:05.690+0000 7efe999fc640 1 -- 172.21.15.104:0/1883001854 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7efe9413f9b0 msgr2=0x7efe9413b740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:09:05.698 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:05.690+0000 7efe999fc640 1 --2- 172.21.15.104:0/1883001854 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7efe9413f9b0 0x7efe9413b740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:09:05.698 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:05.690+0000 7efe999fc640 1 -- 172.21.15.104:0/1883001854 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efe940aa5e0 con 0x7efe940bb8c0 2024-03-28T14:09:05.698 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:05.690+0000 7efe999fc640 1 --2- 172.21.15.104:0/1883001854 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7efe940bb8c0 0x7efe9413b200 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7efe8400d830 tx=0x7efe8400dcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:09:05.698 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:05.690+0000 7efe8affd640 1 -- 172.21.15.104:0/1883001854 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe8400b6c0 con 0x7efe940bb8c0 2024-03-28T14:09:05.699 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:05.690+0000 7efe8affd640 1 -- 172.21.15.104:0/1883001854 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efe84019070 con 0x7efe940bb8c0 2024-03-28T14:09:05.699 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:05.690+0000 7efe8affd640 1 -- 172.21.15.104:0/1883001854 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe8400ba60 con 0x7efe940bb8c0 2024-03-28T14:09:05.699 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:05.690+0000 7efe9c461640 1 -- 172.21.15.104:0/1883001854 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efe94142dd0 con 0x7efe940bb8c0 2024-03-28T14:09:05.699 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:05.690+0000 7efe9c461640 1 -- 172.21.15.104:0/1883001854 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efe940a9480 con 0x7efe940bb8c0 2024-03-28T14:09:05.699 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:05.694+0000 7efe9c461640 1 -- 172.21.15.104:0/1883001854 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efe940aa7b0 con 0x7efe940bb8c0 2024-03-28T14:09:05.699 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:05.694+0000 7efe8affd640 1 -- 172.21.15.104:0/1883001854 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) ==== 87803+0+0 (secure 0 0 0) 0x7efe8401e460 con 0x7efe940bb8c0 2024-03-28T14:09:05.700 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:05.694+0000 7efe8affd640 1 --2- 172.21.15.104:0/1883001854 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7efe78067fc0 0x7efe7806a480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:09:05.700 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:05.694+0000 7efe8affd640 1 -- 172.21.15.104:0/1883001854 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(33..33 src has 1..33) ==== 4943+0+0 (secure 0 0 0) 0x7efe84086830 con 0x7efe940bb8c0 2024-03-28T14:09:05.700 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:05.694+0000 7efe9a1fd640 1 --2- 172.21.15.104:0/1883001854 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7efe78067fc0 0x7efe7806a480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:09:05.701 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:05.694+0000 7efe9a1fd640 1 --2- 172.21.15.104:0/1883001854 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7efe78067fc0 0x7efe7806a480 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7efe90000f70 tx=0x7efe90009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:09:05.701 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:05.698+0000 7efe8affd640 1 -- 172.21.15.104:0/1883001854 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179117 (secure 0 0 0) 0x7efe940aa7b0 con 0x7efe940bb8c0 2024-03-28T14:09:05.847 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:05.842+0000 7efe9c461640 1 -- 172.21.15.104:0/1883001854 --> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7efe940687c0 con 0x7efe78067fc0 2024-03-28T14:09:05.850 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:05.846+0000 7efe8affd640 1 -- 172.21.15.104:0/1883001854 <== mgr.4100 v2:172.21.15.171:6832/1522752412 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+13463 (secure 0 0 0) 0x7efe940687c0 con 0x7efe78067fc0 2024-03-28T14:09:05.850 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-03-28T14:09:05.850 INFO:teuthology.orchestra.run.smithi104.stderr:dumped all 2024-03-28T14:09:05.852 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:05.846+0000 7efe9c461640 1 -- 172.21.15.104:0/1883001854 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7efe78067fc0 msgr2=0x7efe7806a480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:09:05.852 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:05.846+0000 7efe9c461640 1 --2- 172.21.15.104:0/1883001854 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7efe78067fc0 0x7efe7806a480 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7efe90000f70 tx=0x7efe90009040 comp rx=0 tx=0).stop 2024-03-28T14:09:05.852 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:05.846+0000 7efe9c461640 1 -- 172.21.15.104:0/1883001854 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7efe940bb8c0 msgr2=0x7efe9413b200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:09:05.852 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:05.846+0000 7efe9c461640 1 --2- 172.21.15.104:0/1883001854 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7efe940bb8c0 0x7efe9413b200 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7efe8400d830 tx=0x7efe8400dcf0 comp rx=0 tx=0).stop 2024-03-28T14:09:05.852 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:05.846+0000 7efe9c461640 1 -- 172.21.15.104:0/1883001854 shutdown_connections 2024-03-28T14:09:05.853 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:05.846+0000 7efe9c461640 1 --2- 172.21.15.104:0/1883001854 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7efe9413f9b0 0x7efe9413b740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:09:05.853 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:05.846+0000 7efe9c461640 1 --2- 172.21.15.104:0/1883001854 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7efe78067fc0 0x7efe7806a480 secure :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7efe90000f70 tx=0x7efe90009040 comp rx=0 tx=0).stop 2024-03-28T14:09:05.853 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:05.846+0000 7efe9c461640 1 --2- 172.21.15.104:0/1883001854 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7efe940bb8c0 0x7efe9413b200 unknown :-1 s=CLOSED pgs=483 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:09:05.853 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:05.846+0000 7efe9c461640 1 --2- 172.21.15.104:0/1883001854 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7efe940b6790 0x7efe940ab350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:09:05.853 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:05.846+0000 7efe9c461640 1 -- 172.21.15.104:0/1883001854 >> 172.21.15.104:0/1883001854 conn(0x7efe9402c800 msgr2=0x7efe940b41a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T14:09:05.853 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:05.846+0000 7efe9c461640 1 -- 172.21.15.104:0/1883001854 shutdown_connections 2024-03-28T14:09:05.853 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:05.846+0000 7efe9c461640 1 -- 172.21.15.104:0/1883001854 wait complete. 2024-03-28T14:09:05.867 INFO:teuthology.orchestra.run.smithi104.stdout:{"pg_ready":true,"pg_map":{"version":66,"stamp":"2024-03-28T14:09:05.600179+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":57,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":2,"kb":754974720,"kb_used":220304,"kb_used_data":4136,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754754416,"statfs":{"total":773094113280,"available":772868521984,"internally_reserved":0,"allocated":4235264,"data_stored":2351392,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":221236785},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":17,"apply_latency_ms":17,"commit_latency_ns":17000000,"apply_latency_ns":17000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.661974"},"pg_stats":[{"pgid":"1.0","version":"17'117","reported_seq":192,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-28T14:08:59.950018+0000","last_change":"2024-03-28T14:07:49.234448+0000","last_active":"2024-03-28T14:08:59.950018+0000","last_peered":"2024-03-28T14:08:59.950018+0000","last_clean":"2024-03-28T14:08:59.950018+0000","last_became_active":"2024-03-28T14:07:49.233895+0000","last_became_peered":"2024-03-28T14:07:49.233895+0000","last_unstale":"2024-03-28T14:08:59.950018+0000","last_undegraded":"2024-03-28T14:08:59.950018+0000","last_fullsized":"2024-03-28T14:08:59.950018+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T14:07:48.069157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T14:07:48.069157+0000","last_clean_scrub_stamp":"2024-03-28T14:07:48.069157+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:11:39.977557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574867,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27972,"kb_used_data":952,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343868,"statfs":{"total":96636764160,"available":96608120832,"internally_reserved":0,"allocated":974848,"data_stored":736700,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574868,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27396,"kb_used_data":372,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344444,"statfs":{"total":96636764160,"available":96608710656,"internally_reserved":0,"allocated":380928,"data_stored":146332,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574866,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27388,"kb_used_data":372,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344452,"statfs":{"total":96636764160,"available":96608718848,"internally_reserved":0,"allocated":380928,"data_stored":146332,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574867,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27396,"kb_used_data":372,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344444,"statfs":{"total":96636764160,"available":96608710656,"internally_reserved":0,"allocated":380928,"data_stored":146332,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574869,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27392,"kb_used_data":372,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344448,"statfs":{"total":96636764160,"available":96608714752,"internally_reserved":0,"allocated":380928,"data_stored":146332,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574867,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27392,"kb_used_data":372,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344448,"statfs":{"total":96636764160,"available":96608714752,"internally_reserved":0,"allocated":380928,"data_stored":146332,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574865,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27392,"kb_used_data":372,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344448,"statfs":{"total":96636764160,"available":96608714752,"internally_reserved":0,"allocated":380928,"data_stored":146332,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574868,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27976,"kb_used_data":952,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343864,"statfs":{"total":96636764160,"available":96608116736,"internally_reserved":0,"allocated":974848,"data_stored":736700,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-28T14:09:05.869 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2024-03-28T14:07:48.069157+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=28, tm_hour=14, tm_min=7, tm_sec=48, tm_wday=3, tm_yday=88, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=28, tm_hour=14, tm_min=9, tm_sec=0, tm_wday=3, tm_yday=88, tm_isdst=0) 2024-03-28T14:09:05.869 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2024-03-28T14:09:25.871 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-28T14:09:26.000 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:25.997+0000 7f34809f3640 1 Processor -- start 2024-03-28T14:09:26.001 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:25.997+0000 7f34809f3640 1 -- start start 2024-03-28T14:09:26.003 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:25.997+0000 7f34809f3640 1 --2- >> v2:172.21.15.171:3300/0 conn(0x7f347c13faf0 0x7f347c0b0c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:09:26.003 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:25.997+0000 7f34809f3640 1 --2- >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f347c0732a0 0x7f347c0b1170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:09:26.003 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:25.997+0000 7f34809f3640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f347c06dd50 con 0x7f347c06d970 2024-03-28T14:09:26.003 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:25.997+0000 7f34809f3640 1 -- --> v2:172.21.15.171:3300/0 -- mon_getmap magic: 0 -- 0x7f347c0b02a0 con 0x7f347c13faf0 2024-03-28T14:09:26.003 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:25.997+0000 7f34809f3640 1 -- --> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] -- mon_getmap magic: 0 -- 0x7f347c0b2d30 con 0x7f347c0732a0 2024-03-28T14:09:26.004 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:25.997+0000 7f347a59c640 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f347c06d970 0x7f347c13f3e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:59346/0 (socket says 172.21.15.104:59346) 2024-03-28T14:09:26.004 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:25.997+0000 7f347a59c640 1 -- 172.21.15.104:0/734218020 learned_addr learned my addr 172.21.15.104:0/734218020 (peer_addr_for_me v1:172.21.15.104:0/0) 2024-03-28T14:09:26.004 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:25.997+0000 7f347ad9d640 1 --2- 172.21.15.104:0/734218020 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f347c0732a0 0x7f347c0b1170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:09:26.004 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:25.997+0000 7f3479d9b640 1 --2- 172.21.15.104:0/734218020 >> v2:172.21.15.171:3300/0 conn(0x7f347c13faf0 0x7f347c0b0c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:09:26.004 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:25.997+0000 7f347959a640 1 -- 172.21.15.104:0/734218020 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1929757866 0 0) 0x7f347c06dd50 con 0x7f347c06d970 2024-03-28T14:09:26.004 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:25.997+0000 7f347959a640 1 -- 172.21.15.104:0/734218020 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3468003530 con 0x7f347c06d970 2024-03-28T14:09:26.004 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:25.997+0000 7f347959a640 1 -- 172.21.15.104:0/734218020 <== mon.0 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 416645479 0 0) 0x7f3468003530 con 0x7f347c06d970 2024-03-28T14:09:26.004 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:25.997+0000 7f347959a640 1 -- 172.21.15.104:0/734218020 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f347c0732a0 msgr2=0x7f347c0b1170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T14:09:26.005 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:25.997+0000 7f347959a640 1 --2- 172.21.15.104:0/734218020 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f347c0732a0 0x7f347c0b1170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:09:26.005 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:25.997+0000 7f347959a640 1 -- 172.21.15.104:0/734218020 >> v2:172.21.15.171:3300/0 conn(0x7f347c13faf0 msgr2=0x7f347c0b0c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T14:09:26.005 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:25.997+0000 7f347959a640 1 --2- 172.21.15.104:0/734218020 >> v2:172.21.15.171:3300/0 conn(0x7f347c13faf0 0x7f347c0b0c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:09:26.005 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:25.997+0000 7f347959a640 1 -- 172.21.15.104:0/734218020 --> v1:172.21.15.104:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f347c0b3010 con 0x7f347c06d970 2024-03-28T14:09:26.005 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:25.997+0000 7f347959a640 1 -- 172.21.15.104:0/734218020 <== mon.0 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3745580849 0 0) 0x7f346c001680 con 0x7f347c06d970 2024-03-28T14:09:26.005 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:25.997+0000 7f347959a640 1 -- 172.21.15.104:0/734218020 >> v1:172.21.15.104:6789/0 conn(0x7f347c06d970 legacy=0x7f347c13f3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:09:26.005 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:25.997+0000 7f347959a640 1 --2- 172.21.15.104:0/734218020 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f347c0732a0 0x7f34680039c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:09:26.005 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:25.997+0000 7f347959a640 1 --2- 172.21.15.104:0/734218020 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3468003f40 0x7f3468004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:09:26.005 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:25.997+0000 7f347959a640 1 --2- 172.21.15.104:0/734218020 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f3468004d50 0x7f34680051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:09:26.006 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:25.997+0000 7f347959a640 1 -- 172.21.15.104:0/734218020 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f347c0b02a0 con 0x7f3468003f40 2024-03-28T14:09:26.006 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:25.997+0000 7f347959a640 1 -- 172.21.15.104:0/734218020 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f347c0b2d30 con 0x7f3468004d50 2024-03-28T14:09:26.006 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:25.997+0000 7f347959a640 1 -- 172.21.15.104:0/734218020 --> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] -- mon_getmap magic: 0 -- 0x7f346800ba20 con 0x7f347c0732a0 2024-03-28T14:09:26.006 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:25.997+0000 7f3479d9b640 1 --2- 172.21.15.104:0/734218020 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3468003f40 0x7f3468004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:09:26.006 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:25.997+0000 7f347ad9d640 1 --2- 172.21.15.104:0/734218020 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f3468004d50 0x7f34680051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:09:26.006 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:25.997+0000 7f3479d9b640 1 -- 172.21.15.104:0/734218020 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f347c0732a0 msgr2=0x7f34680039c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T14:09:26.006 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:25.997+0000 7f3479d9b640 1 --2- 172.21.15.104:0/734218020 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f347c0732a0 0x7f34680039c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:09:26.006 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:25.997+0000 7f3479d9b640 1 -- 172.21.15.104:0/734218020 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f3468004d50 msgr2=0x7f34680051d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T14:09:26.006 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:25.997+0000 7f3479d9b640 1 --2- 172.21.15.104:0/734218020 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f3468004d50 0x7f34680051d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:09:26.007 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:25.997+0000 7f3479d9b640 1 -- 172.21.15.104:0/734218020 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3468003530 con 0x7f3468003f40 2024-03-28T14:09:26.010 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:26.001+0000 7f3479d9b640 1 --2- 172.21.15.104:0/734218020 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3468003f40 0x7f3468004810 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7f3464009ab0 tx=0x7f346401ca20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=31b5299192a42ed8 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:09:26.010 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:26.001+0000 7f347959a640 1 -- 172.21.15.104:0/734218020 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34640b2070 con 0x7f3468003f40 2024-03-28T14:09:26.010 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:26.001+0000 7f347959a640 1 -- 172.21.15.104:0/734218020 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f34640a55d0 con 0x7f3468003f40 2024-03-28T14:09:26.010 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:26.001+0000 7f347959a640 1 -- 172.21.15.104:0/734218020 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34640096b0 con 0x7f3468003f40 2024-03-28T14:09:26.010 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:26.001+0000 7f34809f3640 1 -- 172.21.15.104:0/734218020 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3468003f40 msgr2=0x7f3468004810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:09:26.011 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:26.001+0000 7f34809f3640 1 --2- 172.21.15.104:0/734218020 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3468003f40 0x7f3468004810 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7f3464009ab0 tx=0x7f346401ca20 comp rx=0 tx=0).stop 2024-03-28T14:09:26.011 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:26.001+0000 7f347ad9d640 1 -- 172.21.15.104:0/734218020 reap_dead start 2024-03-28T14:09:26.011 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:26.001+0000 7f34809f3640 1 -- 172.21.15.104:0/734218020 shutdown_connections 2024-03-28T14:09:26.011 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:26.001+0000 7f34809f3640 1 -- 172.21.15.104:0/734218020 >> 172.21.15.104:0/734218020 conn(0x7f347c0b58a0 msgr2=0x7f347c0b5cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T14:09:26.011 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:26.001+0000 7f34809f3640 1 -- 172.21.15.104:0/734218020 shutdown_connections 2024-03-28T14:09:26.011 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:26.001+0000 7f34809f3640 1 -- 172.21.15.104:0/734218020 wait complete. 2024-03-28T14:09:26.011 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:26.001+0000 7f34809f3640 1 Processor -- start 2024-03-28T14:09:26.011 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:26.001+0000 7f34809f3640 1 -- start start 2024-03-28T14:09:26.012 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:26.001+0000 7f34809f3640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f347c13a010 0x7f347c1de530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:09:26.012 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:26.001+0000 7f34809f3640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f347c142f70 0x7f347c072390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:09:26.012 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:26.001+0000 7f34809f3640 1 --2- >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f347c13a3f0 0x7f347c142240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:09:26.012 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:26.001+0000 7f34809f3640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f347c1507f0 con 0x7f347c13a010 2024-03-28T14:09:26.012 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:26.001+0000 7f34809f3640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f347c150670 con 0x7f347c142f70 2024-03-28T14:09:26.012 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:26.001+0000 7f34809f3640 1 -- --> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] -- mon_getmap magic: 0 -- 0x7f347c150970 con 0x7f347c13a3f0 2024-03-28T14:09:26.013 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:26.001+0000 7f347a59c640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f347c13a010 0x7f347c1de530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:09:26.013 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:26.001+0000 7f347a59c640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f347c13a010 0x7f347c1de530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:56808/0 (socket says 172.21.15.104:56808) 2024-03-28T14:09:26.013 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:26.001+0000 7f347a59c640 1 -- 172.21.15.104:0/2115324745 learned_addr learned my addr 172.21.15.104:0/2115324745 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-03-28T14:09:26.013 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:26.001+0000 7f347ad9d640 1 --2- 172.21.15.104:0/2115324745 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f347c13a3f0 0x7f347c142240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:09:26.013 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:26.001+0000 7f3479d9b640 1 --2- 172.21.15.104:0/2115324745 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f347c142f70 0x7f347c072390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:09:26.013 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:26.001+0000 7f347a59c640 1 -- 172.21.15.104:0/2115324745 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f347c13a3f0 msgr2=0x7f347c142240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:09:26.013 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:26.001+0000 7f347a59c640 1 --2- 172.21.15.104:0/2115324745 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f347c13a3f0 0x7f347c142240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:09:26.014 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:26.001+0000 7f347a59c640 1 -- 172.21.15.104:0/2115324745 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f347c142f70 msgr2=0x7f347c072390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:09:26.014 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:26.001+0000 7f347a59c640 1 --2- 172.21.15.104:0/2115324745 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f347c142f70 0x7f347c072390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:09:26.015 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:26.001+0000 7f347a59c640 1 -- 172.21.15.104:0/2115324745 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f347c070c50 con 0x7f347c13a010 2024-03-28T14:09:26.015 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:26.001+0000 7f347a59c640 1 --2- 172.21.15.104:0/2115324745 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f347c13a010 0x7f347c1de530 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7f346c00ca70 tx=0x7f346c00cf30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:09:26.015 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:26.001+0000 7f3472ffd640 1 -- 172.21.15.104:0/2115324745 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f346c018070 con 0x7f347c13a010 2024-03-28T14:09:26.015 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:26.001+0000 7f3472ffd640 1 -- 172.21.15.104:0/2115324745 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f346c004cc0 con 0x7f347c13a010 2024-03-28T14:09:26.015 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:26.001+0000 7f34809f3640 1 -- 172.21.15.104:0/2115324745 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f347c070f30 con 0x7f347c13a010 2024-03-28T14:09:26.015 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:26.001+0000 7f3472ffd640 1 -- 172.21.15.104:0/2115324745 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f346c007310 con 0x7f347c13a010 2024-03-28T14:09:26.015 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:26.005+0000 7f34809f3640 1 -- 172.21.15.104:0/2115324745 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f347c0728d0 con 0x7f347c13a010 2024-03-28T14:09:26.015 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:26.005+0000 7f34809f3640 1 -- 172.21.15.104:0/2115324745 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f347c0687c0 con 0x7f347c13a010 2024-03-28T14:09:26.016 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:26.005+0000 7f3472ffd640 1 -- 172.21.15.104:0/2115324745 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) ==== 87803+0+0 (secure 0 0 0) 0x7f346c01b070 con 0x7f347c13a010 2024-03-28T14:09:26.016 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:26.005+0000 7f3472ffd640 1 --2- 172.21.15.104:0/2115324745 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f345c067fc0 0x7f345c06a480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:09:26.017 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:26.005+0000 7f3472ffd640 1 -- 172.21.15.104:0/2115324745 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(33..33 src has 1..33) ==== 4943+0+0 (secure 0 0 0) 0x7f346c085a40 con 0x7f347c13a010 2024-03-28T14:09:26.017 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:26.005+0000 7f3479d9b640 1 --2- 172.21.15.104:0/2115324745 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f345c067fc0 0x7f345c06a480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:09:26.017 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:26.005+0000 7f3479d9b640 1 --2- 172.21.15.104:0/2115324745 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f345c067fc0 0x7f345c06a480 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f34640af000 tx=0x7f3464008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:09:26.017 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:26.009+0000 7f3472ffd640 1 -- 172.21.15.104:0/2115324745 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179117 (secure 0 0 0) 0x7f346c056560 con 0x7f347c13a010 2024-03-28T14:09:26.166 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:26.161+0000 7f34809f3640 1 -- 172.21.15.104:0/2115324745 --> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f347c1449a0 con 0x7f345c067fc0 2024-03-28T14:09:26.168 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:26.165+0000 7f3472ffd640 1 -- 172.21.15.104:0/2115324745 <== mgr.4100 v2:172.21.15.171:6832/1522752412 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+13354 (secure 0 0 0) 0x7f347c1449a0 con 0x7f345c067fc0 2024-03-28T14:09:26.168 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-03-28T14:09:26.169 INFO:teuthology.orchestra.run.smithi104.stderr:dumped all 2024-03-28T14:09:26.170 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:26.165+0000 7f34809f3640 1 -- 172.21.15.104:0/2115324745 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f345c067fc0 msgr2=0x7f345c06a480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:09:26.171 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:26.165+0000 7f34809f3640 1 --2- 172.21.15.104:0/2115324745 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f345c067fc0 0x7f345c06a480 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f34640af000 tx=0x7f3464008040 comp rx=0 tx=0).stop 2024-03-28T14:09:26.171 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:26.165+0000 7f34809f3640 1 -- 172.21.15.104:0/2115324745 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f347c13a010 msgr2=0x7f347c1de530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:09:26.171 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:26.165+0000 7f34809f3640 1 --2- 172.21.15.104:0/2115324745 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f347c13a010 0x7f347c1de530 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7f346c00ca70 tx=0x7f346c00cf30 comp rx=0 tx=0).stop 2024-03-28T14:09:26.171 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:26.165+0000 7f34809f3640 1 -- 172.21.15.104:0/2115324745 shutdown_connections 2024-03-28T14:09:26.171 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:26.165+0000 7f34809f3640 1 --2- 172.21.15.104:0/2115324745 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f347c13a3f0 0x7f347c142240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:09:26.172 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:26.165+0000 7f34809f3640 1 --2- 172.21.15.104:0/2115324745 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f347c142f70 0x7f347c072390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:09:26.172 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:26.165+0000 7f34809f3640 1 --2- 172.21.15.104:0/2115324745 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f345c067fc0 0x7f345c06a480 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:09:26.172 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:26.165+0000 7f34809f3640 1 --2- 172.21.15.104:0/2115324745 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f347c13a010 0x7f347c1de530 unknown :-1 s=CLOSED pgs=486 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:09:26.172 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:26.165+0000 7f34809f3640 1 -- 172.21.15.104:0/2115324745 >> 172.21.15.104:0/2115324745 conn(0x7f347c0b58a0 msgr2=0x7f347c13b410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T14:09:26.172 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:26.165+0000 7f34809f3640 1 -- 172.21.15.104:0/2115324745 shutdown_connections 2024-03-28T14:09:26.172 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:26.165+0000 7f34809f3640 1 -- 172.21.15.104:0/2115324745 wait complete. 2024-03-28T14:09:26.185 INFO:teuthology.orchestra.run.smithi104.stdout:{"pg_ready":true,"pg_map":{"version":76,"stamp":"2024-03-28T14:09:25.605463+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":119,"ondisk_log_size":119,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":2,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":2,"kb":754974720,"kb_used":220292,"kb_used_data":4136,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754754428,"statfs":{"total":773094113280,"available":772868534272,"internally_reserved":0,"allocated":4235264,"data_stored":2351392,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":221236785},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003171"},"pg_stats":[{"pgid":"1.0","version":"33'119","reported_seq":203,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-28T14:09:05.712985+0000","last_change":"2024-03-28T14:09:05.713066+0000","last_active":"2024-03-28T14:09:05.712985+0000","last_peered":"2024-03-28T14:09:05.712985+0000","last_clean":"2024-03-28T14:09:05.712985+0000","last_became_active":"2024-03-28T14:07:49.233895+0000","last_became_peered":"2024-03-28T14:07:49.233895+0000","last_unstale":"2024-03-28T14:09:05.712985+0000","last_undegraded":"2024-03-28T14:09:05.712985+0000","last_fullsized":"2024-03-28T14:09:05.712985+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'119","last_scrub_stamp":"2024-03-28T14:09:05.712789+0000","last_deep_scrub":"33'119","last_deep_scrub_stamp":"2024-03-28T14:09:05.712789+0000","last_clean_scrub_stamp":"2024-03-28T14:09:05.712789+0000","objects_scrubbed":2,"log_size":119,"log_dups_size":0,"ondisk_log_size":119,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:37:50.551827+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":119,"ondisk_log_size":119,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574871,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27968,"kb_used_data":952,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343872,"statfs":{"total":96636764160,"available":96608124928,"internally_reserved":0,"allocated":974848,"data_stored":736700,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574872,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27396,"kb_used_data":372,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344444,"statfs":{"total":96636764160,"available":96608710656,"internally_reserved":0,"allocated":380928,"data_stored":146332,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574870,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27388,"kb_used_data":372,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344452,"statfs":{"total":96636764160,"available":96608718848,"internally_reserved":0,"allocated":380928,"data_stored":146332,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574871,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27396,"kb_used_data":372,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344444,"statfs":{"total":96636764160,"available":96608710656,"internally_reserved":0,"allocated":380928,"data_stored":146332,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574873,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27392,"kb_used_data":372,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344448,"statfs":{"total":96636764160,"available":96608714752,"internally_reserved":0,"allocated":380928,"data_stored":146332,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574871,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27392,"kb_used_data":372,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344448,"statfs":{"total":96636764160,"available":96608714752,"internally_reserved":0,"allocated":380928,"data_stored":146332,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574869,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27392,"kb_used_data":372,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344448,"statfs":{"total":96636764160,"available":96608714752,"internally_reserved":0,"allocated":380928,"data_stored":146332,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574872,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27968,"kb_used_data":952,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343872,"statfs":{"total":96636764160,"available":96608124928,"internally_reserved":0,"allocated":974848,"data_stored":736700,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-28T14:09:26.186 DEBUG:teuthology.orchestra.run.smithi104:> sudo ceph --cluster ceph config set global mon_health_to_clog false 2024-03-28T14:09:26.295 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:26.289+0000 7f5bc8559640 1 Processor -- start 2024-03-28T14:09:26.295 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:26.289+0000 7f5bc8559640 1 -- start start 2024-03-28T14:09:26.296 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:26.289+0000 7f5bc8559640 1 --2- >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f5bc00b4aa0 0x7f5bc00af660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:09:26.296 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:26.289+0000 7f5bc8559640 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f5bc0147bf0 0x7f5bc00b22d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:09:26.297 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:26.289+0000 7f5bc8559640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5bc0148030 con 0x7f5bc00b2810 2024-03-28T14:09:26.297 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:26.289+0000 7f5bc8559640 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 -- 0x7f5bc00b4e80 con 0x7f5bc0147bf0 2024-03-28T14:09:26.297 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:26.289+0000 7f5bc8559640 1 -- --> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] -- mon_getmap magic: 0 -- 0x7f5bc00b0890 con 0x7f5bc00b4aa0 2024-03-28T14:09:26.297 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:26.289+0000 7f5bc5af4640 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f5bc0147bf0 0x7f5bc00b22d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:09:26.297 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:26.293+0000 7f5bc5af4640 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f5bc0147bf0 0x7f5bc00b22d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:56812/0 (socket says 172.21.15.104:56812) 2024-03-28T14:09:26.297 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:26.293+0000 7f5bc5af4640 1 -- 172.21.15.104:0/2844395856 learned_addr learned my addr 172.21.15.104:0/2844395856 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-03-28T14:09:26.297 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:26.293+0000 7f5bc62f5640 1 --2- 172.21.15.104:0/2844395856 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f5bc00b4aa0 0x7f5bc00af660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:09:26.297 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:26.293+0000 7f5bc5af4640 1 -- 172.21.15.104:0/2844395856 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f5bc00b4aa0 msgr2=0x7f5bc00af660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T14:09:26.298 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:26.293+0000 7f5bc5af4640 1 --2- 172.21.15.104:0/2844395856 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f5bc00b4aa0 0x7f5bc00af660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:09:26.298 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:26.293+0000 7f5bc5af4640 1 -- 172.21.15.104:0/2844395856 >> v1:172.21.15.171:6789/0 conn(0x7f5bc00b2810 legacy=0x7f5bc006c690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:09:26.298 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:26.293+0000 7f5bc5af4640 1 -- 172.21.15.104:0/2844395856 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5bc00ae0a0 con 0x7f5bc0147bf0 2024-03-28T14:09:26.298 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:26.293+0000 7f5bc62f5640 1 --2- 172.21.15.104:0/2844395856 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f5bc00b4aa0 0x7f5bc00af660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T14:09:26.299 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:26.293+0000 7f5bc5af4640 1 --2- 172.21.15.104:0/2844395856 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5bc0147bf0 0x7f5bc00b22d0 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7f5bbc009870 tx=0x7f5bbc01c4d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b2d1cf668dca2bbe server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:09:26.299 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:26.293+0000 7f5bc52f3640 1 -- 172.21.15.104:0/2844395856 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5bbc0b1070 con 0x7f5bc0147bf0 2024-03-28T14:09:26.300 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:26.293+0000 7f5bc52f3640 1 -- 172.21.15.104:0/2844395856 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5bbc0a8050 con 0x7f5bc0147bf0 2024-03-28T14:09:26.300 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:26.293+0000 7f5bc52f3640 1 -- 172.21.15.104:0/2844395856 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5bbc0ac860 con 0x7f5bc0147bf0 2024-03-28T14:09:26.301 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:26.297+0000 7f5bc8559640 1 -- 172.21.15.104:0/2844395856 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5bc0147bf0 msgr2=0x7f5bc00b22d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:09:26.301 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:26.297+0000 7f5bc8559640 1 --2- 172.21.15.104:0/2844395856 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5bc0147bf0 0x7f5bc00b22d0 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7f5bbc009870 tx=0x7f5bbc01c4d0 comp rx=0 tx=0).stop 2024-03-28T14:09:26.301 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:26.297+0000 7f5bc8559640 1 -- 172.21.15.104:0/2844395856 shutdown_connections 2024-03-28T14:09:26.301 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:26.297+0000 7f5bc8559640 1 --2- 172.21.15.104:0/2844395856 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5bc0147bf0 0x7f5bc00b22d0 unknown :-1 s=CLOSED pgs=487 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:09:26.301 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:26.297+0000 7f5bc8559640 1 --2- 172.21.15.104:0/2844395856 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f5bc00b4aa0 0x7f5bc00af660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:09:26.301 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:26.297+0000 7f5bc8559640 1 -- 172.21.15.104:0/2844395856 >> 172.21.15.104:0/2844395856 conn(0x7f5bc00b8f00 msgr2=0x7f5bc00b9310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T14:09:26.301 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:26.297+0000 7f5bc8559640 1 -- 172.21.15.104:0/2844395856 shutdown_connections 2024-03-28T14:09:26.301 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:26.297+0000 7f5bc8559640 1 -- 172.21.15.104:0/2844395856 wait complete. 2024-03-28T14:09:26.304 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:26.297+0000 7f5bc8559640 1 Processor -- start 2024-03-28T14:09:26.304 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:26.297+0000 7f5bc8559640 1 -- start start 2024-03-28T14:09:26.304 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:26.297+0000 7f5bc8559640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5bc00b2810 0x7f5bc0140b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:09:26.304 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:26.297+0000 7f5bc8559640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f5bc00b4aa0 0x7f5bc01e0330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:09:26.304 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:26.297+0000 7f5bc8559640 1 --2- >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f5bc0147bf0 0x7f5bc014d490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:09:26.304 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:26.297+0000 7f5bc8559640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f5bc00aeb20 con 0x7f5bc00b2810 2024-03-28T14:09:26.304 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:26.297+0000 7f5bc8559640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f5bc00ae9a0 con 0x7f5bc00b4aa0 2024-03-28T14:09:26.304 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:26.297+0000 7f5bc8559640 1 -- --> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] -- mon_getmap magic: 0 -- 0x7f5bc0151f40 con 0x7f5bc0147bf0 2024-03-28T14:09:26.305 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:26.297+0000 7f5bc62f5640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5bc00b2810 0x7f5bc0140b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:09:26.305 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:26.297+0000 7f5bc62f5640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5bc00b2810 0x7f5bc0140b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:56826/0 (socket says 172.21.15.104:56826) 2024-03-28T14:09:26.305 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:26.297+0000 7f5bc62f5640 1 -- 172.21.15.104:0/334649832 learned_addr learned my addr 172.21.15.104:0/334649832 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-03-28T14:09:26.305 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:26.297+0000 7f5bc5af4640 1 --2- 172.21.15.104:0/334649832 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f5bc00b4aa0 0x7f5bc01e0330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:09:26.305 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:26.297+0000 7f5bc6af6640 1 --2- 172.21.15.104:0/334649832 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f5bc0147bf0 0x7f5bc014d490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:09:26.305 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:26.297+0000 7f5bc62f5640 1 -- 172.21.15.104:0/334649832 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f5bc0147bf0 msgr2=0x7f5bc014d490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:09:26.305 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:26.297+0000 7f5bc62f5640 1 --2- 172.21.15.104:0/334649832 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f5bc0147bf0 0x7f5bc014d490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:09:26.305 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:26.297+0000 7f5bc62f5640 1 -- 172.21.15.104:0/334649832 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f5bc00b4aa0 msgr2=0x7f5bc01e0330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:09:26.306 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:26.297+0000 7f5bc62f5640 1 --2- 172.21.15.104:0/334649832 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f5bc00b4aa0 0x7f5bc01e0330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:09:26.306 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:26.297+0000 7f5bc62f5640 1 -- 172.21.15.104:0/334649832 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5bc014df30 con 0x7f5bc00b2810 2024-03-28T14:09:26.306 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:26.297+0000 7f5bc62f5640 1 --2- 172.21.15.104:0/334649832 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5bc00b2810 0x7f5bc0140b00 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7f5bb000d800 tx=0x7f5bb000dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:09:26.307 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:26.297+0000 7f5bb6ffd640 1 -- 172.21.15.104:0/334649832 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5bb000b6c0 con 0x7f5bc00b2810 2024-03-28T14:09:26.307 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:26.297+0000 7f5bb6ffd640 1 -- 172.21.15.104:0/334649832 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5bb0019070 con 0x7f5bc00b2810 2024-03-28T14:09:26.307 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:26.297+0000 7f5bc8559640 1 -- 172.21.15.104:0/334649832 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5bc0070db0 con 0x7f5bc00b2810 2024-03-28T14:09:26.307 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:26.297+0000 7f5bb6ffd640 1 -- 172.21.15.104:0/334649832 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5bb000baa0 con 0x7f5bc00b2810 2024-03-28T14:09:26.307 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:26.297+0000 7f5bc8559640 1 -- 172.21.15.104:0/334649832 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5bc00ae4e0 con 0x7f5bc00b2810 2024-03-28T14:09:26.307 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:26.297+0000 7f5bc8559640 1 -- 172.21.15.104:0/334649832 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5bc00714c0 con 0x7f5bc00b2810 2024-03-28T14:09:26.308 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:26.301+0000 7f5bb6ffd640 1 -- 172.21.15.104:0/334649832 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) ==== 87803+0+0 (secure 0 0 0) 0x7f5bb000bc40 con 0x7f5bc00b2810 2024-03-28T14:09:26.308 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:26.301+0000 7f5bb6ffd640 1 --2- 172.21.15.104:0/334649832 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f5ba4067fc0 0x7f5ba406a480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:09:26.309 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:26.301+0000 7f5bb6ffd640 1 -- 172.21.15.104:0/334649832 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(33..33 src has 1..33) ==== 4943+0+0 (secure 0 0 0) 0x7f5bb0086640 con 0x7f5bc00b2810 2024-03-28T14:09:26.309 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:26.301+0000 7f5bc5af4640 1 --2- 172.21.15.104:0/334649832 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f5ba4067fc0 0x7f5ba406a480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:09:26.309 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:26.301+0000 7f5bc5af4640 1 --2- 172.21.15.104:0/334649832 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f5ba4067fc0 0x7f5ba406a480 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f5bbc0ae000 tx=0x7f5bbc008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:09:26.309 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:26.305+0000 7f5bb6ffd640 1 -- 172.21.15.104:0/334649832 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179117 (secure 0 0 0) 0x7f5bb00570e0 con 0x7f5bc00b2810 2024-03-28T14:09:26.469 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:26.465+0000 7f5bc8559640 1 -- 172.21.15.104:0/334649832 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command([{prefix=config set, name=mon_health_to_clog}] v 0) -- 0x7f5bc00b9550 con 0x7f5bc00b2810 2024-03-28T14:09:26.473 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:26.469+0000 7f5bb6ffd640 1 -- 172.21.15.104:0/334649832 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_health_to_clog}]=0 v12)=0 v12) ==== 123+0+0 (secure 0 0 0) 0x7f5bb00569e0 con 0x7f5bc00b2810 2024-03-28T14:09:26.474 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:26.469+0000 7f5bb6ffd640 1 -- 172.21.15.104:0/334649832 <== mon.0 v2:172.21.15.104:3300/0 8 ==== config(1 keys) ==== 35+0+0 (secure 0 0 0) 0x7f5bb005ad90 con 0x7f5bc00b2810 2024-03-28T14:09:26.477 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:26.473+0000 7f5bc8559640 1 -- 172.21.15.104:0/334649832 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f5ba4067fc0 msgr2=0x7f5ba406a480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:09:26.478 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:26.473+0000 7f5bc8559640 1 --2- 172.21.15.104:0/334649832 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f5ba4067fc0 0x7f5ba406a480 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f5bbc0ae000 tx=0x7f5bbc008040 comp rx=0 tx=0).stop 2024-03-28T14:09:26.478 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:26.473+0000 7f5bc8559640 1 -- 172.21.15.104:0/334649832 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5bc00b2810 msgr2=0x7f5bc0140b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:09:26.478 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:26.473+0000 7f5bc8559640 1 --2- 172.21.15.104:0/334649832 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5bc00b2810 0x7f5bc0140b00 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7f5bb000d800 tx=0x7f5bb000dcc0 comp rx=0 tx=0).stop 2024-03-28T14:09:26.478 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:26.473+0000 7f5bc8559640 1 -- 172.21.15.104:0/334649832 shutdown_connections 2024-03-28T14:09:26.478 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:26.473+0000 7f5bc8559640 1 --2- 172.21.15.104:0/334649832 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f5bc0147bf0 0x7f5bc014d490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:09:26.478 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:26.473+0000 7f5bc8559640 1 --2- 172.21.15.104:0/334649832 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f5bc00b4aa0 0x7f5bc01e0330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:09:26.478 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:26.473+0000 7f5bc8559640 1 --2- 172.21.15.104:0/334649832 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f5ba4067fc0 0x7f5ba406a480 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:09:26.479 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:26.473+0000 7f5bc8559640 1 --2- 172.21.15.104:0/334649832 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5bc00b2810 0x7f5bc0140b00 unknown :-1 s=CLOSED pgs=488 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:09:26.479 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:26.473+0000 7f5bc8559640 1 -- 172.21.15.104:0/334649832 >> 172.21.15.104:0/334649832 conn(0x7f5bc00b8f00 msgr2=0x7f5bc00b1f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T14:09:26.479 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:26.473+0000 7f5bc8559640 1 -- 172.21.15.104:0/334649832 shutdown_connections 2024-03-28T14:09:26.479 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:26.473+0000 7f5bc8559640 1 -- 172.21.15.104:0/334649832 wait complete. 2024-03-28T14:09:26.492 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-03-28T14:09:26.492 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T14:09:26.601 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:26.597+0000 7f25aa5fc640 1 Processor -- start 2024-03-28T14:09:26.601 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:26.597+0000 7f25aa5fc640 1 -- start start 2024-03-28T14:09:26.603 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:26.597+0000 7f25aa5fc640 1 --2- >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f25a402c460 0x7f25a40b4fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:09:26.603 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:26.597+0000 7f25aa5fc640 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f25a4145210 0x7f25a4141540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:09:26.603 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:26.597+0000 7f25aa5fc640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f25a40b7d80 con 0x7f25a40b7940 2024-03-28T14:09:26.603 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:26.597+0000 7f25aa5fc640 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 -- 0x7f25a402c840 con 0x7f25a4145210 2024-03-28T14:09:26.603 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:26.597+0000 7f25aa5fc640 1 -- --> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] -- mon_getmap magic: 0 -- 0x7f25a40b5690 con 0x7f25a402c460 2024-03-28T14:09:26.603 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:26.597+0000 7f25a8b99640 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f25a4145210 0x7f25a4141540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:09:26.603 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:26.597+0000 7f25a37fe640 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f25a40b7940 0x7f25a4144b00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:48818/0 (socket says 172.21.15.104:48818) 2024-03-28T14:09:26.604 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:26.597+0000 7f25a37fe640 1 -- 172.21.15.104:0/526709940 learned_addr learned my addr 172.21.15.104:0/526709940 (peer_addr_for_me v1:172.21.15.104:0/0) 2024-03-28T14:09:26.604 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:26.597+0000 7f25a3fff640 1 --2- 172.21.15.104:0/526709940 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f25a402c460 0x7f25a40b4fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:09:26.604 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:26.597+0000 7f25a8b99640 1 -- 172.21.15.104:0/526709940 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f25a402c460 msgr2=0x7f25a40b4fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T14:09:26.604 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:26.597+0000 7f25a8b99640 1 --2- 172.21.15.104:0/526709940 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f25a402c460 0x7f25a40b4fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:09:26.604 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:26.597+0000 7f25a8b99640 1 -- 172.21.15.104:0/526709940 >> v1:172.21.15.104:6789/0 conn(0x7f25a40b7940 legacy=0x7f25a4144b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:09:26.604 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:26.597+0000 7f25a8b99640 1 -- 172.21.15.104:0/526709940 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f25a41445e0 con 0x7f25a4145210 2024-03-28T14:09:26.605 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:26.601+0000 7f25a8b99640 1 --2- 172.21.15.104:0/526709940 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f25a4145210 0x7f25a4141540 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7f2594009d90 tx=0x7f259401f150 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7deb154bb1dbd560 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:09:26.605 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:26.601+0000 7f25a2ffd640 1 -- 172.21.15.104:0/526709940 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25940b4070 con 0x7f25a4145210 2024-03-28T14:09:26.606 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:26.601+0000 7f25a2ffd640 1 -- 172.21.15.104:0/526709940 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(1 keys) ==== 35+0+0 (secure 0 0 0) 0x7f25940af600 con 0x7f25a4145210 2024-03-28T14:09:26.606 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:26.601+0000 7f25a2ffd640 1 -- 172.21.15.104:0/526709940 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25940af9f0 con 0x7f25a4145210 2024-03-28T14:09:26.606 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:26.601+0000 7f25aa5fc640 1 -- 172.21.15.104:0/526709940 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f25a4145210 msgr2=0x7f25a4141540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:09:26.606 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:26.601+0000 7f25aa5fc640 1 --2- 172.21.15.104:0/526709940 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f25a4145210 0x7f25a4141540 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7f2594009d90 tx=0x7f259401f150 comp rx=0 tx=0).stop 2024-03-28T14:09:26.606 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:26.601+0000 7f25aa5fc640 1 -- 172.21.15.104:0/526709940 shutdown_connections 2024-03-28T14:09:26.606 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:26.601+0000 7f25aa5fc640 1 --2- 172.21.15.104:0/526709940 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f25a4145210 0x7f25a4141540 unknown :-1 s=CLOSED pgs=490 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:09:26.606 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:26.601+0000 7f25aa5fc640 1 --2- 172.21.15.104:0/526709940 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f25a402c460 0x7f25a40b4fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:09:26.606 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:26.601+0000 7f25aa5fc640 1 -- 172.21.15.104:0/526709940 >> 172.21.15.104:0/526709940 conn(0x7f25a40b9660 msgr2=0x7f25a40b9a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T14:09:26.607 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:26.601+0000 7f25aa5fc640 1 -- 172.21.15.104:0/526709940 shutdown_connections 2024-03-28T14:09:26.607 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:26.601+0000 7f25aa5fc640 1 -- 172.21.15.104:0/526709940 wait complete. 2024-03-28T14:09:26.609 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:26.601+0000 7f25aa5fc640 1 Processor -- start 2024-03-28T14:09:26.609 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:26.601+0000 7f25aa5fc640 1 -- start start 2024-03-28T14:09:26.609 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:26.601+0000 7f25aa5fc640 1 --2- >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f25a402c460 0x7f25a4098280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:09:26.609 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:26.601+0000 7f25aa5fc640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f25a40b7940 0x7f25a41dcf50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:09:26.609 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:26.601+0000 7f25aa5fc640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f25a4145210 0x7f25a41dd490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:09:26.609 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:26.601+0000 7f25aa5fc640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f25a4141c00 con 0x7f25a40b7940 2024-03-28T14:09:26.610 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:26.601+0000 7f25aa5fc640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f25a4141a80 con 0x7f25a4145210 2024-03-28T14:09:26.610 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:26.601+0000 7f25aa5fc640 1 -- --> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] -- mon_getmap magic: 0 -- 0x7f25a4141d80 con 0x7f25a402c460 2024-03-28T14:09:26.610 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:26.601+0000 7f25a37fe640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f25a40b7940 0x7f25a41dcf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:09:26.610 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:26.601+0000 7f25a37fe640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f25a40b7940 0x7f25a41dcf50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:47894/0 (socket says 172.21.15.104:47894) 2024-03-28T14:09:26.610 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:26.601+0000 7f25a37fe640 1 -- 172.21.15.104:0/250350896 learned_addr learned my addr 172.21.15.104:0/250350896 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-03-28T14:09:26.610 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:26.601+0000 7f25a3fff640 1 --2- 172.21.15.104:0/250350896 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f25a402c460 0x7f25a4098280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:09:26.610 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:26.601+0000 7f25a37fe640 1 -- 172.21.15.104:0/250350896 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f25a402c460 msgr2=0x7f25a4098280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:09:26.610 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:26.601+0000 7f25a37fe640 1 --2- 172.21.15.104:0/250350896 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f25a402c460 0x7f25a4098280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:09:26.611 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:26.601+0000 7f25a37fe640 1 -- 172.21.15.104:0/250350896 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f25a4145210 msgr2=0x7f25a41dd490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:09:26.611 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:26.601+0000 7f25a37fe640 1 --2- 172.21.15.104:0/250350896 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f25a4145210 0x7f25a41dd490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:09:26.611 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:26.601+0000 7f25a37fe640 1 -- 172.21.15.104:0/250350896 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f25a408cca0 con 0x7f25a40b7940 2024-03-28T14:09:26.611 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:26.601+0000 7f25a37fe640 1 --2- 172.21.15.104:0/250350896 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f25a40b7940 0x7f25a41dcf50 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7f259800c420 tx=0x7f259800c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:09:26.611 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:26.601+0000 7f25a0ff9640 1 -- 172.21.15.104:0/250350896 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2598017070 con 0x7f25a40b7940 2024-03-28T14:09:26.611 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:26.601+0000 7f25a0ff9640 1 -- 172.21.15.104:0/250350896 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(1 keys) ==== 35+0+0 (secure 0 0 0) 0x7f2598005cf0 con 0x7f25a40b7940 2024-03-28T14:09:26.611 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:26.601+0000 7f25aa5fc640 1 -- 172.21.15.104:0/250350896 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f25a40731b0 con 0x7f25a40b7940 2024-03-28T14:09:26.611 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:26.601+0000 7f25a0ff9640 1 -- 172.21.15.104:0/250350896 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2598007310 con 0x7f25a40b7940 2024-03-28T14:09:26.612 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:26.601+0000 7f25aa5fc640 1 -- 172.21.15.104:0/250350896 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f25a408c9c0 con 0x7f25a40b7940 2024-03-28T14:09:26.612 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:26.605+0000 7f25aa5fc640 1 -- 172.21.15.104:0/250350896 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f25a40687c0 con 0x7f25a40b7940 2024-03-28T14:09:26.612 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:26.605+0000 7f25a0ff9640 1 -- 172.21.15.104:0/250350896 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) ==== 87803+0+0 (secure 0 0 0) 0x7f259801a070 con 0x7f25a40b7940 2024-03-28T14:09:26.612 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:26.605+0000 7f25a0ff9640 1 --2- 172.21.15.104:0/250350896 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f2590067fc0 0x7f259006a480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T14:09:26.613 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:26.605+0000 7f25a0ff9640 1 -- 172.21.15.104:0/250350896 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(33..33 src has 1..33) ==== 4943+0+0 (secure 0 0 0) 0x7f25980856c0 con 0x7f25a40b7940 2024-03-28T14:09:26.613 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:26.605+0000 7f25a3fff640 1 --2- 172.21.15.104:0/250350896 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f2590067fc0 0x7f259006a480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T14:09:26.613 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:26.605+0000 7f25a3fff640 1 --2- 172.21.15.104:0/250350896 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f2590067fc0 0x7f259006a480 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f259c009790 tx=0x7f259c008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T14:09:26.614 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:26.609+0000 7f25a0ff9640 1 -- 172.21.15.104:0/250350896 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179117 (secure 0 0 0) 0x7f25980560d0 con 0x7f25a40b7940 2024-03-28T14:09:26.796 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:26.793+0000 7f25aa5fc640 1 -- 172.21.15.104:0/250350896 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f25a406e6a0 con 0x7f25a40b7940 2024-03-28T14:09:26.797 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:26.793+0000 7f25a0ff9640 1 -- 172.21.15.104:0/250350896 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 16 v16) ==== 94+0+2225 (secure 0 0 0) 0x7f25980559d0 con 0x7f25a40b7940 2024-03-28T14:09:26.797 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-03-28T14:09:26.798 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":16,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4550,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.171:6835/3418793776","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6834","nonce":3418793776},{"type":"v1","addr":"172.21.15.171:6835","nonce":3418793776}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":5268,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6833/1356174773","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":1356174773},{"type":"v1","addr":"172.21.15.104:6833","nonce":1356174773}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":16}],"filesystems":[]} 2024-03-28T14:09:26.798 INFO:teuthology.orchestra.run.smithi104.stderr:dumped fsmap epoch 16 2024-03-28T14:09:26.799 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:26.793+0000 7f25aa5fc640 1 -- 172.21.15.104:0/250350896 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f2590067fc0 msgr2=0x7f259006a480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:09:26.799 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:26.793+0000 7f25aa5fc640 1 --2- 172.21.15.104:0/250350896 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f2590067fc0 0x7f259006a480 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f259c009790 tx=0x7f259c008040 comp rx=0 tx=0).stop 2024-03-28T14:09:26.800 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:26.793+0000 7f25aa5fc640 1 -- 172.21.15.104:0/250350896 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f25a40b7940 msgr2=0x7f25a41dcf50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T14:09:26.800 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:26.793+0000 7f25aa5fc640 1 --2- 172.21.15.104:0/250350896 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f25a40b7940 0x7f25a41dcf50 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7f259800c420 tx=0x7f259800c8e0 comp rx=0 tx=0).stop 2024-03-28T14:09:26.800 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:26.793+0000 7f25aa5fc640 1 -- 172.21.15.104:0/250350896 shutdown_connections 2024-03-28T14:09:26.800 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:26.793+0000 7f25aa5fc640 1 --2- 172.21.15.104:0/250350896 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f25a4145210 0x7f25a41dd490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:09:26.800 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:26.793+0000 7f25aa5fc640 1 --2- 172.21.15.104:0/250350896 >> [v2:172.21.15.171:6832/1522752412,v1:172.21.15.171:6833/1522752412] conn(0x7f2590067fc0 0x7f259006a480 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:09:26.800 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:26.793+0000 7f25aa5fc640 1 --2- 172.21.15.104:0/250350896 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f25a40b7940 0x7f25a41dcf50 unknown :-1 s=CLOSED pgs=491 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:09:26.800 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:26.793+0000 7f25aa5fc640 1 --2- 172.21.15.104:0/250350896 >> [v2:172.21.15.171:3301/0,v1:172.21.15.171:6790/0] conn(0x7f25a402c460 0x7f25a4098280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T14:09:26.800 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:26.793+0000 7f25aa5fc640 1 -- 172.21.15.104:0/250350896 >> 172.21.15.104:0/250350896 conn(0x7f25a40b9660 msgr2=0x7f25a40b2a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T14:09:26.801 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:26.793+0000 7f25aa5fc640 1 -- 172.21.15.104:0/250350896 shutdown_connections 2024-03-28T14:09:26.801 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-28T14:09:26.793+0000 7f25aa5fc640 1 -- 172.21.15.104:0/250350896 wait complete. 2024-03-28T14:09:26.814 DEBUG:tasks.cephfs.filesystem:already dead... 2024-03-28T14:09:26.814 INFO:teuthology.misc:Shutting down mds daemons... 2024-03-28T14:09:26.815 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-03-28T14:09:26.815 INFO:teuthology.orchestra.run:waiting for 300 2024-03-28T14:09:26.824 INFO:tasks.ceph.mds.a:Stopped 2024-03-28T14:09:26.825 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-03-28T14:09:26.825 INFO:teuthology.orchestra.run:waiting for 300 2024-03-28T14:09:26.838 INFO:tasks.ceph.mds.b:Stopped 2024-03-28T14:09:26.838 INFO:teuthology.misc:Shutting down osd daemons... 2024-03-28T14:09:26.838 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2024-03-28T14:09:26.839 INFO:teuthology.orchestra.run:waiting for 300 2024-03-28T14:09:29.304 DEBUG:teuthology.orchestra.run.smithi104:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-28T14:09:29.306 DEBUG:teuthology.orchestra.run.smithi171:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-28T14:09:30.197 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~0s 2024-03-28T14:09:32.940 INFO:tasks.ceph.osd.0:Stopped 2024-03-28T14:09:32.940 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2024-03-28T14:09:32.940 INFO:teuthology.orchestra.run:waiting for 300 2024-03-28T14:09:33.007 INFO:tasks.ceph.osd.1:Stopped 2024-03-28T14:09:33.007 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2024-03-28T14:09:33.007 INFO:teuthology.orchestra.run:waiting for 300 2024-03-28T14:09:33.093 INFO:tasks.ceph.osd.2:Stopped 2024-03-28T14:09:33.093 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2024-03-28T14:09:33.093 INFO:teuthology.orchestra.run:waiting for 300 2024-03-28T14:09:33.181 INFO:tasks.ceph.osd.3:Stopped 2024-03-28T14:09:33.182 DEBUG:tasks.ceph.osd.4:waiting for process to exit 2024-03-28T14:09:33.182 INFO:teuthology.orchestra.run:waiting for 300 2024-03-28T14:09:33.269 INFO:tasks.ceph.osd.4:Stopped 2024-03-28T14:09:33.269 DEBUG:tasks.ceph.osd.5:waiting for process to exit 2024-03-28T14:09:33.269 INFO:teuthology.orchestra.run:waiting for 300 2024-03-28T14:09:33.346 INFO:tasks.ceph.osd.5:Stopped 2024-03-28T14:09:33.346 DEBUG:tasks.ceph.osd.6:waiting for process to exit 2024-03-28T14:09:33.347 INFO:teuthology.orchestra.run:waiting for 300 2024-03-28T14:09:35.803 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~0s 2024-03-28T14:09:35.803 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 has been restored 2024-03-28T14:09:39.448 INFO:tasks.ceph.osd.6:Stopped 2024-03-28T14:09:39.448 DEBUG:tasks.ceph.osd.7:waiting for process to exit 2024-03-28T14:09:39.449 INFO:teuthology.orchestra.run:waiting for 300 2024-03-28T14:09:39.495 INFO:tasks.ceph.osd.7:Stopped 2024-03-28T14:09:39.495 INFO:teuthology.misc:Shutting down mgr daemons... 2024-03-28T14:09:39.496 DEBUG:tasks.ceph.mgr.y:waiting for process to exit 2024-03-28T14:09:39.496 INFO:teuthology.orchestra.run:waiting for 300 2024-03-28T14:09:39.523 INFO:tasks.ceph.mgr.y:Stopped 2024-03-28T14:09:39.523 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2024-03-28T14:09:39.524 INFO:teuthology.orchestra.run:waiting for 300 2024-03-28T14:09:39.552 INFO:tasks.ceph.mgr.x:Stopped 2024-03-28T14:09:39.552 INFO:teuthology.misc:Shutting down mon daemons... 2024-03-28T14:09:39.553 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-03-28T14:09:39.553 INFO:teuthology.orchestra.run:waiting for 300 2024-03-28T14:09:39.584 INFO:tasks.ceph.mon.a:Stopped 2024-03-28T14:09:39.584 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-03-28T14:09:39.584 INFO:teuthology.orchestra.run:waiting for 300 2024-03-28T14:09:39.608 INFO:tasks.ceph.mon.b:Stopped 2024-03-28T14:09:39.608 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-03-28T14:09:39.608 INFO:teuthology.orchestra.run:waiting for 300 2024-03-28T14:09:39.665 INFO:tasks.ceph.mon.c:Stopped 2024-03-28T14:09:39.665 INFO:tasks.ceph:Checking cluster log for badness... 2024-03-28T14:09:39.665 DEBUG:teuthology.orchestra.run.smithi104:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v FS_DEGRADED | egrep -v FS_INLINE_DATA_DEPRECATED | egrep -v FS_WITH_FAILED_MDS | egrep -v MDS_ALL_DOWN | egrep -v MDS_DAMAGE | egrep -v MDS_DEGRADED | egrep -v MDS_FAILED | egrep -v MDS_INSUFFICIENT_STANDBY | egrep -v MDS_UP_LESS_THAN_MAX | egrep -v 'filesystem is online with fewer MDS than max_mds' | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'overall HEALTH_' | egrep -v 'Replacing daemon' | egrep -v 'deprecated feature inline_data' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | egrep -v PG_AVAILABILITY | egrep -v PG_DEGRADED | egrep -v 'Reduced data availability' | head -n 1 2024-03-28T14:09:39.683 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi104.front.sepia.ceph.com 2024-03-28T14:09:39.683 DEBUG:teuthology.orchestra.run.smithi104:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2024-03-28T14:09:39.926 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi104.front.sepia.ceph.com 2024-03-28T14:09:39.926 DEBUG:teuthology.orchestra.run.smithi104:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2024-03-28T14:09:40.080 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi104.front.sepia.ceph.com 2024-03-28T14:09:40.081 DEBUG:teuthology.orchestra.run.smithi104:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2024-03-28T14:09:40.240 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-3 on ubuntu@smithi104.front.sepia.ceph.com 2024-03-28T14:09:40.240 DEBUG:teuthology.orchestra.run.smithi104:> sync && sudo umount -f /var/lib/ceph/osd/ceph-3 2024-03-28T14:09:40.408 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-4 on ubuntu@smithi171.front.sepia.ceph.com 2024-03-28T14:09:40.408 DEBUG:teuthology.orchestra.run.smithi171:> sync && sudo umount -f /var/lib/ceph/osd/ceph-4 2024-03-28T14:09:40.646 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-5 on ubuntu@smithi171.front.sepia.ceph.com 2024-03-28T14:09:40.646 DEBUG:teuthology.orchestra.run.smithi171:> sync && sudo umount -f /var/lib/ceph/osd/ceph-5 2024-03-28T14:09:40.805 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 has been restored 2024-03-28T14:09:40.819 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-6 on ubuntu@smithi171.front.sepia.ceph.com 2024-03-28T14:09:40.820 DEBUG:teuthology.orchestra.run.smithi171:> sync && sudo umount -f /var/lib/ceph/osd/ceph-6 2024-03-28T14:09:40.987 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-7 on ubuntu@smithi171.front.sepia.ceph.com 2024-03-28T14:09:40.988 DEBUG:teuthology.orchestra.run.smithi171:> sync && sudo umount -f /var/lib/ceph/osd/ceph-7 2024-03-28T14:09:41.161 INFO:tasks.ceph:Archiving mon data... 2024-03-28T14:09:41.162 DEBUG:teuthology.misc:Transferring archived files from smithi104:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/pdonnell-2024-03-28_07:14:07-fs-wip-batrick-testing-20240327.230800-distro-default-smithi/7627919/data/mon.a.tgz 2024-03-28T14:09:41.163 DEBUG:teuthology.orchestra.run.smithi104:> mktemp 2024-03-28T14:09:41.168 INFO:teuthology.orchestra.run.smithi104.stdout:/tmp/tmp.iDj2OFImXX 2024-03-28T14:09:41.169 DEBUG:teuthology.orchestra.run.smithi104:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.iDj2OFImXX 2024-03-28T14:09:41.283 DEBUG:teuthology.orchestra.run.smithi104:> sudo chmod 0666 /tmp/tmp.iDj2OFImXX 2024-03-28T14:09:41.345 DEBUG:teuthology.orchestra.remote:smithi104:/tmp/tmp.iDj2OFImXX is 272KB 2024-03-28T14:09:41.406 DEBUG:teuthology.orchestra.run.smithi104:> rm -fr /tmp/tmp.iDj2OFImXX 2024-03-28T14:09:41.411 DEBUG:teuthology.misc:Transferring archived files from smithi171:/var/lib/ceph/mon/ceph-b to /home/teuthworker/archive/pdonnell-2024-03-28_07:14:07-fs-wip-batrick-testing-20240327.230800-distro-default-smithi/7627919/data/mon.b.tgz 2024-03-28T14:09:41.411 DEBUG:teuthology.orchestra.run.smithi171:> mktemp 2024-03-28T14:09:41.417 INFO:teuthology.orchestra.run.smithi171.stdout:/tmp/tmp.3QIiGCR3rL 2024-03-28T14:09:41.418 DEBUG:teuthology.orchestra.run.smithi171:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-b -- . > /tmp/tmp.3QIiGCR3rL 2024-03-28T14:09:41.544 DEBUG:teuthology.orchestra.run.smithi171:> sudo chmod 0666 /tmp/tmp.3QIiGCR3rL 2024-03-28T14:09:41.605 DEBUG:teuthology.orchestra.remote:smithi171:/tmp/tmp.3QIiGCR3rL is 367KB 2024-03-28T14:09:41.667 DEBUG:teuthology.orchestra.run.smithi171:> rm -fr /tmp/tmp.3QIiGCR3rL 2024-03-28T14:09:41.672 DEBUG:teuthology.misc:Transferring archived files from smithi171:/var/lib/ceph/mon/ceph-c to /home/teuthworker/archive/pdonnell-2024-03-28_07:14:07-fs-wip-batrick-testing-20240327.230800-distro-default-smithi/7627919/data/mon.c.tgz 2024-03-28T14:09:41.672 DEBUG:teuthology.orchestra.run.smithi171:> mktemp 2024-03-28T14:09:41.724 INFO:teuthology.orchestra.run.smithi171.stdout:/tmp/tmp.GS2kdDeGpj 2024-03-28T14:09:41.725 DEBUG:teuthology.orchestra.run.smithi171:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-c -- . > /tmp/tmp.GS2kdDeGpj 2024-03-28T14:09:41.860 DEBUG:teuthology.orchestra.run.smithi171:> sudo chmod 0666 /tmp/tmp.GS2kdDeGpj 2024-03-28T14:09:41.936 DEBUG:teuthology.orchestra.remote:smithi171:/tmp/tmp.GS2kdDeGpj is 367KB 2024-03-28T14:09:41.988 DEBUG:teuthology.orchestra.run.smithi171:> rm -fr /tmp/tmp.GS2kdDeGpj 2024-03-28T14:09:42.003 INFO:tasks.ceph:Cleaning ceph cluster... 2024-03-28T14:09:42.003 DEBUG:teuthology.orchestra.run.smithi104:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-03-28T14:09:42.007 DEBUG:teuthology.orchestra.run.smithi171:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-03-28T14:09:42.070 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-03-28T14:09:42.111 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-03-28T14:09:42.111 INFO:tasks.ceph:Archiving crash dumps... 2024-03-28T14:09:42.120 DEBUG:teuthology.misc:Transferring archived files from smithi104:/var/lib/ceph/crash to /home/teuthworker/archive/pdonnell-2024-03-28_07:14:07-fs-wip-batrick-testing-20240327.230800-distro-default-smithi/7627919/remote/smithi104/crash 2024-03-28T14:09:42.122 DEBUG:teuthology.orchestra.run.smithi104:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-03-28T14:09:42.137 DEBUG:teuthology.misc:Transferring archived files from smithi171:/var/lib/ceph/crash to /home/teuthworker/archive/pdonnell-2024-03-28_07:14:07-fs-wip-batrick-testing-20240327.230800-distro-default-smithi/7627919/remote/smithi171/crash 2024-03-28T14:09:42.139 DEBUG:teuthology.orchestra.run.smithi171:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-03-28T14:09:42.170 INFO:tasks.ceph:Shutting down logrotate 2024-03-28T14:09:42.173 DEBUG:teuthology.orchestra.run.smithi104:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-28T14:09:42.186 DEBUG:teuthology.orchestra.run.smithi171:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-28T14:09:42.260 DEBUG:teuthology.orchestra.run.smithi104:> sudo rm /etc/logrotate.d/ceph-test.conf 2024-03-28T14:09:42.287 DEBUG:teuthology.orchestra.run.smithi171:> sudo rm /etc/logrotate.d/ceph-test.conf 2024-03-28T14:09:42.316 INFO:tasks.ceph:Compressing logs... 2024-03-28T14:09:42.317 DEBUG:teuthology.orchestra.run.smithi104:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-28T14:09:42.330 DEBUG:teuthology.orchestra.run.smithi171:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-28T14:09:42.354 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35988.log 2024-03-28T14:09:42.355 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36705.log 2024-03-28T14:09:42.355 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.35988.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37851.log 2024-03-28T14:09:42.355 INFO:teuthology.orchestra.run.smithi104.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.35988.log.gz 2024-03-28T14:09:42.355 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.36705.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37324.log 2024-03-28T14:09:42.359 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.37851.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33529.log 2024-03-28T14:09:42.359 INFO:teuthology.orchestra.run.smithi104.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.36705.log.gz 2024-03-28T14:09:42.359 INFO:teuthology.orchestra.run.smithi104.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.37851.log.gz 2024-03-28T14:09:42.360 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.37324.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37272.log 2024-03-28T14:09:42.360 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.33529.log: gzip -5 --verbose -- /var/log/ceph/ceph-mgr.y.log 2024-03-28T14:09:42.360 INFO:teuthology.orchestra.run.smithi104.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.37324.log.gz 2024-03-28T14:09:42.360 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.37272.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.33529.log.gz 2024-03-28T14:09:42.360 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38033.log 2024-03-28T14:09:42.360 INFO:teuthology.orchestra.run.smithi104.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.37272.log.gz 2024-03-28T14:09:42.360 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35944.log 2024-03-28T14:09:42.360 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-mgr.y.log: /var/log/ceph/ceph-client.admin.38033.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36679.log 2024-03-28T14:09:42.361 INFO:teuthology.orchestra.run.smithi104.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.38033.log.gz 2024-03-28T14:09:42.361 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.35944.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35599.log 2024-03-28T14:09:42.361 INFO:teuthology.orchestra.run.smithi104.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.35944.log.gz 2024-03-28T14:09:42.361 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.36679.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35520.log 2024-03-28T14:09:42.361 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.35599.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.36679.log.gz 2024-03-28T14:09:42.361 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34495.log 2024-03-28T14:09:42.361 INFO:teuthology.orchestra.run.smithi104.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.35599.log.gz 2024-03-28T14:09:42.361 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.35520.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35199.log 2024-03-28T14:09:42.407 INFO:teuthology.orchestra.run.smithi171.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27679.log 2024-03-28T14:09:42.408 INFO:teuthology.orchestra.run.smithi171.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.7.log 2024-03-28T14:09:42.408 INFO:teuthology.orchestra.run.smithi171.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.6.log 2024-03-28T14:09:42.408 INFO:teuthology.orchestra.run.smithi171.stderr:/var/log/ceph/ceph-client.admin.27679.log: /var/log/ceph/ceph-osd.7.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.5.log 2024-03-28T14:09:42.408 INFO:teuthology.orchestra.run.smithi171.stderr:/var/log/ceph/ceph-osd.6.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27751.log 2024-03-28T14:09:42.408 INFO:teuthology.orchestra.run.smithi171.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.27679.log.gz 2024-03-28T14:09:42.409 INFO:teuthology.orchestra.run.smithi171.stderr:/var/log/ceph/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2024-03-28T14:09:42.409 INFO:teuthology.orchestra.run.smithi171.stderr:/var/log/ceph/ceph-client.admin.27751.log: gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-03-28T14:09:42.409 INFO:teuthology.orchestra.run.smithi171.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.27751.log.gz 2024-03-28T14:09:42.409 INFO:teuthology.orchestra.run.smithi171.stderr:/var/log/ceph/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27727.log 2024-03-28T14:09:42.409 INFO:teuthology.orchestra.run.smithi171.stderr:/var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27655.log 2024-03-28T14:09:42.409 INFO:teuthology.orchestra.run.smithi171.stderr:/var/log/ceph/ceph-client.admin.27727.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.c.log 2024-03-28T14:09:42.410 INFO:teuthology.orchestra.run.smithi171.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.27727.log.gz/var/log/ceph/ceph-client.admin.27655.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27631.log 2024-03-28T14:09:42.410 INFO:teuthology.orchestra.run.smithi171.stderr: 2024-03-28T14:09:42.410 INFO:teuthology.orchestra.run.smithi171.stderr: 91.8% 87.1% -- replaced with /var/log/ceph/ceph-client.admin.27655.log.gz 2024-03-28T14:09:42.410 INFO:teuthology.orchestra.run.smithi171.stderr:/var/log/ceph/ceph-mon.c.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-03-28T14:09:42.410 INFO:teuthology.orchestra.run.smithi171.stderr: -- replaced with /var/log/ceph/ceph.log.gz 2024-03-28T14:09:42.410 INFO:teuthology.orchestra.run.smithi171.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mds.b.log 2024-03-28T14:09:42.411 INFO:teuthology.orchestra.run.smithi171.stderr:/var/log/ceph/ceph-client.admin.27631.log: /var/log/ceph/ceph.audit.log: 88.7%gzip -5 --verbose -- /var/log/ceph/ceph-mon.b.log 2024-03-28T14:09:42.411 INFO:teuthology.orchestra.run.smithi171.stderr: -- replaced with /var/log/ceph/ceph-client.admin.27631.log.gz 2024-03-28T14:09:42.411 INFO:teuthology.orchestra.run.smithi171.stderr:/var/log/ceph/ceph-mds.b.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.4.log 2024-03-28T14:09:42.411 INFO:teuthology.orchestra.run.smithi171.stderr:/var/log/ceph/ceph-mon.b.log: 91.7% -- replaced with /var/log/ceph/ceph-mds.b.log.gz 2024-03-28T14:09:42.411 INFO:teuthology.orchestra.run.smithi171.stderr: 93.7% -- replaced with /var/log/ceph/ceph.audit.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27703.log 2024-03-28T14:09:42.411 INFO:teuthology.orchestra.run.smithi171.stderr: 2024-03-28T14:09:42.412 INFO:teuthology.orchestra.run.smithi171.stderr:/var/log/ceph/ceph-osd.4.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27583.log 2024-03-28T14:09:42.412 INFO:teuthology.orchestra.run.smithi171.stderr:/var/log/ceph/ceph-client.admin.27703.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.27703.log.gz 2024-03-28T14:09:42.412 INFO:teuthology.orchestra.run.smithi171.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27607.log 2024-03-28T14:09:42.412 INFO:teuthology.orchestra.run.smithi171.stderr:/var/log/ceph/ceph-client.admin.27583.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.27583.log.gz 2024-03-28T14:09:42.412 INFO:teuthology.orchestra.run.smithi171.stderr:/var/log/ceph/ceph-client.admin.27607.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.27607.log.gz 2024-03-28T14:09:42.424 INFO:teuthology.orchestra.run.smithi104.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.35520.log.gz 2024-03-28T14:09:42.424 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.34495.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36549.log 2024-03-28T14:09:42.424 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.35199.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.0.log 2024-03-28T14:09:42.424 INFO:teuthology.orchestra.run.smithi104.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.34495.log.gz 2024-03-28T14:09:42.424 INFO:teuthology.orchestra.run.smithi104.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.35199.log.gz 2024-03-28T14:09:42.425 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.36549.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36376.log 2024-03-28T14:09:42.425 INFO:teuthology.orchestra.run.smithi104.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.36549.log.gz 2024-03-28T14:09:42.425 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35468.log 2024-03-28T14:09:42.425 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.36376.log: 92.5% -- replaced with /var/log/ceph/ceph-mgr.y.log.gz 2024-03-28T14:09:42.425 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37539.log 2024-03-28T14:09:42.425 INFO:teuthology.orchestra.run.smithi104.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.36376.log.gz 2024-03-28T14:09:42.426 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.35468.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37421.log 2024-03-28T14:09:42.426 INFO:teuthology.orchestra.run.smithi104.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.35468.log.gz 2024-03-28T14:09:42.426 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.37539.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37090.log 2024-03-28T14:09:42.426 INFO:teuthology.orchestra.run.smithi104.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.37539.log.gz 2024-03-28T14:09:42.426 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.37421.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33358.log 2024-03-28T14:09:42.426 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.37090.log: gzip 87.9% -5 -- replaced with /var/log/ceph/ceph-client.admin.37421.log.gz --verbose 2024-03-28T14:09:42.427 INFO:teuthology.orchestra.run.smithi104.stderr: -- /var/log/ceph/ceph-client.admin.35017.log 2024-03-28T14:09:42.427 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.33358.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.37090.log.gz 2024-03-28T14:09:42.427 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37168.log 2024-03-28T14:09:42.427 INFO:teuthology.orchestra.run.smithi104.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.33358.log.gz 2024-03-28T14:09:42.427 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.35017.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36220.log 2024-03-28T14:09:42.427 INFO:teuthology.orchestra.run.smithi104.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.35017.log.gz 2024-03-28T14:09:42.427 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.37168.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37669.log 2024-03-28T14:09:42.428 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.36220.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34913.log 2024-03-28T14:09:42.428 INFO:teuthology.orchestra.run.smithi104.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.37168.log.gz 2024-03-28T14:09:42.428 INFO:teuthology.orchestra.run.smithi104.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.36220.log.gz 2024-03-28T14:09:42.428 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.37669.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37194.log 2024-03-28T14:09:42.428 INFO:teuthology.orchestra.run.smithi104.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.37669.log.gz 2024-03-28T14:09:42.428 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.34913.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36627.log 2024-03-28T14:09:42.429 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.37194.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33815.log 2024-03-28T14:09:42.429 INFO:teuthology.orchestra.run.smithi104.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.34913.log.gz 2024-03-28T14:09:42.429 INFO:teuthology.orchestra.run.smithi104.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.37194.log.gz 2024-03-28T14:09:42.429 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.36627.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35147.log 2024-03-28T14:09:42.429 INFO:teuthology.orchestra.run.smithi104.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.36627.log.gz 2024-03-28T14:09:42.429 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34939.log 2024-03-28T14:09:42.429 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.33815.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35866.log 2024-03-28T14:09:42.430 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.35147.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.33815.log.gz 2024-03-28T14:09:42.430 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.34939.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34547.log 2024-03-28T14:09:42.430 INFO:teuthology.orchestra.run.smithi104.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.34939.log.gz 2024-03-28T14:09:42.430 INFO:teuthology.orchestra.run.smithi104.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.35147.log.gz 2024-03-28T14:09:42.430 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.35866.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33611.log 2024-03-28T14:09:42.430 INFO:teuthology.orchestra.run.smithi104.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.35866.log.gz 2024-03-28T14:09:42.431 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.34547.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33466.log 2024-03-28T14:09:42.431 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.33611.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35442.log 2024-03-28T14:09:42.431 INFO:teuthology.orchestra.run.smithi104.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.34547.log.gz 2024-03-28T14:09:42.431 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.33466.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.33611.log.gz 2024-03-28T14:09:42.431 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37142.log 2024-03-28T14:09:42.431 INFO:teuthology.orchestra.run.smithi104.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.33466.log.gz 2024-03-28T14:09:42.431 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.35442.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33715.log 2024-03-28T14:09:42.432 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.37142.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35069.log 2024-03-28T14:09:42.432 INFO:teuthology.orchestra.run.smithi104.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.35442.log.gz 2024-03-28T14:09:42.432 INFO:teuthology.orchestra.run.smithi104.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.37142.log.gz 2024-03-28T14:09:42.432 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.33715.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35121.log 2024-03-28T14:09:42.432 INFO:teuthology.orchestra.run.smithi104.stderr: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.33715.log.gz 2024-03-28T14:09:42.432 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.35069.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36246.log 2024-03-28T14:09:42.432 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.35121.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.2.log 2024-03-28T14:09:42.432 INFO:teuthology.orchestra.run.smithi104.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.35069.log.gz 2024-03-28T14:09:42.433 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.36246.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.35121.log.gz 2024-03-28T14:09:42.433 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35840.log 2024-03-28T14:09:42.433 INFO:teuthology.orchestra.run.smithi104.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.36246.log.gz 2024-03-28T14:09:42.433 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36575.log 2024-03-28T14:09:42.433 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.35840.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33585.log 2024-03-28T14:09:42.433 INFO:teuthology.orchestra.run.smithi104.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.35840.log.gz 2024-03-28T14:09:42.433 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.36575.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37721.log 2024-03-28T14:09:42.433 INFO:teuthology.orchestra.run.smithi104.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.36575.log.gz 2024-03-28T14:09:42.433 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37116.log 2024-03-28T14:09:42.434 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.33585.log: /var/log/ceph/ceph-client.admin.37721.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33254.log 2024-03-28T14:09:42.434 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.37116.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.33585.log.gz 2024-03-28T14:09:42.434 INFO:teuthology.orchestra.run.smithi104.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.37721.log.gz 2024-03-28T14:09:42.434 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35173.log 2024-03-28T14:09:42.434 INFO:teuthology.orchestra.run.smithi104.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.37116.log.gz 2024-03-28T14:09:42.434 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.33254.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34261.log 2024-03-28T14:09:42.434 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.35173.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.33254.log.gz 2024-03-28T14:09:42.435 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35786.log 2024-03-28T14:09:42.435 INFO:teuthology.orchestra.run.smithi104.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.35173.log.gz 2024-03-28T14:09:42.435 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.34261.log: gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.22714.log 2024-03-28T14:09:42.435 INFO:teuthology.orchestra.run.smithi104.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.34261.log.gz 2024-03-28T14:09:42.435 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35494.log 2024-03-28T14:09:42.435 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.35786.log: /var/log/ceph/ceph.tmp-client.admin.22714.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33637.log 2024-03-28T14:09:42.435 INFO:teuthology.orchestra.run.smithi104.stderr: 0.0% -- replaced with /var/log/ceph/ceph.tmp-client.admin.22714.log.gz 2024-03-28T14:09:42.435 INFO:teuthology.orchestra.run.smithi104.stderr: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.35786.log.gz 2024-03-28T14:09:42.436 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.35494.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34235.log 2024-03-28T14:09:42.436 INFO:teuthology.orchestra.run.smithi104.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.35494.log.gz 2024-03-28T14:09:42.436 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.33637.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.a.log 2024-03-28T14:09:42.436 INFO:teuthology.orchestra.run.smithi104.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.33637.log.gz 2024-03-28T14:09:42.436 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.34235.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33332.log 2024-03-28T14:09:42.436 INFO:teuthology.orchestra.run.smithi104.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.34235.log.gz 2024-03-28T14:09:42.436 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35812.log 2024-03-28T14:09:42.436 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.33332.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37591.log 2024-03-28T14:09:42.437 INFO:teuthology.orchestra.run.smithi104.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.33332.log.gz 2024-03-28T14:09:42.437 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33280.log 2024-03-28T14:09:42.437 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.35812.log: /var/log/ceph/ceph-client.admin.37591.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34001.log 2024-03-28T14:09:42.437 INFO:teuthology.orchestra.run.smithi104.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.35812.log.gz 2024-03-28T14:09:42.437 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.33280.log: gzip 89.1% -- replaced with /var/log/ceph/ceph-client.admin.37591.log.gz -5 2024-03-28T14:09:42.437 INFO:teuthology.orchestra.run.smithi104.stderr: --verbose -- /var/log/ceph/ceph-client.admin.35760.log 2024-03-28T14:09:42.437 INFO:teuthology.orchestra.run.smithi104.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.33280.log.gz 2024-03-28T14:09:42.438 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.34001.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36298.log 2024-03-28T14:09:42.438 INFO:teuthology.orchestra.run.smithi104.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.34001.log.gz 2024-03-28T14:09:42.438 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.35760.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35355.log 2024-03-28T14:09:42.438 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.36298.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35277.log 2024-03-28T14:09:42.438 INFO:teuthology.orchestra.run.smithi104.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.35760.log.gz 2024-03-28T14:09:42.438 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.35355.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.36298.log.gz 2024-03-28T14:09:42.438 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37695.log 2024-03-28T14:09:42.438 INFO:teuthology.orchestra.run.smithi104.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.35355.log.gz 2024-03-28T14:09:42.439 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.35277.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38059.log 2024-03-28T14:09:42.439 INFO:teuthology.orchestra.run.smithi104.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.35277.log.gz 2024-03-28T14:09:42.439 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37617.log 2024-03-28T14:09:42.439 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.37695.log: /var/log/ceph/ceph-client.admin.38059.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36070.log 2024-03-28T14:09:42.439 INFO:teuthology.orchestra.run.smithi104.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.37695.log.gz 2024-03-28T14:09:42.439 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.37617.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.38059.log.gz 2024-03-28T14:09:42.439 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35383.log 2024-03-28T14:09:42.440 INFO:teuthology.orchestra.run.smithi104.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.37617.log.gz 2024-03-28T14:09:42.440 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.36070.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35251.log 2024-03-28T14:09:42.440 INFO:teuthology.orchestra.run.smithi104.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.36070.log.gz 2024-03-28T14:09:42.440 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35892.log 2024-03-28T14:09:42.440 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.35383.log: /var/log/ceph/ceph-client.admin.35251.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34887.log 2024-03-28T14:09:42.440 INFO:teuthology.orchestra.run.smithi104.stderr: 88.6% 88.9% -- replaced with /var/log/ceph/ceph-client.admin.35251.log.gz 2024-03-28T14:09:42.440 INFO:teuthology.orchestra.run.smithi104.stderr: -- replaced with /var/log/ceph/ceph-client.admin.35383.log.gz 2024-03-28T14:09:42.441 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.35892.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36324.log 2024-03-28T14:09:42.441 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5/var/log/ceph/ceph-client.admin.34887.log: --verbose -- /var/log/ceph/ceph-client.admin.36194.log 2024-03-28T14:09:42.441 INFO:teuthology.orchestra.run.smithi104.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.35892.log.gz 2024-03-28T14:09:42.441 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.36324.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.34887.log.gz 2024-03-28T14:09:42.441 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34521.log 2024-03-28T14:09:42.441 INFO:teuthology.orchestra.run.smithi104.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.36324.log.gz 2024-03-28T14:09:42.442 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.36194.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36954.log 2024-03-28T14:09:42.442 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37981.log 2024-03-28T14:09:42.442 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.34521.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.36194.log.gz 2024-03-28T14:09:42.442 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.36954.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36980.log 2024-03-28T14:09:42.442 INFO:teuthology.orchestra.run.smithi104.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.34521.log.gz 2024-03-28T14:09:42.442 INFO:teuthology.orchestra.run.smithi104.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.36954.log.gz 2024-03-28T14:09:42.443 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.37981.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33384.log 2024-03-28T14:09:42.443 INFO:teuthology.orchestra.run.smithi104.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.37981.log.gz 2024-03-28T14:09:42.443 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.36980.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33810.log 2024-03-28T14:09:42.443 INFO:teuthology.orchestra.run.smithi104.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.36980.log.gz 2024-03-28T14:09:42.443 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34599.log 2024-03-28T14:09:42.443 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.33384.log: /var/log/ceph/ceph-client.admin.33810.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34417.log 2024-03-28T14:09:42.444 INFO:teuthology.orchestra.run.smithi104.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.33384.log.gz 2024-03-28T14:09:42.444 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.34599.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.33810.log.gz 2024-03-28T14:09:42.444 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33121.log 2024-03-28T14:09:42.444 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.34417.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34965.log 2024-03-28T14:09:42.444 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.33121.log: 88.7% 91.3% -- replaced with /var/log/ceph/ceph-client.admin.34417.log.gz -- replaced with /var/log/ceph/ceph-client.admin.34599.log.gz 2024-03-28T14:09:42.444 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-03-28T14:09:42.444 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33808.log 2024-03-28T14:09:42.445 INFO:teuthology.orchestra.run.smithi104.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.33121.log.gz 2024-03-28T14:09:42.445 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.34965.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36653.log 2024-03-28T14:09:42.445 INFO:teuthology.orchestra.run.smithi104.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.34965.log.gz 2024-03-28T14:09:42.445 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.33808.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.3.log 2024-03-28T14:09:42.445 INFO:teuthology.orchestra.run.smithi104.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.33808.log.gz 2024-03-28T14:09:42.445 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.36653.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34861.log 2024-03-28T14:09:42.446 INFO:teuthology.orchestra.run.smithi104.stderr:gzip 89.2% -5 -- replaced with /var/log/ceph/ceph-client.admin.36653.log.gz --verbose 2024-03-28T14:09:42.446 INFO:teuthology.orchestra.run.smithi104.stderr: -- /var/log/ceph/ceph-client.admin.38007.log 2024-03-28T14:09:42.446 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-osd.3.log: /var/log/ceph/ceph-client.admin.34861.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36015.log 2024-03-28T14:09:42.446 INFO:teuthology.orchestra.run.smithi104.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.34861.log.gz 2024-03-28T14:09:42.446 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.38007.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37799.log 2024-03-28T14:09:42.446 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35573.log 2024-03-28T14:09:42.447 INFO:teuthology.orchestra.run.smithi104.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.38007.log.gz 2024-03-28T14:09:42.447 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.36015.log: /var/log/ceph/ceph-client.admin.37799.log: 88.8%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35095.log 2024-03-28T14:09:42.447 INFO:teuthology.orchestra.run.smithi104.stderr: -- replaced with /var/log/ceph/ceph-client.admin.36015.log.gz 2024-03-28T14:09:42.447 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.35573.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34654.log 2024-03-28T14:09:42.447 INFO:teuthology.orchestra.run.smithi104.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.37799.log.gz 2024-03-28T14:09:42.447 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.35095.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37350.log 2024-03-28T14:09:42.448 INFO:teuthology.orchestra.run.smithi104.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.35573.log.gz 2024-03-28T14:09:42.448 INFO:teuthology.orchestra.run.smithi104.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.35095.log.gz 2024-03-28T14:09:42.448 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.34654.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.34654.log.gz 2024-03-28T14:09:42.448 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37063.log 2024-03-28T14:09:42.448 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.37350.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37447.log 2024-03-28T14:09:42.448 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.37063.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.37350.log.gz 2024-03-28T14:09:42.448 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.1.log 2024-03-28T14:09:42.449 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.37447.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.37063.log.gz 2024-03-28T14:09:42.449 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37473.log 2024-03-28T14:09:42.449 INFO:teuthology.orchestra.run.smithi104.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.37447.log.gz 2024-03-28T14:09:42.449 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37747.log 2024-03-28T14:09:42.449 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.37473.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37395.log 2024-03-28T14:09:42.449 INFO:teuthology.orchestra.run.smithi104.stderr: 88.2%/var/log/ceph/ceph-client.admin.37747.log: -- replaced with /var/log/ceph/ceph-client.admin.37473.log.gz 2024-03-28T14:09:42.450 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34835.log 2024-03-28T14:09:42.450 INFO:teuthology.orchestra.run.smithi104.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.37747.log.gz 2024-03-28T14:09:42.450 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.37395.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.37395.log.gz 2024-03-28T14:09:42.450 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37929.log 2024-03-28T14:09:42.450 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.34835.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36116.log 2024-03-28T14:09:42.450 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.37929.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.34835.log.gz 2024-03-28T14:09:42.451 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35734.log 2024-03-28T14:09:42.451 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33741.log 2024-03-28T14:09:42.451 INFO:teuthology.orchestra.run.smithi104.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.37929.log.gz/var/log/ceph/ceph-client.admin.36116.log: 2024-03-28T14:09:42.451 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.35734.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.36116.log.gz 2024-03-28T14:09:42.451 INFO:teuthology.orchestra.run.smithi104.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.35734.log.gz 2024-03-28T14:09:42.451 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37006.log 2024-03-28T14:09:42.452 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.33741.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33306.log 2024-03-28T14:09:42.452 INFO:teuthology.orchestra.run.smithi104.stderr: 87.8%/var/log/ceph/ceph-client.admin.37006.log: -- replaced with /var/log/ceph/ceph-client.admin.33741.log.gz 2024-03-28T14:09:42.452 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36928.log 2024-03-28T14:09:42.452 INFO:teuthology.orchestra.run.smithi104.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.37006.log.gz 2024-03-28T14:09:42.452 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.33306.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.33306.log.gz 2024-03-28T14:09:42.452 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33555.log 2024-03-28T14:09:42.452 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.36928.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36272.log 2024-03-28T14:09:42.453 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.33555.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.36928.log.gz 2024-03-28T14:09:42.453 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34313.log 2024-03-28T14:09:42.453 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37032.log 2024-03-28T14:09:42.453 INFO:teuthology.orchestra.run.smithi104.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.33555.log.gz 2024-03-28T14:09:42.453 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.36272.log: /var/log/ceph/ceph-client.admin.34313.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.36272.log.gz 2024-03-28T14:09:42.453 INFO:teuthology.orchestra.run.smithi104.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.34313.log.gz 2024-03-28T14:09:42.454 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35625.log 2024-03-28T14:09:42.454 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36350.log 2024-03-28T14:09:42.454 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.37032.log: /var/log/ceph/ceph-client.admin.35625.log: gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-03-28T14:09:42.454 INFO:teuthology.orchestra.run.smithi104.stderr: 89.3% 87.7% -- replaced with /var/log/ceph/ceph-client.admin.37032.log.gz 2024-03-28T14:09:42.454 INFO:teuthology.orchestra.run.smithi104.stderr: -- replaced with /var/log/ceph/ceph-client.admin.35625.log.gz/var/log/ceph/ceph-client.admin.36350.log: 2024-03-28T14:09:42.454 INFO:teuthology.orchestra.run.smithi104.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.36350.log.gz 2024-03-28T14:09:42.455 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34023.log 2024-03-28T14:09:42.455 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34991.log 2024-03-28T14:09:42.455 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.34023.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36432.log 2024-03-28T14:09:42.455 INFO:teuthology.orchestra.run.smithi104.stderr: 85.4% -- replaced with /var/log/ceph/ceph.log.gz 88.6% -- replaced with /var/log/ceph/ceph-client.admin.34023.log.gz 2024-03-28T14:09:42.455 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-03-28T14:09:42.455 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.34991.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.34991.log.gz 2024-03-28T14:09:42.455 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27479.log 2024-03-28T14:09:42.456 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34443.log 2024-03-28T14:09:42.456 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.36432.log: /var/log/ceph/ceph-client.admin.27479.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35682.log 2024-03-28T14:09:42.456 INFO:teuthology.orchestra.run.smithi104.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.36432.log.gz 2024-03-28T14:09:42.456 INFO:teuthology.orchestra.run.smithi104.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.27479.log.gz 2024-03-28T14:09:42.456 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.34443.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.34443.log.gz 2024-03-28T14:09:42.456 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.34782.log 2024-03-28T14:09:42.457 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.35682.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34010.log 2024-03-28T14:09:42.457 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.0.34782.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.35682.log.gz 2024-03-28T14:09:42.457 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33818.log 2024-03-28T14:09:42.457 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33767.log 2024-03-28T14:09:42.457 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.34010.log: /var/log/ceph/ceph-client.admin.33818.log: 89.5% -- replaced with /var/log/ceph/ceph-client.0.34782.log.gz 2024-03-28T14:09:42.457 INFO:teuthology.orchestra.run.smithi104.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.33818.log.gz 2024-03-28T14:09:42.457 INFO:teuthology.orchestra.run.smithi104.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.34010.log.gz 2024-03-28T14:09:42.458 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35651.log 2024-03-28T14:09:42.458 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.33767.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36731.log 2024-03-28T14:09:42.458 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.35651.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.33767.log.gz 2024-03-28T14:09:42.458 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33689.log 2024-03-28T14:09:42.458 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33816.log 2024-03-28T14:09:42.458 INFO:teuthology.orchestra.run.smithi104.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.35651.log.gz/var/log/ceph/ceph-client.admin.36731.log: 2024-03-28T14:09:42.459 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.33689.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.33689.log.gz 2024-03-28T14:09:42.459 INFO:teuthology.orchestra.run.smithi104.stderr: 91.1% -- replaced with /var/log/ceph/ceph-client.admin.36731.log.gz 2024-03-28T14:09:42.459 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34573.log 2024-03-28T14:09:42.459 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.33816.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37825.log 2024-03-28T14:09:42.459 INFO:teuthology.orchestra.run.smithi104.stderr: 88.3%/var/log/ceph/ceph-client.admin.34573.log: -- replaced with /var/log/ceph/ceph-client.admin.33816.log.gz 2024-03-28T14:09:42.459 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33438.log 2024-03-28T14:09:42.460 INFO:teuthology.orchestra.run.smithi104.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.34573.log.gz 2024-03-28T14:09:42.460 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.37825.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.37825.log.gz 2024-03-28T14:09:42.460 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38083.log 2024-03-28T14:09:42.460 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36493.log 2024-03-28T14:09:42.460 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.33438.log: /var/log/ceph/ceph-client.admin.38083.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33821.log 2024-03-28T14:09:42.460 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.36493.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.33438.log.gz 2024-03-28T14:09:42.460 INFO:teuthology.orchestra.run.smithi104.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.38083.log.gz 2024-03-28T14:09:42.461 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34365.log 2024-03-28T14:09:42.461 INFO:teuthology.orchestra.run.smithi104.stderr: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.36493.log.gz 2024-03-28T14:09:42.461 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35708.log 2024-03-28T14:09:42.461 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.33821.log: /var/log/ceph/ceph-client.admin.34365.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35225.log 2024-03-28T14:09:42.461 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.35708.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.34365.log.gz 89.1% -- replaced with /var/log/ceph/ceph-client.admin.33821.log.gz 2024-03-28T14:09:42.461 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-03-28T14:09:42.462 INFO:teuthology.orchestra.run.smithi104.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.35708.log.gz 2024-03-28T14:09:42.462 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35918.log 2024-03-28T14:09:42.462 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36898.log 2024-03-28T14:09:42.462 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.35225.log: gzip/var/log/ceph/ceph-client.admin.35918.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.27588.log 2024-03-28T14:09:42.462 INFO:teuthology.orchestra.run.smithi104.stderr: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.35225.log.gz 2024-03-28T14:09:42.462 INFO:teuthology.orchestra.run.smithi104.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.35918.log.gz 2024-03-28T14:09:42.462 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.36898.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.36898.log.gz 2024-03-28T14:09:42.463 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-03-28T14:09:42.464 INFO:teuthology.orchestra.run.smithi171.stderr: 91.8% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2024-03-28T14:09:42.482 INFO:teuthology.orchestra.run.smithi171.stderr: 92.1% -- replaced with /var/log/ceph/ceph-mon.b.log.gz 2024-03-28T14:09:42.484 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.27588.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38109.log 2024-03-28T14:09:42.484 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph.audit.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.27588.log.gz 2024-03-28T14:09:42.484 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33208.log 2024-03-28T14:09:42.484 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mds.a.log 2024-03-28T14:09:42.484 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.38109.log: /var/log/ceph/ceph-client.admin.33208.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.38109.log.gz 2024-03-28T14:09:42.485 INFO:teuthology.orchestra.run.smithi104.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.33208.log.gz 2024-03-28T14:09:42.485 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37903.log 2024-03-28T14:09:42.485 INFO:teuthology.orchestra.run.smithi104.stderr: 89.1% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-03-28T14:09:42.485 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-mds.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36041.log 2024-03-28T14:09:42.485 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37877.log 2024-03-28T14:09:42.486 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.37903.log: /var/log/ceph/ceph-client.admin.36041.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.37903.log.gz 2024-03-28T14:09:42.486 INFO:teuthology.orchestra.run.smithi104.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.36041.log.gz 2024-03-28T14:09:42.486 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34051.log 2024-03-28T14:09:42.486 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.37877.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.37877.log.gz 2024-03-28T14:09:42.486 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33809.log 2024-03-28T14:09:42.486 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36467.log 2024-03-28T14:09:42.486 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.34051.log: /var/log/ceph/ceph-client.admin.33809.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.34051.log.gz 2024-03-28T14:09:42.487 INFO:teuthology.orchestra.run.smithi104.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.33809.log.gz 2024-03-28T14:09:42.487 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37565.log 2024-03-28T14:09:42.487 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35547.log 2024-03-28T14:09:42.487 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.36467.log: /var/log/ceph/ceph-client.admin.37565.log: 88.0% 89.5% -- replaced with /var/log/ceph/ceph-client.admin.37565.log.gz -- replaced with /var/log/ceph/ceph-client.admin.36467.log.gz 2024-03-28T14:09:42.487 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-03-28T14:09:42.487 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36142.log 2024-03-28T14:09:42.488 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33817.log 2024-03-28T14:09:42.488 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.35547.log: /var/log/ceph/ceph-client.admin.36142.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.35547.log.gz 2024-03-28T14:09:42.488 INFO:teuthology.orchestra.run.smithi104.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.36142.log.gz 2024-03-28T14:09:42.488 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37643.log 2024-03-28T14:09:42.488 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.33817.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.33817.log.gz 2024-03-28T14:09:42.488 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35303.log 2024-03-28T14:09:42.488 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37513.log 2024-03-28T14:09:42.489 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.37643.log: /var/log/ceph/ceph-client.admin.35303.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.37643.log.gz 2024-03-28T14:09:42.489 INFO:teuthology.orchestra.run.smithi104.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.35303.log.gz 2024-03-28T14:09:42.489 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33178.log 2024-03-28T14:09:42.489 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.37513.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.37513.log.gz 2024-03-28T14:09:42.489 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34013.log 2024-03-28T14:09:42.489 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35043.log 2024-03-28T14:09:42.490 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.33178.log: /var/log/ceph/ceph-client.admin.34013.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.33178.log.gz 2024-03-28T14:09:42.490 INFO:teuthology.orchestra.run.smithi104.stderr: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.34013.log.gz 2024-03-28T14:09:42.490 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34024.log 2024-03-28T14:09:42.490 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34025.log 2024-03-28T14:09:42.490 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.35043.log: /var/log/ceph/ceph-client.admin.34024.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.35043.log.gz 2024-03-28T14:09:42.491 INFO:teuthology.orchestra.run.smithi104.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.34024.log.gz 2024-03-28T14:09:42.491 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36404.log 2024-03-28T14:09:42.491 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35416.log 2024-03-28T14:09:42.491 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.34025.log: /var/log/ceph/ceph-client.admin.36404.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.34025.log.gz 2024-03-28T14:09:42.491 INFO:teuthology.orchestra.run.smithi104.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.36404.log.gz 2024-03-28T14:09:42.491 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33503.log 2024-03-28T14:09:42.492 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34287.log 2024-03-28T14:09:42.492 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.35416.log: /var/log/ceph/ceph-client.admin.33503.log: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.35416.log.gz 2024-03-28T14:09:42.492 INFO:teuthology.orchestra.run.smithi104.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.33503.log.gz 2024-03-28T14:09:42.492 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34009.log 2024-03-28T14:09:42.492 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34391.log 2024-03-28T14:09:42.492 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.34287.log: /var/log/ceph/ceph-client.admin.34009.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.34287.log.gz 2024-03-28T14:09:42.493 INFO:teuthology.orchestra.run.smithi104.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.34009.log.gz 2024-03-28T14:09:42.493 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36601.log 2024-03-28T14:09:42.493 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37955.log 2024-03-28T14:09:42.493 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.34391.log: /var/log/ceph/ceph-client.admin.36601.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.34391.log.gz 2024-03-28T14:09:42.493 INFO:teuthology.orchestra.run.smithi104.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.36601.log.gz 2024-03-28T14:09:42.493 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36519.log 2024-03-28T14:09:42.494 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34339.log 2024-03-28T14:09:42.494 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.37955.log: /var/log/ceph/ceph-client.admin.36519.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.37955.log.gz 2024-03-28T14:09:42.494 INFO:teuthology.orchestra.run.smithi104.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.36519.log.gz 2024-03-28T14:09:42.494 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.36837.log 2024-03-28T14:09:42.494 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33663.log 2024-03-28T14:09:42.494 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.34339.log: /var/log/ceph/ceph-client.0.36837.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.34339.log.gz 2024-03-28T14:09:42.495 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37220.log 2024-03-28T14:09:42.495 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.33663.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.33663.log.gz 2024-03-28T14:09:42.495 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36781.log 2024-03-28T14:09:42.495 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.37220.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.37220.log.gz 2024-03-28T14:09:42.495 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34469.log 2024-03-28T14:09:42.495 INFO:teuthology.orchestra.run.smithi104.stderr: 93.3% -- replaced with /var/log/ceph/ceph-client.0.36837.log.gz 2024-03-28T14:09:42.495 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.36781.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.36781.log.gz 2024-03-28T14:09:42.496 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34209.log 2024-03-28T14:09:42.496 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27698.log 2024-03-28T14:09:42.496 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.34469.log: /var/log/ceph/ceph-client.admin.34209.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.34469.log.gz 2024-03-28T14:09:42.496 INFO:teuthology.orchestra.run.smithi104.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.34209.log.gz 2024-03-28T14:09:42.496 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36168.log 2024-03-28T14:09:42.496 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37246.log 2024-03-28T14:09:42.496 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.27698.log: /var/log/ceph/ceph-client.admin.36168.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.27698.log.gz 2024-03-28T14:09:42.496 INFO:teuthology.orchestra.run.smithi104.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.36168.log.gz 2024-03-28T14:09:42.496 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37773.log 2024-03-28T14:09:42.497 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.37246.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.37246.log.gz 2024-03-28T14:09:42.497 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35329.log 2024-03-28T14:09:42.497 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37298.log 2024-03-28T14:09:42.497 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.37773.log: /var/log/ceph/ceph-client.admin.35329.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.37773.log.gz 2024-03-28T14:09:42.497 INFO:teuthology.orchestra.run.smithi104.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.35329.log.gz 2024-03-28T14:09:42.497 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33410.log 2024-03-28T14:09:42.497 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.37298.log: /var/log/ceph/ceph-client.admin.33410.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.37298.log.gz 2024-03-28T14:09:42.497 INFO:teuthology.orchestra.run.smithi104.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.33410.log.gz 2024-03-28T14:09:42.498 INFO:teuthology.orchestra.run.smithi104.stderr: 91.5% -- replaced with /var/log/ceph/ceph-mds.a.log.gz 2024-03-28T14:09:42.498 INFO:teuthology.orchestra.run.smithi104.stderr: 93.5% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2024-03-28T14:09:42.498 INFO:teuthology.orchestra.run.smithi104.stderr: 93.7% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2024-03-28T14:09:42.498 INFO:teuthology.orchestra.run.smithi104.stderr: 93.6% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2024-03-28T14:09:42.499 INFO:teuthology.orchestra.run.smithi171.stderr: 93.7% -- replaced with /var/log/ceph/ceph-osd.4.log.gz 2024-03-28T14:09:42.499 INFO:teuthology.orchestra.run.smithi171.stderr: 91.7% -- replaced with /var/log/ceph/ceph-mon.c.log.gz 2024-03-28T14:09:42.501 INFO:teuthology.orchestra.run.smithi171.stderr: 93.5% -- replaced with /var/log/ceph/ceph-osd.6.log.gz 2024-03-28T14:09:42.502 INFO:teuthology.orchestra.run.smithi171.stderr: 93.8% -- replaced with /var/log/ceph/ceph-osd.5.log.gz 2024-03-28T14:09:42.518 INFO:teuthology.orchestra.run.smithi104.stderr: 93.8% -- replaced with /var/log/ceph/ceph-osd.3.log.gz 2024-03-28T14:09:42.541 INFO:teuthology.orchestra.run.smithi171.stderr: 93.9% -- replaced with /var/log/ceph/ceph-osd.7.log.gz 2024-03-28T14:09:42.542 INFO:teuthology.orchestra.run.smithi171.stderr: 2024-03-28T14:09:42.542 INFO:teuthology.orchestra.run.smithi171.stderr:real 0m0.176s 2024-03-28T14:09:42.542 INFO:teuthology.orchestra.run.smithi171.stderr:user 0m0.672s 2024-03-28T14:09:42.542 INFO:teuthology.orchestra.run.smithi171.stderr:sys 0m0.050s 2024-03-28T14:09:42.552 INFO:teuthology.orchestra.run.smithi104.stderr: 91.2% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2024-03-28T14:09:42.554 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-03-28T14:09:42.554 INFO:teuthology.orchestra.run.smithi104.stderr:real 0m0.222s 2024-03-28T14:09:42.554 INFO:teuthology.orchestra.run.smithi104.stderr:user 0m0.830s 2024-03-28T14:09:42.555 INFO:teuthology.orchestra.run.smithi104.stderr:sys 0m0.098s 2024-03-28T14:09:42.555 INFO:tasks.ceph:Archiving logs... 2024-03-28T14:09:42.555 DEBUG:teuthology.misc:Transferring archived files from smithi104:/var/log/ceph to /home/teuthworker/archive/pdonnell-2024-03-28_07:14:07-fs-wip-batrick-testing-20240327.230800-distro-default-smithi/7627919/remote/smithi104/log 2024-03-28T14:09:42.556 DEBUG:teuthology.orchestra.run.smithi104:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-28T14:09:42.822 DEBUG:teuthology.misc:Transferring archived files from smithi171:/var/log/ceph to /home/teuthworker/archive/pdonnell-2024-03-28_07:14:07-fs-wip-batrick-testing-20240327.230800-distro-default-smithi/7627919/remote/smithi171/log 2024-03-28T14:09:42.823 DEBUG:teuthology.orchestra.run.smithi171:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-28T14:09:42.916 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-03-28T14:09:42.932 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-03-28T14:09:42.933 DEBUG:teuthology.orchestra.run.smithi104:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-28T14:09:42.935 DEBUG:teuthology.orchestra.run.smithi171:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-28T14:09:43.007 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-03-28T14:09:43.017 INFO:teuthology.task.clock:Checking final clock skew... 2024-03-28T14:09:43.018 DEBUG:teuthology.orchestra.run.smithi104:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-28T14:09:43.020 DEBUG:teuthology.orchestra.run.smithi171:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-28T14:09:43.031 INFO:teuthology.orchestra.run.smithi104.stdout: remote refid st t when poll reach delay offset jitter 2024-03-28T14:09:43.031 INFO:teuthology.orchestra.run.smithi104.stdout:============================================================================== 2024-03-28T14:09:43.031 INFO:teuthology.orchestra.run.smithi104.stdout:*hv01.front.sepi 44.4.53.2 2 u 4 64 77 0.083 -2.339 1.055 2024-03-28T14:09:43.031 INFO:teuthology.orchestra.run.smithi104.stdout:+hv02.front.sepi 96.245.170.99 3 u 6 64 77 0.087 +0.094 0.729 2024-03-28T14:09:43.031 INFO:teuthology.orchestra.run.smithi104.stdout:+hv03.front.sepi 74.6.168.72 3 u 8 64 77 0.074 -0.730 0.757 2024-03-28T14:09:43.032 INFO:teuthology.orchestra.run.smithi104.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-03-28T14:09:43.033 INFO:teuthology.orchestra.run.smithi171.stdout: remote refid st t when poll reach delay offset jitter 2024-03-28T14:09:43.033 INFO:teuthology.orchestra.run.smithi171.stdout:============================================================================== 2024-03-28T14:09:43.033 INFO:teuthology.orchestra.run.smithi171.stdout:*hv01.front.sepi 44.4.53.2 2 u 2 64 77 0.136 -1.487 0.659 2024-03-28T14:09:43.033 INFO:teuthology.orchestra.run.smithi171.stdout:+hv02.front.sepi 96.245.170.99 3 u 4 64 77 0.198 +0.334 0.673 2024-03-28T14:09:43.033 INFO:teuthology.orchestra.run.smithi171.stdout:+hv03.front.sepi 74.6.168.72 3 u 66 64 37 0.089 -1.009 0.736 2024-03-28T14:09:43.034 INFO:teuthology.orchestra.run.smithi171.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-03-28T14:09:43.035 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-03-28T14:09:43.047 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-03-28T14:09:43.047 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-03-28T14:09:43.062 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-03-28T14:09:43.076 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-03-28T14:09:43.088 INFO:teuthology.task.internal:Duration was 763.044329 seconds 2024-03-28T14:09:43.088 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-03-28T14:09:43.102 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-03-28T14:09:43.102 DEBUG:teuthology.orchestra.run.smithi104:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-28T14:09:43.105 DEBUG:teuthology.orchestra.run.smithi171:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-28T14:09:43.143 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-03-28T14:09:43.143 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi104.front.sepia.ceph.com 2024-03-28T14:09:43.143 DEBUG:teuthology.orchestra.run.smithi104:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-28T14:09:43.189 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi171.front.sepia.ceph.com 2024-03-28T14:09:43.189 DEBUG:teuthology.orchestra.run.smithi171:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-28T14:09:43.201 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-03-28T14:09:43.201 DEBUG:teuthology.orchestra.run.smithi104:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-28T14:09:43.233 DEBUG:teuthology.orchestra.run.smithi171:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-28T14:09:43.274 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-03-28T14:09:43.274 DEBUG:teuthology.orchestra.run.smithi104:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-28T14:09:43.301 DEBUG:teuthology.orchestra.run.smithi171:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-28T14:09:43.371 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-03-28T14:09:43.381 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-03-28T14:09:43.381 DEBUG:teuthology.orchestra.run.smithi104:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-28T14:09:43.394 DEBUG:teuthology.orchestra.run.smithi171:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-28T14:09:43.406 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-03-28T14:09:43.418 DEBUG:teuthology.orchestra.run.smithi104:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-28T14:09:43.438 DEBUG:teuthology.orchestra.run.smithi171:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-28T14:09:43.447 INFO:teuthology.orchestra.run.smithi104.stdout:kernel.core_pattern = core 2024-03-28T14:09:43.459 INFO:teuthology.orchestra.run.smithi171.stdout:kernel.core_pattern = core 2024-03-28T14:09:43.480 DEBUG:teuthology.orchestra.run.smithi104:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-28T14:09:43.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-28T14:09:43.504 DEBUG:teuthology.orchestra.run.smithi171:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-28T14:09:43.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-28T14:09:43.524 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-03-28T14:09:43.537 INFO:teuthology.task.internal:Transferring archived files... 2024-03-28T14:09:43.537 DEBUG:teuthology.misc:Transferring archived files from smithi104:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/pdonnell-2024-03-28_07:14:07-fs-wip-batrick-testing-20240327.230800-distro-default-smithi/7627919/remote/smithi104 2024-03-28T14:09:43.538 DEBUG:teuthology.orchestra.run.smithi104:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-28T14:09:43.571 DEBUG:teuthology.misc:Transferring archived files from smithi171:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/pdonnell-2024-03-28_07:14:07-fs-wip-batrick-testing-20240327.230800-distro-default-smithi/7627919/remote/smithi171 2024-03-28T14:09:43.572 DEBUG:teuthology.orchestra.run.smithi171:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-28T14:09:43.591 INFO:teuthology.task.internal:Removing archive directory... 2024-03-28T14:09:43.592 DEBUG:teuthology.orchestra.run.smithi104:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-28T14:09:43.615 DEBUG:teuthology.orchestra.run.smithi171:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-28T14:09:43.637 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-03-28T14:09:43.648 INFO:teuthology.task.internal:Not uploading archives. 2024-03-28T14:09:43.649 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-03-28T14:09:43.662 INFO:teuthology.task.internal:Tidying up after the test... 2024-03-28T14:09:43.662 DEBUG:teuthology.orchestra.run.smithi104:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-28T14:09:43.665 DEBUG:teuthology.orchestra.run.smithi171:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-28T14:09:43.668 INFO:teuthology.orchestra.run.smithi104.stdout: 658463 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 28 14:09 /home/ubuntu/cephtest 2024-03-28T14:09:43.685 INFO:teuthology.orchestra.run.smithi171.stdout: 658626 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 28 14:09 /home/ubuntu/cephtest 2024-03-28T14:09:43.687 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-03-28T14:09:43.697 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-03-28T14:09:43.724 INFO:teuthology.nuke:Checking targets against current locks 2024-03-28T14:09:43.754 DEBUG:teuthology.nuke:shortname: smithi104 2024-03-28T14:09:43.755 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-28T14:09:43.778 DEBUG:teuthology.nuke:shortname: smithi171 2024-03-28T14:09:43.779 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-28T14:09:43.782 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi104.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/pdonnell-2024-03-28_07:14:07-fs-wip-batrick-testing-20240327.230800-distro-default-smithi/7627919', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '22.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-28 13:48:17.374019', 'locked_by': 'scheduled_pdonnell@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBGXlb930Fp5uCSc+HMfKppybY3GP8wFgyECDsGraqfNuCbbML8e2DjaXRdJia3zcv5XKdUADEYo0s8OGWouN7lM='} 2024-03-28T14:09:43.809 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi171.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/pdonnell-2024-03-28_07:14:07-fs-wip-batrick-testing-20240327.230800-distro-default-smithi/7627919', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '22.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-28 13:48:17.376125', 'locked_by': 'scheduled_pdonnell@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBBOgEufC093sZkDP4SZ4QwXTRVg3TWB3hAP05pEnLzCzkgNvfFQrEf22R9X6bBjKRSoeKd/zi5NHter+nJ2py2c='} 2024-03-28T14:09:43.841 INFO:teuthology.orchestra.console.smithi104:Power off 2024-03-28T14:09:43.842 DEBUG:teuthology.orchestra.console.smithi104:pexpect command: ipmitool -H smithi104.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-28T14:09:43.865 DEBUG:teuthology.orchestra.console.smithi104:power off output: Chassis Power Control: Down/Off 2024-03-28T14:09:43.866 DEBUG:teuthology.orchestra.console.smithi104:pexpect command: ipmitool -H smithi104.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-28T14:09:43.882 INFO:teuthology.orchestra.console.smithi171:Power off 2024-03-28T14:09:43.882 DEBUG:teuthology.orchestra.console.smithi171:pexpect command: ipmitool -H smithi171.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-28T14:09:43.893 DEBUG:teuthology.orchestra.console.smithi104:check power output: Chassis Power is on 2024-03-28T14:09:43.906 DEBUG:teuthology.orchestra.console.smithi171:power off output: Chassis Power Control: Down/Off 2024-03-28T14:09:43.906 DEBUG:teuthology.orchestra.console.smithi171:pexpect command: ipmitool -H smithi171.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-28T14:09:43.930 DEBUG:teuthology.orchestra.console.smithi171:check power output: Chassis Power is on 2024-03-28T14:09:47.894 DEBUG:teuthology.orchestra.console.smithi104:pexpect command: ipmitool -H smithi104.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-28T14:09:47.931 DEBUG:teuthology.orchestra.console.smithi171:pexpect command: ipmitool -H smithi171.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-28T14:09:48.004 DEBUG:teuthology.orchestra.console.smithi104:check power output: Chassis Power is on 2024-03-28T14:09:48.042 DEBUG:teuthology.orchestra.console.smithi171:check power output: Chassis Power is on 2024-03-28T14:09:52.007 DEBUG:teuthology.orchestra.console.smithi104:pexpect command: ipmitool -H smithi104.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-28T14:09:52.043 DEBUG:teuthology.orchestra.console.smithi171:pexpect command: ipmitool -H smithi171.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-28T14:09:52.117 DEBUG:teuthology.orchestra.console.smithi104:check power output: Chassis Power is on 2024-03-28T14:09:52.153 DEBUG:teuthology.orchestra.console.smithi171:check power output: Chassis Power is on 2024-03-28T14:09:56.119 DEBUG:teuthology.orchestra.console.smithi104:pexpect command: ipmitool -H smithi104.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-28T14:09:56.154 DEBUG:teuthology.orchestra.console.smithi171:pexpect command: ipmitool -H smithi171.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-28T14:09:56.230 DEBUG:teuthology.orchestra.console.smithi104:check power output: Chassis Power is off 2024-03-28T14:09:56.265 DEBUG:teuthology.orchestra.console.smithi171:check power output: Chassis Power is off 2024-03-28T14:09:56.331 INFO:teuthology.orchestra.console.smithi104:Power off completed 2024-03-28T14:09:56.366 INFO:teuthology.orchestra.console.smithi171:Power off completed 2024-03-28T14:09:56.471 INFO:teuthology.run:Summary data: description: fs/shell/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1-mds-1-client-coloc conf/{client mds mgr mon osd} distro/ubuntu_latest mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile pg_health} tasks/cephfs-shell} duration: 763.0443294048309 failure_reason: 'Test failure: test_cd_with_args (tasks.cephfs.test_cephfs_shell.TestCD)' flavor: default owner: scheduled_pdonnell@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=7a9146a91cbe4b9c9cf66cd4591d6be4 status: fail success: false 2024-03-28T14:09:56.472 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-28T14:09:56.581 INFO:teuthology.run:FAIL